tom/ytstream
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Deploy it, and fix the six things installation found

Browse Source 
Both units are installed and running, 10 of 119 channels approved, 251 episodes
live in Jellyfin with verified DirectPlay. 345 tests. Six problems surfaced that no
test could have, and two of them were mine in the deploy scripts.

deploy.sh had a circular dependency with bootstrap.sh: deploy started the units and
told the operator to run bootstrap, but bootstrap refused to run until the state
directory existed, which only deploy creates. The units started against a
non-existent venv, failed 203/EXEC and restart-looped 17 and 21 times. deploy.sh now
creates the directory, calls bootstrap itself through runuser so the venv is not left
root-owned, and refuses to start units when the venv is still missing.

Deno was absent, and `doctor` is the only reason we know. It is mandatory rather than
nice-to-have — without a JS runtime yt-dlp cannot solve the n challenge, which
youtube-automate measured on this machine as 22 formats instead of 29 plus
throttling. Nothing else would have complained; playback would just have quietly
degraded. bootstrap.sh now installs it and asserts yt-dlp reports it.

Episodes had no synopsis at all, because materialise passed plot=None while both
sources hand us descriptions for free. Now plumbed through from RSS
(media:group/media:description) and from videos.list, which carries
snippet.description in the call already being made for durations — so the ~40% of
episodes older than RSS reaches get one too. That needed a schema v2 migration; v1
was left exactly as shipped so a fresh install and a migrated one are identical, and
a test asserts it.

`materialise --all` — the documented recovery from a Jellyfin metadata wipe — was
itself creating duplicates. Episode numbers were re-derived each run, and
next_episode() excludes the row being numbered, so re-materialising a day's videos in
a different order renumbered them and orphaned the old files. One run left 102
orphaned NFOs against 251 episodes. An episode number is now permanent once assigned,
and a video whose rel_path changes has its old files removed first. Running it twice
is now a no-op.

Two Jellyfin behaviours worth having in writing. It ignores <runtime> and
<durationinseconds> for episodes while reading the rest of the NFO happily, so a
.strm shows no duration until first played — not fixable without probing, which is
the one thing this design exists to avoid. And a plain /Library/Refresh does not
reliably re-read a rewritten NFO: after rewriting all 251, fifty kept their old empty
metadata. The fix is metadataRefreshMode=Default with replaceAllMetadata=false, which
took plots from 201 to 251 while the proxy served zero requests. §5's prohibition on
replaceAllMetadata=true still stands — that one probes. Exposed as
`ytstream refresh-metadata` and run automatically after `materialise --all`.

The measurement §5 has been waiting for: a full Jellyfin scan of 251 .strm files
took ~119 s, about 8 minutes per 1,000 episodes, and made zero media probes. That
last number is the fact the whole design rests on, now confirmed at scale on the real
library rather than on seven PoC files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Tom Flux
2026-08-12 17:21:35 +01:00
co-authored by Claude Opus 5
parent 61cc1672ec
commit d3bf8d6f19
14 changed files with 490 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plan.md
+102 -8
View File
@@ -1,15 +1,17 @@
# `ytstream` — implementation plan
**Target machine:** `susan`
**Status:** **built.** Phases 0–4 of §13 are code-complete with 337 passing tests, verified against
the live YouTube Data API and the running proxy. What remains is installation, which needs root
(`sudo deploy/deploy.sh`), and the cut-over in §12. The streaming PoC measurements this plan was
designed around are in **`FINDINGS.md`** alongside this file; what the build itself changed is in
**§17**.
**Status:** **deployed and running.** Both systemd units are installed and active, 10 of the 119
mirrored channels are approved, and 251 episodes are live in Jellyfin with correct metadata and
verified DirectPlay. 345 tests pass. What remains is the cron entries, curating the rest of the
subscription list, and the cut-over in §12.
**Repo:** `/opt/ytstream`, pushed to `/disks/git-repos/ytstream.git`, branch `main`. The PoC code
still lives in `/home/susan/ytstream` and is *not* under version control; Phase 2 moves it in and
retires that directory.
The streaming PoC measurements this plan was designed around are in **`FINDINGS.md`** alongside this
file. What the build changed is in **§17**; what deployment changed is in **§18**.
**Repo:** `/opt/ytstream`, pushed to `/disks/git-repos/ytstream.git`, branch `main`. The PoC
scaffolding under `/home/susan/ytstream` is superseded; the proxy now lives in `proxy/` and the
PoC-era media tree was moved to `/disks/Plex/_cache/ytstream-poc-tree-backup`.
**Relationship to `youtube-automate`:** ytstream **replaces** it. The two are entirely separate
trees, databases, services and Jellyfin libraries, and they will run side by side only for as long
@@ -959,3 +961,95 @@ real name.
| Asianometry backfill | 6 episodes |
| Generated `.strm` played through the proxy | h264 720p + aac, ranges honoured |
| NFO `durationinseconds` vs API truth | 889 vs 889 |
---
## 18. What deployment changed — 2026-08-12
Installing it found six things the tests could not. Two were my bugs in the deploy
scripts, one was a missing dependency, and three were Jellyfin behaviours that only
appear against a real library.
### The deploy scripts had a circular dependency
`deploy.sh` installed and started the units, then told the operator to run
`bootstrap.sh` — but `bootstrap.sh` refused to run until `/var/lib/ytstream`
existed, and only `deploy.sh` creates it. Neither could go first. The units started
against a venv that did not exist, failed `203/EXEC`, and restart-looped 17 and 21
times until the venv appeared.
`deploy.sh` now creates the state directory, calls `bootstrap.sh` itself via
`runuser` so the venv is not left root-owned, and refuses to start the units at all
if the venv is still missing. One command, correct order.
### Deno was missing, and `doctor` caught it
**Mandatory, not optional.** Without a JS runtime yt-dlp cannot solve the `n`
challenge; youtube-automate measured the consequence on this machine as 22 formats
instead of 29 and throttled downloads. The new venv had `yt-dlp-ejs` but no `deno`,
and nothing else would have noticed until playback quietly degraded.
`bootstrap.sh` now installs it — preferring a copy from the youtube-automate venv
while that still exists, falling back to the GitHub release — and verifies yt-dlp
reports `JS runtimes: deno`.
### Jellyfin ignores `<runtime>` and `<durationinseconds>` for episodes
It reads the rest of the NFO — `aired` and the `youtube` provider id both arrive —
but runtime comes only from a media probe, so a `.strm` episode shows **no duration
until it has been played once**. Not fixable from our side: the only way to supply
one is to probe, which means fetching every episode, which is the one thing this
design exists to avoid. Accepted limitation, stated here so nobody re-litigates it.
### Episodes had no plot at all
`strm.materialise` passed `plot=None`, so every synopsis was empty — while both
sources hand us descriptions for free. Now plumbed through: RSS carries
`media:group/media:description`, and `videos.list` carries `snippet.description` in
the call already being made for durations, so the ~40% of episodes older than RSS
reaches still get one. That needed a **schema v2 migration**; v1 was left exactly as
it shipped so a fresh install and a migrated one end up identical, which is asserted
by a test.
### `materialise --all` created duplicates instead of repairing
The documented recovery path from a metadata wipe was itself broken. Episode numbers
were re-derived on every run, and `next_episode()` excludes the row it is numbering,
so re-materialising a day's videos in a different order renumbered them — new
filenames, old files left behind. One run left **102 orphaned NFOs against 251
episodes**.
Two fixes: an episode number, once assigned, is now permanent and reused from the
row; and materialising a video that already has a different `rel_path` removes the
old files first. Running `materialise --all` twice in a row is now a no-op, verified
on the live tree and pinned by tests.
### There is a safe metadata refresh, and this is it
§5 says never to use `replaceAllMetadata=true`, and that stands. But a plain
`/Library/Refresh` does **not** reliably re-read a rewritten NFO — after rewriting
all 251, fifty kept their old empty metadata. The middle ground works:
```
POST /Items/{id}/Refresh?metadataRefreshMode=Default&imageRefreshMode=Default
&replaceAllMetadata=false&recursive=true
```
Measured across the whole 251-episode library: **plots and aired dates went from 201
to 251, and the proxy served 0 requests.** The safety is entirely in
`replaceAllMetadata=false` — with it `true`, Jellyfin discards what it has and
re-derives from the media. Exposed as `ytstream refresh-metadata`, and run
automatically after `materialise --all`.
### Measured on the deployed service
| | |
|---|---|
| Channels approved (of 119 queued) | 10 |
| Episodes materialised | **251** in 70 s |
| Filtered as Shorts / livestreams | 2 / 4 |
| Tree size | 49 MB for 753 files |
| **Jellyfin full scan** | **251 episodes in ~119 s** (~8 min per 1,000) |
| **Media probes during that scan** | **0** |
| Playback via Jellyfin `PlaybackInfo` | DirectPlay h264 720p + aac, 2049 s runtime |
| Episodes with plot / aired after refresh | 251 / 251 |
| `doctor` | all fatal checks pass |