From 8cca4fe1e2318e6cc57ea498d26e1ef425009c40 Mon Sep 17 00:00:00 2001 From: Tom Flux Date: Wed, 12 Aug 2026 15:26:40 +0100 Subject: [PATCH] Plan ytstream as a service, and mirror a YouTube account's subscriptions MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The streaming proof of concept works — FINDINGS.md records the measurements — but it runs by hand out of a home directory. This is the plan for making it a real service, plus the design for the new requirement: my brother subscribes to a channel on YouTube and it appears in Jellyfin without anyone touching an admin page. Three things drove the shape of the plan. It is a fork of youtube-automate, not a green-field build. That project is 3,543 lines and most of it — RSS discovery, episode numbering, NFO schema, scrypt auth, the admin UI — has nothing to do with downloading and was verified on this machine on 2026-08-11. Only download.py and reap.py actually die. Re-deriving the rest would mean re-earning knowledge we already paid for. Getting a Google API key for the subscription feature turns out to pay for itself twice over, so the plan leans on it much harder than the feature needs. subscriptions.list, playlistItems.list and videos.list are 1 unit each against 10,000/day, which means the whole metadata path can move off yt-dlp: exact upload dates from contentDetails.videoPublishedAt instead of approximate_date (measured wrong by up to 2 days, and episode numbers are derived from it), durations without an extraction per video, and no residential-IP request storm when a channel is added. yt-dlp is then only ever invoked by the proxy, for one video, because a human pressed play. That is a much better boundary than the one we have. The brother's channel is the sole source of truth. An earlier draft modelled subscriptions as multi-source claims so a manually-added channel could not be deleted by someone else's list; with one source that is dead weight, and speculative generality in the code path that deletes things is the wrong place to spend it. What survives instead is paranoia about removals, which now matter more rather than less: a 403, a timeout or an empty response is never read as an unsubscribe, a channel must be absent from three consecutive healthy syncs, and even then the tree stays on disk and the channel merely goes inactive. Verified while writing this, rather than assumed: - subscriptions.list accepts a channelId filter and is not documented as needing an authorized request, so an API key is enough. It returns 403 — not an empty list — when subscriptions are private, which is what makes the removal safeguards able to tell "he made them private again" apart from "he unsubscribed from everything". - There is no scraping fallback. /@cflux1030/channels returns 200 but silently serves the Home tab: four tabs, zero channel ids in ytInitialData. YouTube retired the public subscriptions tab, so the privacy checkbox is mandatory rather than merely convenient. - @cflux1030 resolves to UCPcTWaLV8zwx4WP4QExHj4Q, so no channels.list call is needed at build time. - OAuth as a fallback carries a trap worth writing down: a consent screen in "Testing" status issues refresh tokens that expire in 7 days, and youtube.readonly is not one of the exempt basic scopes. Catalogue depth is bounded at 3 months or 300 videos per channel, whichever comes first. The plan says plainly that this bounds the initial build and not the library, which grows forward indefinitely — roughly 15,000 episodes a year at 20 channels — so the Jellyfin scan-cost measurement in Phase 3 is what tells us when that becomes a problem. Deliberately not solved now. One number is still open: subsync_max_new depends on how many channels he actually follows, which only the day-one API call can tell us. Phase 0 closes it. Co-Authored-By: Claude Opus 5 --- .gitignore | 5 + FINDINGS.md | 454 ++++++++++++++++++++++++++++++++ plan.md | 741 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 3 files changed, 1200 insertions(+) create mode 100644 .gitignore create mode 100644 FINDINGS.md create mode 100644 plan.md diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..dfcfab8 --- /dev/null +++ b/.gitignore @@ -0,0 +1,5 @@ +__pycache__/ +*.py[cod] +*.egg-info/ +.pytest_cache/ +.venv/ diff --git a/FINDINGS.md b/FINDINGS.md new file mode 100644 index 0000000..f88e6b6 --- /dev/null +++ b/FINDINGS.md @@ -0,0 +1,454 @@ +# `ytstream` — results on susan + +Date: 2026-08-12. Answers every open question in the `ytstream` handover: §3, §6, +and §7 steps 1–4. Both Jellyfin gates pass (§6 below). §7 step 5 (integrating +with youtube-automate) is **not** done and `/opt/youtube-automate` is unmodified. + +Environment note that matters: the PoC calls bare `yt-dlp`. On susan, `PATH` +resolves that to `/usr/local/bin/yt-dlp`, which is **2023.11.16** and has no POT +plugin. The working install is the automation's venv: + +``` +export PATH=/var/lib/youtube-automate/venv/bin:$PATH # yt-dlp 2026.07.04 + # + bgutil-ytdlp-pot-provider 1.3.1 +``` + +That venv's plugin matches the running `bgutil-provider` container (1.3.1) and +retrieves gvs PO tokens correctly. Before this was noticed the container had been +up 15 hours with nothing able to talk to it — no yt-dlp plugin directory existed. + +--- + +## 1. Verified against real videos + +Four videos, 2 to 46 minutes. Two are from the live `subs.db`, so they are +representative of what the automation actually handles. + +| video | length | TTFB (`--growing`) | ready (default mode) | duration served | truth | +|---|---|---|---|---|---| +| `D6ocsyQLKy8` | 2 min | 7.1s | ~7s | 120.62s | 121s | +| `dQw4w9WgXcQ` | 3.5 min | 7.1s | ~9s | 213.13s | 213s | +| `J9O3sxoMs5U` | 21 min | 6.9s | **20s measured** | 1259.47s | 1259s | +| `NH2MhBFQm9w` | 46 min | 6.5s | ~50s | 2789.45s | 2789s | + +Full `ffmpeg -f null -` decode exits 0 in every case, with both system ffmpeg +5.1.7 and Jellyfin's bundled 7.1.3. Codecs are always `h264 1280x720` + `aac`. +"ready" figures other than the 21-minute one are derived from the measured +~8 MB/s pull rate, not separately timed. + +**`--load-info-json` reuses the extraction.** This was the load-bearing unknown. +Verbose output shows zero re-extraction markers and no second POT fetch — it goes +straight to `[info] Downloading 1 format(s): 136`. The one-extraction design holds. + +**Concurrency is not a problem.** 5/5 clean trials of the exact PoC pattern (one +fresh extraction, two concurrent fetches). §3's worry about the POT provider under +concurrent access looks unfounded. + +**Throughput ~8 MB/s**, roughly 60× realtime. 368 MB for 46 minutes of 720p avc1 +≈ **480 MB/hour**, about half the handover's 1 GB/hour estimate. + +--- + +## 2. Two bugs found in the format picker — both would have shipped + +Both came from the same root cause: YouTube advertises variants with **identical +`abr`**, so `max()` tie-broke on list order. + +| video | picked | consequence | +|---|---|---| +| `NH2MhBFQm9w` | `140-drc` | HTTP 403 — DRC variants are advertised but not served | +| `J9O3sxoMs5U` | `140-0` | **German audio on an English podcast** (9-language upload) | + +The second is the dangerous one: silent, and it would have played. + +Fixed by ranking `language_preference` first (original track = 10, dubs = -1), +then non-DRC, then codec, then bitrate. Language correctness deliberately outranks +the DRC check: a 403 is a loud failure, wrong-language audio is a silent one. +`pick_formats` now also logs the chosen language and warns if a non-original track +is selected on a multi-language upload. + +Verified: `140-drc` and `140-0` reproducibly 403 while `140`, `140-7`, `139`, +`251` and `136` all download fine from the same `info.json`. + +**Later correction — both bugs had a common upstream cause.** The DRC and dubbed +variants existed *only* because `mweb` was in `player_client`. With +`player_client=default` alone, extraction returns four audio formats (139, 249, +140, 251), no DRC variants and no language variants at all — even on the +nine-language video. So "DRC formats 403" was really "mweb formats 403", and the +German-audio bug could only ever have happened via mweb. `mweb` is now dropped +(§9). The `language_preference` and DRC rules stay in the picker as cheap +insurance if a yt-dlp update changes what `default` returns. + +--- + +## 3. Two defects not mentioned in the handover — both now fixed + +### Duration collapses while the file grows + +Jellyfin's own ffprobe 7.1.3 on the 46-minute video: + +``` +during growth : format.duration = 5.92 <- wrong +after complete: format.duration = 2789.45 <- correct +``` + +§5 says scrubbing won't work while growing. It is worse than that: the **duration +itself** is wrong, which would break Jellyfin's runtime display, seek bar, resume +point and watched state. + +This cannot be fixed at the container level. Both attempts failed: + +- Patching `mvhd.duration` in place — ignored. +- Inserting an `mehd` box with the correct `fragment_duration` — the box is + written correctly (`mehd v0 fragment_duration=60000`, file still decodes clean), + and ffmpeg **still** ignores it and recomputes from the fragments present. + +Also confirmed: ffmpeg writes `mvhd.duration=0` and **no `mehd`** regardless of +whether it knows the input duration — a mux from regular files and one from FIFOs +produce byte-identical headers. `mp4boxes.py` and `mehd_patch.py` in this +directory reproduce both results. + +**Fix:** waiting for the mux is now the default. Correct duration, `Accept-Ranges` +and working seeks on the very first request. Costs TTFB — 20s for a 21-minute +video instead of 6.9s. `--growing` keeps the old low-latency behaviour and now +prints a warning. Given yt-dlp runs at ~60× realtime the wait is bounded and +predictable, and it is the same order of magnitude as the 30-second non-seekable +window §5 already accepted. + +The Jellyfin gate has since settled this: Jellyfin takes its runtime **only** from +its own probe of the stream, never from the NFO (§6). So `--growing` would hand it +~6 seconds and there is no way to claw the TTFB back. Wait-for-complete is +mandatory, not a preference. + +### Range end was ignored + +`bytes=1000000-1000999` returned **367 MB** instead of 1000 bytes — the server +parsed only the start offset and always streamed to EOF. This directly undermines +§6's mitigation #2, which depends on serving small bounded probe ranges. + +Now parses both ends, honours `bytes=N-M`, `bytes=N-` and the suffix form +`bytes=-N`, clamps an end past EOF, and returns `416` with `Content-Range: */total` +for an unsatisfiable start. `test_range.py` covers all of it — 25 assertions, +no network required. + +--- + +## 4. One correction to §6's premise + +The `` blocks in the existing episode NFOs were written +by **Jellyfin**, not by the automation — ``, ``, `micodec` +and absolute-path `` are Jellyfin's own NFO output, produced after it probed +the real `.mp4` files. + +So there was no evidence that pre-seeding `streamdetails` suppresses probing — +mitigation #1 was **untested**, not partly proven. It has since been tested with a +control group and it does nothing at all (§6). + +--- + +## 5. Intermittent 403s — found, and now retried + +Two 403s were observed across roughly ten real runs, and they **moved between +formats** — once on video `136`, once on audio `140`, on the same video that had +succeeded minutes earlier. + +The first guess was transient rate limiting, because the failures clustered after +~1.5 GB had been pulled from one video. **That guess is wrong.** A retry two +seconds later with a fresh extraction succeeded every single time, and no rate +limit clears in two seconds. See §9 for what was actually measured. + +What matters is the consequence: a single 403 on either stream killed the whole +playback with no recovery. It then struck again during the Jellyfin test itself, +so it is now **fixed in `ytstream.py`**: up to `--max-retries` attempts (default +2), each with a fresh extraction, retried only while nothing has been served yet. +That recovered every occurrence seen — including the one during the gate, where +attempt 1 failed and attempt 2 succeeded. + +Retry is a mitigation, not a root-cause fix. `ytstream_poc.py` still has no retry; +it is kept as the record of what was originally verified. + +--- + +## 6. Jellyfin: both gates PASS + +§7 step 3 (playback) and §6 (scan probing) are now answered against the live +Jellyfin 10.11.4 on susan, using the two channels already subscribed in +`subs.db` — 7 episodes, 2 shows, real metadata. + +Library `YouTube (stream)` → `/disks/Plex/_ytstream`, built by +`make_strm_tree.py`. `SaveLocalMetadata=False` on purpose: the real YouTube +library has it **True**, which is why Jellyfin rewrote those NFOs, and leaving it +on here would have overwritten the test NFOs and destroyed the control group. + +### §6 scan-time probing — the open risk does not materialise on a normal scan + +| operation | probes | pipelines started | +|---|---|---| +| first scan of a new library | **0** | 0 | +| repeat scan, `replaceAllMetadata=false` | **0** | 0 | +| refresh with `replaceAllMetadata=true` | **yes, immediately** | would be all of them | + +A normal scan ingests all 7 `.strm` episodes — correct show, season, episode +numbers, aired dates — **without a single request to the proxy**. No scan storm. +Episode `RunTimeTicks` stays null until something plays the item. + +The dangerous path is a **full metadata replace**, which a user can trigger from +the UI. Jellyfin probes the `.strm` target immediately. Two corrections to §6's +assumptions while doing so: + +- Probes are **open-ended `Range: bytes=0-`**, not the "small ranges from offset + 0" §6 assumed. You cannot identify a probe from its Range header, so + mitigation #2 cannot key off request shape. It can still work by serving the + first few MB from a permanent cache — the probe takes what it needs and hangs + up — but the discriminator has to be something else. +- **Refusing a probe causes a retry storm.** Returning 503 produced 9 retries in + ~2 seconds from `Lavf/61.7.100`, after which the refresh aborted and left six + of seven episodes with their episode numbers wiped. Whatever the mitigation is, + it must answer, not refuse. + +`--max-pipelines` (default 2) is the backstop: a full replace throttles to two +concurrent fetches instead of seven. + +### §6 mitigation #1 — pre-seeded streamdetails do nothing + +The `alternate` control group settles it. Episodes carrying full +`` with `` behaved **identically** to +those without: `RunTimeTicks` null after a scan in both groups. Jellyfin took +`ProviderIds` and `PremiereDate` from the same NFOs, so they were definitely +being read — it simply does not use NFO stream details or runtime for a `.strm` +item. Mitigation #1 is a dead end, and it is also unnecessary given a normal scan +does not probe at all. + +This also settles the `--growing` question from §3: Jellyfin has **no** duration +until it probes the stream itself, so a growing file would hand it ~6 seconds. +**Stay in the default wait-for-complete mode.** + +### §7 step 3 playback — passes, and direct-plays + +Driven through Jellyfin's own API (`PlaybackInfo`, then `/Videos/{id}/stream`): + +| | warm cache | cold, 46-minute episode | +|---|---|---| +| `PlaybackInfo` latency | 0s | **55s — Jellyfin waited, no timeout** | +| `Container` | `mp4` | `mp4` | +| `RunTimeTicks` | 120.62s (truth 120.62) | **2789.45s (truth 2789)** | +| `SupportsDirectPlay` | **True** | **True** | +| streams | h264 1280x720 + aac | h264 1280x720 + aac | + +Fetching the episode through Jellyfin's streaming endpoint returned 10,654,544 +bytes **byte-identical** to what the proxy produced, decoding cleanly. So +Jellyfin direct-plays it with no transcode — which was the whole point of +preferring avc1+mp4a. + +Two things worth knowing: + +- A cold 46-minute episode makes the user wait ~55s at the play button. That is + the price of correct duration and seeking. It is bounded and predictable. +- One ffprobe of a 368 MB file generated **481 range requests** to the proxy. + All local and cheap, but it makes the access log noisy and is worth a look if + throughput ever matters. + +### Artwork works, and costs nothing extra + +Episode thumbnails need **no API call**: the URL is derivable from the video id, +which is already in the filename. `add_thumbnails.py` fetches +`i.ytimg.com/vi//maxresdefault.jpg` (~200 KB, 1280x720) with `hqdefault.jpg` +(~22 KB) as a fallback, writing `-thumb.jpg`. Channel avatar and banner +become `poster.jpg` / `fanart.jpg` and do need one yt-dlp call per channel. + +Verified in Jellyfin: all 7 episodes carry a 16:9 `Primary` image, both series +have a poster and a backdrop, and an image refresh triggers **0 probes**. Artwork +is entirely independent of the streaming path. + +Two things learned while doing it: + +- **Jellyfin re-reads an NFO when its mtime changes.** Episode names were + filename-derived until the NFOs were rewritten, after which they picked up the + real titles from `` — full parity with the .mp4 library. So correcting + metadata does not need a full replace (which would probe); just rewrite the NFO + and run a `Default` refresh. +- A `title` collision bug shipped briefly: the tree generator joins `video` and + `channel`, **both of which have a `title` column**, so `row["title"]` was the + video title and both shows got renamed after whichever episode sorted first. + Fixed by passing show fields explicitly. Worth remembering for §7 step 5, where + the same join exists. + +### Catalogue size — the current library is small only because of retention + +The 7 episodes are not a limit of the design, they are what `subs.db` holds: +`backfill_days = 7` and `retention_days = 9`, both driven by disk. A streaming +library has no such constraint, so it should carry the whole back catalogue. + +Actual sizes, from one `--flat-playlist` call per channel: + +| channel | videos in catalogue | in subs.db | +|---|---|---| +| Pitch Side | **1249** | 15 | +| The Pyramid Podcast | 3 | 3 | + +At 1249 episodes the cost is trivial for `.strm` + `.nfo` (a few MB) and ~250 MB +of thumbnails at maxres. But two things get harder: + +1. **`upload_date` is not in a flat listing.** Season and episode numbers are + `year` and `MMDD*10 + ordinal`, so dates are required. + `--extractor-args youtubetab:approximate_date` supplies timestamps in the same + single request, but they are **wrong by up to 2 days** (measured against + subs.db: `J9O3sxoMs5U` reported 08-12, actually 08-10). That yields wrong + episode numbers and, worse, *unstable* ones — a regeneration would renumber + episodes and Jellyfin would lose watched state. If approximate dates are used + they must be persisted on first sight and never recomputed, exactly as + specs.md already mandates for `dir_name`. Accurate dates cost one metadata + extraction per video: ~1249 requests per channel, one-time. +2. **The full-metadata-replace footgun scales with the catalogue.** At 7 episodes + it is an annoyance; at 1249 it would try to fetch every video in the channel. + `--max-pipelines 2` turns a stampede into a slow grind but does not stop it. A + cold-start rate limit should land before any full catalogue does. + +### Running it for real + +The proxy must outlive a shell. `ytstream.service` is written but **not +installed** — that needs sudo: + +``` +sudo cp /home/susan/ytstream/ytstream.service /etc/systemd/system/ +sudo systemctl daemon-reload && sudo systemctl enable --now ytstream +``` + +The unit pins `PATH` to the automation venv, which is load-bearing: bare `yt-dlp` +would otherwise resolve to the 2023.11.16 binary in `/usr/local/bin`. + +To remove everything: delete the `YouTube (stream)` library in the Jellyfin UI, +then `rm -rf /disks/Plex/_ytstream`. + +--- + +## 7. Files + +| file | purpose | +|---|---| +| `ytstream.py` | **the proxy.** Multi-video: `/watch/<video_id>`, `/healthz`. Retry, pipeline cap, LRU cache, `--no-fetch` | +| `ytstream.service` | systemd unit, not installed (needs sudo) | +| `make_strm_tree.py` | builds the .strm/.nfo library from subs.db | +| `add_thumbnails.py` | episode thumbnails + channel poster/fanart, idempotent | +| `ytstream_poc.py` | the original single-video PoC, kept as the record of what was verified | +| `test_range.py` | 25 range-server assertions, no network | +| `test_limits.py` | 18 assertions for the rate limiter, pipeline cap and --no-fetch | +| `run_test.sh` | one end-to-end acceptance run; prints TTFB, codecs, duration | +| `trial_403.sh` | measures the intermittent-403 rate for the PoC's access pattern | +| `mp4boxes.py` | dumps fMP4 boxes and the duration fields | +| `mehd_patch.py` | the `mehd` insertion — kept because the negative result matters | + +## 8. Where this stands + +The handover's §8 advice was to build the batch downloader first. That is already +done — `/opt/youtube-automate` is live — so this proxy is the second-phase +experiment it was meant to be. + +Every unknown in §3 and §7 resolves in the design's favour and both Jellyfin gates +pass. The architecture holds: ffmpeg never touches googlevideo, the mux is a pure +copy, and Jellyfin direct-plays the result with the correct duration. + +Three things are known-imperfect rather than unknown: + +1. **Cold start costs ~55s on a 46-minute episode** (~21s for 22 minutes). + Inherent to wait-for-complete, and wait-for-complete is mandatory because + Jellyfin takes duration only from its own probe. +2. **A full metadata replace probes every episode.** Now bounded by + `--max-starts` as well as `--max-pipelines` (§9), but still don't run one over + a large library. +3. **The intermittent 403 is still unexplained.** Several plausible causes are + ruled out (§9) and retry recovers it every time, but that is a mitigation. + +Not done, and the honest next steps: + +- `ytstream.service` is written but **not installed** — that needs sudo. Until + then the library only plays while the proxy runs in a shell. +- Nothing writes `.strm` automatically. `make_strm_tree.py` is a one-shot from + `subs.db`; §7 step 5 is untouched and `/opt/youtube-automate` is unmodified. +- The library covers 7 episodes because that is what `subs.db` holds. A real + deployment should carry the full catalogue — see the catalogue section in §6 + for the two things that get harder at 1249 episodes. +- Two libraries coexist by choice: `YouTube` (.mp4) and `YouTube (stream)` + (.strm), same 7 episodes, for comparison. +- tmpfs LRU eviction is implemented but has never fired; the cap was never + reached in testing. + +--- + +## 9. Hardening pass (2026-08-12, after the gates passed) + +### The 403 investigation — the obvious fix would have made it far worse + +`player_client` was `default,mweb`. Inspecting the `c=` and `pot=` parameters on +the chosen format URLs showed something that looked like an open-and-shut cause: + +| `player_client` | video/audio served from | PO token in URL | +|---|---|---| +| `default,mweb` | `ANDROID_VR` | **no** | +| `mweb` | `MWEB` | **yes** | + +The obvious conclusion — "we are using token-less URLs, pin the client that gets a +token" — is **wrong**, and testing it was worth the ten minutes: + +``` +player_client=mweb -> 0 ok / 6 fail (403 on both streams, every trial) +player_client=default -> works +``` + +mweb fails **both** as a single process and via `--load-info-json`, so it is not a +round-trip problem — mweb URLs are simply rejected right now, PO token and all. +`ANDROID_VR` *without* a token works; `mweb` *with* one does not. Had this been +"fixed" by pinning mweb, every playback would have broken. + +Full client sweep on the same video: + +| client | result | +|---|---| +| `default` (→ android_vr) | **works** | +| `android_vr` | works | +| `mweb` | 403, every time | +| `web`, `ios`, `web_safari`, `tv` | no usable formats — SABR forced | + +So there is no client-level fix available: `default` is the only thing that works, +and it is already what was in use. What changed: + +- **`mweb` dropped from `player_client`.** It contributed nothing usable, and it + was the sole source of the DRC and dubbed-language formats behind both original + picker bugs (§2). Removing it eliminates those hazards before the picker sees + them, and cut extraction time from ~5s to **2.3s** — one fewer client API call, + which comes straight off cold-start latency. +- Left as `default` rather than pinned to `android_vr`, so a yt-dlp update can + follow YouTube if android_vr stops working. + +**The 403 remains unexplained and still occurs** — one retry fired during the +post-hardening Jellyfin test (`retried: 1, failed: 0`). Retry with re-extraction +recovers it every time observed, so it stays the mitigation. What is now ruled +out: rate limiting (a 2s retry succeeds), `--load-info-json` (fails identically +without it), concurrency (5/5 clean), and PO token absence (token-less URLs are +the ones that work). + +### Cold-start rate limiter + +`--max-pipelines` caps concurrency, which only slows a runaway metadata refresh +down — over 1249 episodes it would still fetch the whole catalogue, just two at a +time. `--max-starts` (default 20 per `--starts-window`, default 3600s) bounds the +total instead. + +The discriminator is that it counts **cold starts only**. Cache hits are never +limited, so re-watching and resuming keep working with the budget exhausted. A +person starts a handful of new videos an hour; a refresh storm hits 20 in seconds. + +When the budget is spent the proxy refuses with 503 and logs loudly. From the §6 +measurements, a refused probe produces a short libavformat retry burst and then +aborts the refresh — which is the point: an aborted refresh costs some +re-derivable metadata, a runaway one costs the entire catalogue in YouTube +traffic. + +`test_limits.py` covers the limiter, the concurrency cap and `--no-fetch` with the +pipeline runner stubbed out — 18 assertions, no network. + +### Known gap + +Sessions live in memory, so a proxy restart orphans whatever is already on tmpfs +and the next request re-fetches. Harmless on this host (tmpfs is cleared on +reboot anyway) but worth knowing if a service restart ever looks like a stall. diff --git a/plan.md b/plan.md new file mode 100644 index 0000000..a64cfce --- /dev/null +++ b/plan.md @@ -0,0 +1,741 @@ +# `ytstream` — implementation plan + +**Target machine:** `susan` +**Status:** streaming PoC verified end to end against real videos and real Jellyfin — every +measurement behind this plan is written up in **`FINDINGS.md`** alongside this file. Nothing is +installed as a service yet. This document is the plan for turning it into one. + +**Repo:** `/opt/ytstream`, pushed to `/disks/git-repos/ytstream.git`, branch `main`. The PoC code +still lives in `/home/susan/ytstream` and is *not* under version control; Phase 2 moves it in and +retires that directory. + +**Relationship to `youtube-automate`:** ytstream **replaces** it. The two are entirely separate +trees, databases, services and Jellyfin libraries, and they will run side by side only for as long +as it takes to satisfy §13. `youtube-automate` is then decommissioned (§12). Nothing in +`/opt/youtube-automate` is modified by this work. + +--- + +## 1. What we're building + +A DVR-shaped YouTube library for Jellyfin that **stores no video bytes**. + +`youtube-automate` downloads each video to disk and points Jellyfin at the file. ytstream writes a +~50-byte `.strm` file containing a URL, and materialises the actual video only when somebody +presses play — fetched on demand by a local proxy, held on tmpfs, dropped when the cache fills. + +### Explicitly in scope + +- Mirroring **one YouTube account's subscriptions** as Jellyfin TV series, one episode per video — + that account is the source of truth for what exists (§4) +- **Automatic subscription pickup from a YouTube account's public subscription list** (§4) — the + new requirement +- A bounded back catalogue per channel — **3 months or 300 videos, whichever comes first** — that + then grows forward indefinitely (§3, §5) +- Just-in-time streaming via the proxy, with per-video caching +- NFO metadata, episode thumbnails, channel poster/fanart +- An admin UI for subscriptions, status and settings +- Running as real systemd services with real logs and real alerting + +### Explicitly out of scope — do not build these + +- Any form of transcoding. susan is a dual Westmere Xeon with **no AVX**; software transcode is off + the table. The proxy produces `-c copy` fMP4 that Jellyfin direct-plays, and that is the only + supported path. +- Downloading and keeping video files. If we want a permanent copy of something, that is a + different tool. +- Playlists, Shorts, livestreams, comments, community posts, memberships. +- Anything that runs on `victoria` (the Linode). susan's residential IP is a load-bearing part of + not getting flagged by YouTube. +- A YouTube *account* login / cookies. PO tokens only, exactly as `youtube-automate` does it. + +--- + +## 2. This is a fork, not a green-field rewrite + +`youtube-automate` is 3,543 lines and most of it is correct, verified, and has nothing to do with +downloading. Rewriting it from scratch would mean re-deriving the RSS filtering, the episode +numbering, the NFO schema and the auth code — all of which were validated on the live machine on +2026-08-11 and are documented in `/opt/youtube-automate/specs.md`. + +So: copy the tree, then add, delete and replace. + +| `youtube_automate` module | Fate in `ytstream` | Note | +|---|---|---| +| `discovery.py` | **Lift, then extend** | UULF-feed polling is the cheapest correct incremental source and stays. Gains an API-backed full-catalogue backfill (§3). | +| `naming.py` | **Lift unchanged** | Season = upload year, episode = `MMDD*10 + ordinal`. Keep byte-identical so both trees sort the same during the overlap. | +| `nfo.py` | **Lift, small change** | Drop `<fileinfo><streamdetails>` — measured to accomplish nothing (FINDINGS §6). Keep `<durationinseconds>`. | +| `channels.py` | **Lift, minus artwork fetch** | Artwork moves to the API/`i.ytimg.com` path already built in `add_thumbnails.py`. | +| `db.py`, `settings.py`, `util.py`, `config.py` | **Lift** | New DB file and new schema version (§7). | +| `web/` (auth, server, templates) | **Lift** | 838 lines of working scrypt auth + admin UI. New routes for subscription sources. | +| `jellyfin.py` | **Lift, harden** | Library refresh must never be `replaceAllMetadata` (§5). | +| `doctor.py` | **Lift, extend** | Add checks for API key validity, proxy health, `.strm` orphan count. | +| `ytdlp.py` | **Lift, narrow** | Only the proxy calls yt-dlp now. | +| `download.py` (330 lines) | **Delete** | Replaced by `strm.py`, which writes a text file. | +| `reap.py` (150 lines) | **Delete, replace** | Reaping existed to reclaim disk. There is no disk to reclaim (§3). Replaced by a much smaller "retire a video whose channel is gone" path. | +| `runner.py` | **Rewrite** | The run loop changes shape: sync subscriptions → poll → materialise → refresh. | +| — | **New: `strm.py`** | Writes `.strm` + `.nfo` + thumbnail for one video. | +| — | **New: `subsync.py`** | The brother-subscription puller (§4). | +| — | **New: `api.py`** | YouTube Data API v3 client (§3). | +| — | **New: `proxy/`** | `/home/susan/ytstream/ytstream.py`, moved in and split up. | + +Everything lifted keeps its tests. `youtube-automate` has 12 test modules; they come across too. + +--- + +## 3. The pivot: the YouTube Data API becomes the metadata source + +The subscription feature (§4) forces us to get a Google API key. Once we have one, it is worth +noticing what else it buys, because it resolves the two problems that would otherwise make the +full back catalogue impractical. + +**Verified quota costs** (`developers.google.com/youtube/v3/determine_quota_cost`): a new project +gets **10,000 units/day** shared across everything except `search.list` and `videos.insert`, which +have their own 100-call/day buckets. `subscriptions.list`, `playlistItems.list`, `videos.list` and +`channels.list` are **1 unit each**, and each returns up to 50 items. + +That changes the arithmetic completely: + +| Job | Method | Calls | Units | +|---|---|---|---| +| Sync one account's subscriptions, hourly | `subscriptions.list` | 24/day (1 page) | 24 | +| Backfill one channel to the §5 bound (≤300 videos) | `playlistItems.list` @50 | ≤6 | ≤6 | +| Durations for those videos | `videos.list` @50 ids | ≤6 | ≤6 | +| Steady-state incremental discovery | **RSS feed** | — | **0** | + +A 20-channel initial build costs at most **240 units** — under 3% of one day's quota, once. Steady +state costs under 100 units/day. **We will never come close to the limit**, provided we never touch +`search.list` (100 calls/day, and we have no use for it). + +### Why this matters more than the quota + +**It takes yt-dlp out of the cataloguing path entirely.** Today, `youtube-automate` needs a yt-dlp +extraction per video to learn an upload date during backfill, at roughly 2.3s each. For a 20-channel +build at the §5 bound that is 6,000 videos ≈ **4 hours of continuous requests** to YouTube from a +residential IP — the "scan storm" failure mode the original handover warned about, and it recurs +every time a channel is added. The API is a documented, keyed, quota-metered endpoint that does not +care about PO tokens, SABR, or client selection, and cannot be rate-limited by YouTube's anti-bot +heuristics. + +So the split becomes: + +- **Cataloguing** (what exists, when, how long, what it's called) → **YouTube Data API + RSS**. No + yt-dlp, no PO token, no IP-flagging risk. +- **Playback** (actual media bytes) → **yt-dlp inside the proxy**, one video at a time, only when a + human pressed play. + +That is a much better boundary than the current one, and it is the main reason to build ytstream as +a new service rather than patch `youtube-automate`. + +### It also fixes exact upload dates + +Measured during the PoC: `--flat-playlist` reports `timestamp: None`, and +`youtubetab:approximate_date` is **wrong by up to 2 days**. Since season/episode is derived from +the upload date, an approximate date means episodes numbered into the wrong day — and Jellyfin +caches episode numbers, so fixing it later is a metadata-wipe operation. `playlistItems.list` +returns `contentDetails.videoPublishedAt` as an exact RFC-3339 timestamp. Use it. + +### One thing to verify before relying on it + +`youtube-automate` polls the **UULF** playlist (`UU` with `LF` spliced in), which is undocumented +but excludes Shorts and livestreams at the cheapest possible point — verified in specs.md §4. +Whether `playlistItems.list` accepts a UULF id is **unverified**; only `UU` is documented. + +- If UULF works: use it, and the existing filtering carries over unchanged. +- If it 404s: fall back to `UU` (definitely works, includes Shorts and livestreams) and filter with + the `videos.list` call we are making anyway — `contentDetails.duration < min_duration_seconds` + drops Shorts, and the presence of `liveStreamingDetails` drops streams. + +Either way it is one extra unit per 50 videos. Verify with a single curl on day one: + +```sh +curl -s "https://www.googleapis.com/youtube/v3/playlistItems?part=contentDetails\ +&playlistId=UULF2EvK7nHUOEw1IvWFpTourQ&maxResults=5&key=$KEY" | head -40 +``` + +--- + +## 4. Subscription mirroring — the new requirement + +**Goal:** the brother subscribes to a channel on YouTube, and it appears in Jellyfin without anyone +touching an admin page. + +### The account to mirror — resolved + +| | | +|---|---| +| Handle | `@cflux1030` | +| Channel id | **`UCPcTWaLV8zwx4WP4QExHj4Q`** | +| Display name | `C Flux` | + +Resolved via yt-dlp on 2026-08-12, so no `channels.list` call is needed at build time — seed it as +the single `source` row. + +**This account is the sole source of truth.** There is no second source and no parallel manual +subscription workflow: what he follows on YouTube is what exists in Jellyfin. Tom's own +subscriptions are explicitly not part of this. That is a simplification (§4.3) but it moves all the +weight onto the removal safeguards (§4.4), because nothing else protects a channel any more. + +### It works, with one condition on his side + +Confirmed against the API reference: `subscriptions.list` accepts a `channelId` filter — +*"The API will only return that channel's subscriptions"* — and unlike `mine`, +`mySubscribers` and `myRecentSubscribers` it is **not** documented as requiring an authorized +request. So a plain API key is enough. The condition is that his subscriptions must be public: +the implementation guide states the API returns **403** if the channel *"does not publicly expose +its subscriptions and the request is not authorized by the channel's owner"*, and the errors table +lists `subscriptionForbidden` (403) — *"The requester is not allowed to access the requested +subscriptions."* + +This is a good failure mode: **403, not an empty list.** We can tell "he made his subscriptions +private again" apart from "he has no subscriptions", which matters a lot for §4.4. + +**There is no scraping fallback — verified.** `https://www.youtube.com/@cflux1030/channels` returns +200 but silently serves the Home tab: the rendered `ytInitialData` lists exactly four tabs (Home, +Videos, Playlists, Search) and contains **zero** channel ids. YouTube retired the public +subscriptions tab, so the API is the *only* route to this list. That makes the privacy checkbox +genuinely mandatory rather than merely the convenient path, and it means a 403 has no workaround +short of §4.2. + +### 4.1 Setup — what has to happen once + +**Google side** (Tom, ~5 minutes, free, no billing account required): + +1. `console.cloud.google.com` → new project, e.g. `ytstream`. +2. APIs & Services → Library → **YouTube Data API v3** → Enable. +3. Credentials → Create credentials → **API key**. +4. Restrict the key: Application restrictions → *None* (it is called from a server, so referrer and + Android/iOS restrictions do not apply; an IP restriction is optional and breaks if susan's + residential IP rotates). API restrictions → **YouTube Data API v3 only**. +5. Paste it into the ytstream admin UI. It is stored in the `setting` table like + `jellyfin_api_key` already is — never in the repo, never in a systemd unit. + +No OAuth consent screen. No app verification. No user-facing consent flow. + +**Brother's side** (one checkbox, and it is the only thing he has to do): + +- youtube.com → Settings → **Privacy** → uncheck **"Keep all my subscriptions private"**. + +**Verify immediately** — before writing any of `subsync.py` — that his account actually returns +data: + +```sh +curl -s "https://www.googleapis.com/youtube/v3/subscriptions?part=snippet\ +&channelId=UCPcTWaLV8zwx4WP4QExHj4Q&maxResults=50&key=$KEY" \ + | python3 -c 'import json,sys; d=json.load(sys.stdin); \ + print(d.get("pageInfo")); \ + [print(i["snippet"]["resourceId"]["channelId"], i["snippet"]["title"]) \ + for i in d.get("items",[])]' +``` + +A 200 with `items[]` confirms the design on the real account. A 403 means the privacy checkbox has +not taken effect — and per the finding above, there is no way around it. + +**This call also produces the number that sets `subsync_max_new`** (§4.4): `pageInfo.totalResults` +is how many channels he is subscribed to today, which is the one input the cap needs and which +cannot be discovered any other way. Record it here when known: + +> `totalResults` = **TBD — Phase 0** + +### 4.2 Fallback if `channelId` turns out not to work + +Ranked, if and only if the day-one curl fails. Note that all three require him to do something — +there is no silent workaround, because the public HTML route is gone. + +1. **OAuth with `mine=true`.** Robust and privacy-setting-independent, but carries a trap: + confirmed in Google's OAuth 2.0 docs, a project whose consent screen publishing status is + **"Testing"** is *"issued a refresh token expiring in 7 days"* unless the only scopes are basic + profile ones. `youtube.readonly` is not, so the token would die weekly. Avoiding that means + moving the app to "In production", which for a sensitive scope means Google verification — a + disproportionate amount of process for one brother. Only do this if option 3 is unacceptable. +2. **Google Takeout subscription CSV**, dropped into the admin UI. Manual, but a 30-second job + whenever he adds channels, and zero moving parts. +3. **Just tell him to use the admin UI.** It already exists and it is one text box. + +Design `subsync.py` so the source of a channel list is pluggable (§4.3 makes this fall out +naturally), so swapping between these is a small change rather than a rewrite. + +### 4.3 One source of truth — the sync is authoritative + +His subscription list *is* the subscription list. So the model is simply "make the DB match the API +response", and it needs no reconciliation machinery: + +- A channel in his list is subscribed. A channel that leaves it is unsubscribed, subject to §4.4. +- Two columns on `channel` carry what the sync needs: `source` (provenance) and `missing_syncs` (the + §4.4 counter). No join table. + +An earlier draft modelled this as multi-source *claims*, so that a manually-added channel could not +be deleted by someone else's list. With one source that machinery is dead weight, and speculative +generality in the part of the system that deletes things is the wrong place to spend it. If a second +account is ever mirrored, `source` is already there to key on and the claims model can come back +then. + +**One escape hatch survives:** `source = 'manual'` marks a channel the sync will never remove. It is +for pinning something during debugging, not a workflow, and it is not exposed as "subscribe to a +channel" in the UI — the way to add a channel is to subscribe to it on YouTube. + +**The cost of this choice, stated plainly:** if he unsubscribes from a channel, its series stops +updating and disappears from the library view. That is the correct behaviour for a mirror, and it is +exactly why §4.4 refuses to act on a single bad response. + +### 4.4 Removals must be slow and loud; additions must be capped + +Two failure modes here are genuinely destructive, and both are cheap to defend against. + +**Runaway additions.** If he has 400 subscriptions, the first sync queues 400 channels — at the §5 +bound that is 120,000 Jellyfin episodes, which will not end well. So: + +- A sync that would add more than **`subsync_max_new`** channels adds none of them. It records them + as `pending_approval`, alerts, and waits for a click in the admin UI. +- The cap applies per sync run, so ordinary drip-feed additions never trip it. + +**Separate the first sync from steady state.** Otherwise the cap always trips on day one, whatever +it is set to, and the guard trains everyone to ignore it. So a source's **first** sync is an +explicit bulk import: the admin UI shows the whole list with per-channel checkboxes and a count of +the episodes it implies, and nothing is subscribed until someone confirms. From the second sync +onwards the cap is a runaway guard, and `subsync_max_new` is set from the day-one `totalResults` +(§4.1) — a sensible rule is **max(10, ceil(totalResults × 0.2))**, so a genuine burst of activity +gets through but an order-of-magnitude jump does not. Provisional default **25** until Phase 0 +produces the real number. + +**Runaway removals.** A transient 403, a network blip, or him re-ticking the privacy box all look +like "he unsubscribed from everything". Deleting 20 channels' worth of Jellyfin metadata on that +basis would be unrecoverable in any pleasant way. So: + +- **A 403, a 5xx, a timeout, or a zero-item 200 is never treated as a removal.** It increments the + source's failure counter, alerts, and changes nothing. A genuinely empty list is + indistinguishable from a broken one in consequence, and we prefer the harmless reading. +- A channel missing from an otherwise-healthy response increments `missing_syncs`. Only at + **`subsync_missing_threshold`** (default **3** consecutive syncs, so ~3 hours) is it unsubscribed. +- Unsubscribing **does not delete anything on disk**. It sets the channel `inactive`: polling stops, + the tree stays, and re-subscribing on YouTube restores it with no refetching. Actual deletion is a + separate, explicit, human-initiated admin action. `.strm` files cost ~50 bytes each; there is no + pressure to reclaim anything, so there is no reason to ever delete automatically. +- `source = 'manual'` channels are exempt from all of the above. + +**Ordering:** the sync runs *before* the poll in the same run, so a channel added at 14:00 has its +catalogue built in the same pass. + +### 4.5 Alerting + +Existing convention on susan is `runitor` + healthchecks.io at `hc.jihakuz.xyz`, one UUID per job. +Follow it. A sync that hits 403, trips the add cap, or accumulates failures must exit non-zero so +the check goes red — a silently-broken subscription mirror is the worst possible outcome, because +nothing appears to be wrong until someone asks why a channel never showed up. + +### 4.6 The brother gets admin access, so auth becomes multi-user + +This is the right call — the approval queue (§4.4) and the "channel went missing" alerts are about +*his* subscriptions, so they should be his to action rather than landing on Tom every time. But it +does mean a change: `youtube-automate` has exactly one credential, `admin_password_hash` in the +`setting` table. Sharing it would mean neither party can have their access revoked or their password +changed independently, and the approval log would not record who approved what. + +So replace the single setting with a minimal `user` table (§7): `username`, `password_hash` +(scrypt, same parameters), `created_at`, `last_login_at`. Two rows. **No roles, no permissions +system** — both accounts can do everything, which is correct for two brothers and one media server, +and the alternative is a permission model nobody will maintain. + +Everything else in `web/auth.py` carries over unchanged, and it is adequate for an +internet-facing login: scrypt hashing and per-address failed-login throttling that reads +`X-Forwarded-For` (which nginx must set — it already does, see §12 step 4). + +Record `approved_by` on `pending_approval` resolutions. It costs one column and it answers "why is +this channel here" six months later. + +--- + +## 5. Scale — bounded backfill, unbounded growth + +Measured during the PoC: **Pitch Side has 1,249 videos.** The `youtube-automate` DB holds 15 of +them, because a 9-day retention window and 1.3 TB of free disk is what bounded it. `.strm` files +remove that bound entirely, so the bound is now a policy choice, and the choice is: + +> **Backfill 3 months or 300 videos per channel, whichever comes first.** + +Both settings exist (`backfill_max_days` = 90, `backfill_max_videos` = 300) and whichever is reached +first stops the walk. The two are well matched in practice: a channel uploading 2–3 times a day — +which is what Pitch Side's episode numbering implies — produces roughly 270 videos in 90 days, so +neither bound dominates. A weekly uploader gets ~13 videos and is bounded by time; a daily-podcast +firehose gets 300 and is bounded by count. That is the right behaviour in both directions. + +**This bounds the *initial* build, not the library.** Nothing ages out — `.strm` files cost ~50 +bytes and there is no disk to reclaim, so aging out would destroy metadata for no gain. The +consequence is honest and worth stating: the library **grows forever from the subscribe date**. At +20 channels averaging 2 uploads/day that is ~15,000 new episodes a year, so the 20,000-episode +question in point 1 below is deferred by about eighteen months, not answered. The lever, if it ever +bites: a `hidden` video state that removes the `.strm` and `.nfo` from disk while keeping the DB row, +so the catalogue stays re-materialisable. **Do not build it now** — build the measurement that tells +us when we need it. + +**Initial build:** ~20 channels × ≤300 = **≤6,000 episodes, ≤12,000 files.** Comfortable. + +1. **Jellyfin scan cost is still unmeasured**, and is the number that decides everything above. + Every episode is a `.strm` + a `.nfo` + a thumbnail. Normal scans were measured to make **0 media + probes** (FINDINGS §6), which is the thing that makes this viable at all — but stat-ing and + NFO-parsing 12,000 files on a Westmere with the library DB on spinning disk is its own cost. + **Measure in Phase 3** by building one channel to the bound, timing a full scan, and recording + seconds-per-1,000-episodes here. That single figure sizes both the initial build and the growth + runway. +2. **`replaceAllMetadata` remains a catastrophe rather than an annoyance.** It is the one operation + verified to probe media, and at 6,000 items it means 6,000 cold starts. The cold-start rate + limiter (20/hour) contains the damage to YouTube's side, but the library-side result is 6,000 + items whose metadata got wiped and not re-derived. Defences, all of them: + - the proxy 503s past the budget (built, tested — `test_limits.py`) + - `ytstream` never itself issues a refresh with `replaceAllMetadata=true` + - the admin UI documents "do not click Replace all metadata" in the place where someone would be + tempted to + - **NFOs are the source of truth**, so recovery is a re-materialise pass over the tree, not a + re-fetch from YouTube. Make sure that pass exists and is one CLI command. +3. **Episode numbering holds.** `MMDD*10 + ordinal` clamps at 10 uploads/channel/day, computed + against the DB rather than the batch, so it is stable across a 300-video backfill. Season = + upload year means a 3-month window usually spans one season and occasionally two (a January + subscribe reaches back into the previous year), which is correct and needs no special handling. +4. **The `.strm` URL is baked into every file.** Changing the proxy's host or port means rewriting + all of them. Cheap (it is a tree walk) but it must be a supported CLI command, not a `sed` + someone invents under pressure. Settle the URL now: `http://127.0.0.1:8099/watch/<id>`. Jellyfin + is the only client that ever reads it, it runs on the same host, and 127.0.0.1 means the proxy is + unreachable from the network by construction. +5. **Backfill must be resumable.** Quota is a non-issue but a crash halfway through 12,000 files + needs to resume, not restart. Track progress per channel (`channel.backfill_cursor`), commit per + page of 50. + +--- + +## 6. Components + +Four processes. Two are long-running services, two are cron jobs. + +### 6.1 `ytstream-proxy` — systemd service + +The PoC, productionised. `/home/susan/ytstream/ytstream.py`, 31 KB, currently running by hand. + +Already built and verified: `/watch/<id>` and `/healthz`, single extraction reused by both +yt-dlp legs via `--load-info-json`, FIFO→ffmpeg `-c copy` fMP4 on tmpfs, correct HTTP range +handling (25 assertions), LRU cache, concurrency cap, retry-on-403, cold-start rate limiter +(18 assertions), counters. + +Outstanding before it is a service: + +- **Install the unit.** `ytstream.service` is written but not installed; susan has **no + passwordless sudo**, so this goes in `deploy/deploy.sh` for the operator to run once. The unit + pins `Environment=PATH=/var/lib/ytstream/venv/bin:...` (load-bearing — the venv yt-dlp is the + only one with the POT plugin), `User=susan Group=mediaserver UMask=0002`, and + `Requires=docker.service` for the POT provider container. +- **Startup sweep of the work dir.** Sessions live in memory; a restart orphans whatever is in + `/dev/shm/ytstream`. Wipe it on start. +- **Prove LRU eviction.** It has never fired in testing. Add a test with a tiny `--cache-gb`. +- **Decide the tmpfs budget.** `/dev/shm` is 24 GB of 47 GB RAM; throughput measured at ~480 + MB/hour of video. `--cache-gb 8` holds ~16 hours of content, which is ample. Do not raise it + without thinking about what else on susan wants RAM. +- **Log to journald** rather than a scratch file, and keep the access log behind a flag. + +**Unresolved, and staying that way:** the intermittent 403 has no known cause. Ruled out during the +hardening pass: rate limiting, `--load-info-json` round-tripping, concurrency, and — the +plausible-sounding one — missing PO tokens. Pinning `player_client=mweb` to get token-bearing URLs +measured **0 ok / 6 fail**, while token-less `ANDROID_VR` URLs work; had that "fix" shipped it would +have broken every playback. Retry with a fresh extraction is the mitigation, it works, and it fires +roughly once per dozen cold starts. Keep the `retried` counter visible so a change in that rate is +noticeable. + +### 6.2 `ytstream-admin` — systemd service + +Lifted from `youtube_automate/web/`. scrypt password hashes, now per user in the `user` table (§4.6) +rather than one shared setting, per-address failed-login throttling reading `X-Forwarded-For`, behind +nginx on `127.0.0.1:8086` (8085 is `youtube-automate`'s during the overlap). + +New routes beyond what exists: subscription **sources** (add/remove a mirrored YouTube account, show +its last sync, its failure count, its `pending_approval` queue with an approve/reject action), the +first-sync bulk-import screen, per-channel backfill-depth controls, account management for the two +users, and the API key field. + +**Two people use this**, so the copy matters more than it did with a single operator: the +"do not click Replace all metadata" warning (§5) has to be where someone would be tempted, and the +approval screen has to show what a channel implies — an episode count — before it is approved. + +### 6.3 `ytstream run` — hourly cron + +`sync sources → poll feeds → materialise → refresh Jellyfin`. Under `flock` on +`/var/lib/ytstream/run.lock`, under `runitor` with its own healthchecks UUID, via +`sg mediaserver "..."` to match the existing convention. + +### 6.4 `ytstream backfill` — manual / one-off per channel + +The expensive path (§3, §5). Deliberately not on a timer: it runs when a channel is first +subscribed, and when someone explicitly asks for a deeper catalogue. + +--- + +## 7. Data model + +`/var/lib/ytstream/ytstream.db`. Start from `youtube-automate`'s schema — `channel`, `video`, +`setting` are the right shape — with these changes: + +**`channel`** — add `backfill_cursor TEXT` (resumable backfill), `backfill_max_videos INTEGER` and +`backfill_max_days INTEGER` (per-channel overrides of the §5 bounds; NULL = use the global setting), +`active INTEGER NOT NULL DEFAULT 1` (§4.4 soft-delete), `uploads_playlist TEXT` (whether UULF or UU +won, §3). Drop `retention_days`. + +**`video`** — drop `size_bytes`, `downloaded_at`, `deleted_at`, `attempts`, `last_error`; none of +them mean anything when there is no download. Keep `duration` (now from `videos.list`, and it feeds +`<durationinseconds>`). `state` collapses to `{listed, materialised, skipped_short, skipped_live, +hidden}` — no `pending`/`downloading`/`failed`, because materialising a `.strm` cannot +meaningfully fail. Add `published_at TEXT` (exact RFC-3339 from the API) alongside the existing +date-only `upload_date` that naming uses. + +**`channel`, continued** — two more columns instead of the join table an earlier draft had (§4.3): +`source TEXT NOT NULL` (`'youtube'` for synced, `'manual'` for the pinning escape hatch) and +`missing_syncs INTEGER NOT NULL DEFAULT 0` (the §4.4 counter). + +**`source`** — new, and in practice **exactly one row**: `key`, `label`, `channel_id`, `enabled`, +`last_sync_at`, `last_sync_ok`, `consecutive_failures`, `last_error`. A one-row table rather than six +`setting` keys, because these are fields of one thing and they change together. + +**`pending_approval`** — new, the §4.4 add-cap and first-sync import queue: `source`, `channel_id`, +`title`, `seen_at`, `resolved_at`, `resolution` (`approved`/`rejected`), `approved_by`. + +**`user`** — new (§4.6), replacing the single `admin_password_hash` setting: `username` (PK), +`password_hash` (scrypt, same parameters as `youtube-automate`), `created_at`, `last_login_at`. Two +rows, no roles. + +**`setting`** — carry over what still applies (`jellyfin_url`, `jellyfin_api_key`, +`pot_provider_url`, `max_height`, `min_duration_seconds`, `session_secret`, `last_run_at`). Drop +`retention_days`, `disk_cap_gb`, `write_subs`, `sub_langs`, `sponsorblock_mark`, `max_attempts`, +`backfill_days`, `admin_password_hash` — download-era concepts plus the credential that moved to +`user`. Add `youtube_api_key`, `subsync_max_new` (25, provisional — §4.4), +`subsync_missing_threshold` (3), `proxy_base_url` (`http://127.0.0.1:8099`), `backfill_max_videos` +(**300**), `backfill_max_days` (**90**). + +**No migration from `subs.db`.** It holds 2 channels and 18 videos. Re-subscribe by hand and let +the backfill do the rest; a migration script would be more code than the data is worth. + +--- + +## 8. On-disk layout and the `.strm` contract + +Media root: **`/disks/Plex/_ytstream/`** — already created and already wired to the *YouTube +(stream)* Jellyfin library, with 2 channels in it from the PoC. Keep it. (The name has a leading +underscore matching `_ingest`/`_cache` on that volume; `/disks/Plex/YouTube` stays with +`youtube-automate` until §12 retires it.) + +``` +/disks/Plex/_ytstream/ +└── Pitch Side/ + ├── tvshow.nfo + ├── poster.jpg # channel avatar + ├── fanart.jpg # channel banner + └── Season 2026/ + ├── Pitch Side - S2026E8120 - Title [dQw4w9WgXcQ].strm + ├── Pitch Side - S2026E8120 - Title [dQw4w9WgXcQ].nfo + └── Pitch Side - S2026E8120 - Title [dQw4w9WgXcQ]-thumb.jpg +``` + +Naming, sanitisation and `dir_name` stability rules are `youtube-automate`'s, unchanged +(specs.md §5) — identical rules in both trees is what makes the overlap period sane. + +**`.strm` contents:** one line, no trailing newline required: + +``` +http://127.0.0.1:8099/watch/dQw4w9WgXcQ +``` + +**NFO:** `episodedetails` with `title`, `season`, `episode`, `aired`, `plot`, `runtime`, +`durationinseconds`, `uniqueid type="youtube"`. **No `<fileinfo><streamdetails>`** — pre-seeding it +was measured to change nothing about whether Jellyfin probes (FINDINGS §6), so it is dead weight. + +**No `.work/` dir under the media root.** The proxy's scratch space is `/dev/shm/ytstream`, outside +the library entirely, which is strictly better than the dot-dir-plus-`.ignore` belt-and-braces that +`youtube-automate` needs. + +--- + +## 9. Environment and paths + +Verified on the machine, 2026-08-12. + +| Purpose | Path | +|---|---| +| Source (bare repo) | `/disks/git-repos/ytstream.git` — **exists but is not yet `git init --bare`** | +| Checkout | `/opt/ytstream` — exists, `susan:automation` `0770` | +| Entry point | `/usr/local/bin/ytstream` (needs root, → `deploy.sh`) | +| Virtualenv | `/var/lib/ytstream/venv` | +| State DB | `/var/lib/ytstream/ytstream.db` | +| Lock file | `/var/lib/ytstream/run.lock` | +| Media root | `/disks/Plex/_ytstream/` | +| Proxy scratch | `/dev/shm/ytstream` (tmpfs, 24 GB available) | +| Admin UI | `127.0.0.1:8086`, nginx → `tube.jihakuz.xyz` (§12) | +| Proxy | `127.0.0.1:8099`, not exposed | + +Facts that constrain the design: + +- **Ownership:** the media tree convention is `susan:mediaserver`, dirs `0770`, files `0664`. + `/disks/Plex` is `0770` with no setgid; Jellyfin reaches the tree only through its `mediaserver` + supplementary group. Both units run `User=susan Group=mediaserver UMask=0002`; cron goes through + `sg mediaserver`. `/disks/Plex/_ytstream` already has the setgid bit — keep it. +- **No passwordless sudo.** Everything touching `/usr/local/bin`, `/etc/systemd/system` or + `/etc/nginx` belongs in `deploy/deploy.sh`. +- **yt-dlp:** `/usr/local/bin/yt-dlp` is a **2023.11.16** binary, far too old to work. Nothing + references it; leave it alone. ytstream gets its own venv with `yt-dlp[default]` (the extra that + ships `yt-dlp-ejs`, mandatory for JS challenge solving), `bgutil-ytdlp-pot-provider==1.3.1` + matching the container tag, and `curl-cffi<0.16`. Note that during the overlap **two** venvs will + hold yt-dlp; keep both current, and do not let a bare `yt-dlp` on `PATH` be what either service + resolves. +- **POT provider:** container on `127.0.0.1:4416`, `--restart unless-stopped`, Docker enabled at + boot. Shared with `youtube-automate` during the overlap and inherited afterwards. It is the + reason the proxy unit declares `Requires=docker.service`. +- **Client selection is settled — do not re-litigate it.** `CLIENT_ARGS = + "youtube:player_client=default"`. `mweb` is deliberately absent: its formats 403 on every + attempt, and it was the sole source of the DRC and dubbed-language variants behind both original + format-picker bugs. `web`/`ios`/`web_safari`/`tv` are SABR-only and yield nothing usable. Left as + `default` rather than pinned to `android_vr` so a yt-dlp update can follow YouTube. +- **Jellyfin 10.11.4**, native systemd, `0.0.0.0:8096`, API key already in the DB. + +--- + +## 10. Playback UX — state it plainly + +Measured: wait-for-complete is ready in **20 s** for a 21-minute video and **~55 s** for a +46-minute one, at ~8 MB/s (~60× realtime). `--growing` mode gives TTFB of **6.5–7.1 s** but ffmpeg +writes `mvhd.duration=0` and no `mehd` box regardless of what it knows about the input — verified, +including an attempt to patch the boxes afterwards, which ffmpeg ignores — so the player sees a +video of unknown, growing length. Seeking and progress bars misbehave. + +**Ship wait-for-complete.** A ~20–55 s spinner that then behaves like a normal file beats instant +playback with a broken timeline. Keep `--growing` as a flag for experimentation. Revisit only if +someone complains, and if they do, the fix is a pre-warm on library browse rather than a change to +the mux. + +--- + +## 11. Configuration and secrets + +Secrets live in `/var/lib/ytstream/ytstream.db`, mode `0640` `susan:automation`, and are entered +through the admin UI. Never in the repo, never in a systemd unit, never in a cron line. This matches +`youtube-automate`, where `jellyfin_api_key` and `session_secret` already live that way. + +Three kinds: `youtube_api_key` (new) and `jellyfin_api_key` in `setting`; `session_secret` in +`setting`; and two scrypt password hashes in `user` (§4.6). + +The YouTube API key is restricted to the YouTube Data API v3 and has read-only reach over public +data. Worst case on leak is quota exhaustion; rotation is a two-minute job in the console. + +--- + +## 12. Decommissioning `youtube-automate` + +Not until §13 phase 5 passes. Order matters — the point is that every step is reversible until the +last one. + +1. **Stop new work.** Comment out the `17 * * * *` cron entry. Leave the service running so the + admin UI still answers. +2. **Watch for a week** with both libraries live in Jellyfin. This is the real acceptance test: + does anyone reach for the old library? +3. **Retire the Jellyfin library.** Remove *YouTube* (`/disks/Plex/YouTube`) via + `/Library/VirtualFolders`. Rename *YouTube (stream)* → *YouTube*. **Note from the PoC:** + deleting and recreating a library at the same path returns the **same** `ItemId` and reuses the + old items — if a clean slate is ever needed, use a fresh path, as `_ytstream` already is. +4. **Free the hostname.** `tube.jihakuz.xyz` is served by a leftover TubeArchivist `server` block + inside `sites-available/jihakuz.xyz`, which owns the Let's Encrypt cert and wins because nginx + takes the first matching block. Repoint it at 8086 the same way `deploy/fix-nginx-tube.sh` did + for 8085; do not install a competing vhost file. +5. **Disable the service.** `systemctl disable --now youtube-automate.service`; remove the unit. +6. **Reclaim the bytes.** Estimated 5–10 GB under `/disks/Plex/YouTube`. Delete only after step 2 + has actually elapsed. +7. **Keep, do not delete:** `/var/lib/youtube-automate/subs.db` (copy it aside — it is the only + record of what was subscribed and when), `/opt/youtube-automate` and its bare repo, and + `specs.md`, which remains the reference for every rule ytstream inherited. + +`update-ytdlp.sh` (Mondays 04:40) must be **repointed, not removed** — it becomes ytstream's, and it +is the thing that keeps playback working as YouTube changes. + +--- + +## 13. Build order + +Each phase ends in something checkable. Do not start the next one until it does. + +**Phase 0 — verify the assumptions the plan rests on, and get the one number it is missing.** +~15 minutes, before any code. `git init --bare /disks/git-repos/ytstream.git`. Get the API key +(§4.1). Then three curls: + +1. `subscriptions.list` on `UCPcTWaLV8zwx4WP4QExHj4Q` returns 200 with items — **and record + `pageInfo.totalResults`**, which is what sets `subsync_max_new` (§4.4). This is a hard blocker: + there is no fallback that does not involve him doing something (§4.2). +2. `playlistItems.list` accepts a UULF playlist id — or does not, and we take the documented `UU` + fallback plus duration filtering (§3). +3. `videos.list` returns `contentDetails.duration` for a batch of 50 ids. + +→ *Done when: all three answers are written back into this file as verified facts or corrections, +and `subsync_max_new` has a real value.* + +**Phase 1 — skeleton and lift.** Fork the tree, new package name, new DB path, new schema (§7), +lifted modules and their tests passing. No new behaviour. +→ *Done when: `pytest` is green and `ytstream doctor` reports a healthy environment.* + +**Phase 2 — the proxy as a service.** Move it in, split it up, add the startup sweep and the LRU +test, write `deploy/deploy.sh`, operator runs it. +→ *Done when: `systemctl status ytstream-proxy` is active after a reboot, `/healthz` answers, and +Jellyfin direct-plays a cold video end to end.* + +**Phase 3 — catalogue.** `api.py`, `strm.py`, the 3-month/300-video bounded resumable backfill, the +hourly run. Build **Pitch Side alone** and **time a Jellyfin scan** — the §5 measurement. Worth +doing twice, once at the 300-video bound and once unbounded at 1,249, since the second gives the +seconds-per-1,000-episodes figure that sizes the growth runway for free. +→ *Done when: the bounded episode count is visible with correct titles, dates, durations and +thumbnails; scan time per 1,000 episodes is recorded in §5; and one of them plays.* + +**Phase 4 — subscription sync.** `subsync.py`, the first-sync bulk import, the add cap, the +missing-threshold, the `user` table and two accounts (§4.6), the admin routes, the healthchecks UUID. +→ *Done when: the brother logs in with his own credentials and approves the first import; then he +subscribes to a new channel on YouTube and within an hour it is a series in Jellyfin with episodes +that play, nobody having touched the admin UI. Plus: a forced 403 and a forced empty response both +leave the DB untouched and turn the check red.* + +**Phase 5 — cut over.** §12 steps 1–2, run for a week. +→ *Done when: nothing has broken and nobody has used the old library.* + +**Phase 6 — decommission.** §12 steps 3–7. + +--- + +## 14. Decisions — answered 2026-08-12 + +1. **Account to mirror:** `@cflux1030` → `UCPcTWaLV8zwx4WP4QExHj4Q` ("C Flux"), resolved and + recorded in §4. **Sole source of truth** — Tom's own subscriptions are not part of this at all, + which removed the multi-source claim model from §4.3 and put the weight on §4.4 instead. +2. **Catalogue depth:** 3 months or 300 videos, whichever comes first. §5 rewritten around it, and + both bounds are settings with per-channel overrides. Note the consequence recorded there: this + bounds the initial build, not the library, which grows forward indefinitely. +3. **`subsync_max_new`:** to be set from `pageInfo.totalResults` on the day-one + `subscriptions.list` call, since that is the only way to learn how many channels he follows. + Provisional 25; rule of thumb `max(10, ceil(totalResults × 0.2))` (§4.4). **The one open number + in this plan** — Phase 0 closes it. +4. **Brother gets admin access:** yes, which turns the single shared password into a two-row `user` + table (§4.6) and adds `approved_by` to the approval log. + +--- + +## 15. Gotchas carried forward + +Things already paid for once. All of these are verified. + +- **A normal Jellyfin scan makes 0 media probes**, which is the single fact this design depends on. + `replaceAllMetadata=true` **does** probe. Pre-seeded `streamdetails` do not prevent it. +- **`--load-info-json` reuses the extraction** — confirmed twice, once during the original PoC and + again while hunting the 403. Both yt-dlp legs run from one `-J`. +- **ffmpeg will not write a duration into a fragmented MP4**, and will not accept one patched in + afterwards. Hence wait-for-complete (§10). +- **Two picker bugs, both silent, both from `mweb`:** `140-drc` was chosen over `140` on an `abr` + tie and 403'd; `140-0` was chosen on a 9-language video and produced German audio. The audio sort + key ranks `language_preference` **first**, deliberately above the DRC check — a 403 is a loud + failure, wrong-language audio is a silent one that would have shipped. +- **`row["title"]` ambiguity renamed every series once.** The channel/video join has `title` on both + sides. Pass show fields explicitly, never `row["title"]`. Caught only by accident. +- **Jellyfin's `/Items/{id}` needs user context** — use the `/Items?ids=` form. +- **`youtubetab:approximate_date` is wrong by up to 2 days.** Do not derive episode numbers from + it; use `playlistItems.list` (§3). +- **YouTube's public subscriptions tab is gone.** `/@handle/channels` returns 200 and silently + serves Home — four tabs, zero channel ids in `ytInitialData`. Verified 2026-08-12. Nobody should + spend an afternoon trying to scrape it; the API is the only route (§4). +- **`pkill -f 'ytstream.py'` kills the shell that runs it**, because the command string contains its + own pattern. Bracket it: `pkill -f 'ytstrea[m].py'`.