tom/ytstream
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Plan ytstream as a service, and mirror a YouTube account's subscriptions

Browse Source 
The streaming proof of concept works — FINDINGS.md records the measurements —
but it runs by hand out of a home directory. This is the plan for making it a
real service, plus the design for the new requirement: my brother subscribes to
a channel on YouTube and it appears in Jellyfin without anyone touching an
admin page.

Three things drove the shape of the plan.

It is a fork of youtube-automate, not a green-field build. That project is 3,543
lines and most of it — RSS discovery, episode numbering, NFO schema, scrypt auth,
the admin UI — has nothing to do with downloading and was verified on this
machine on 2026-08-11. Only download.py and reap.py actually die. Re-deriving the
rest would mean re-earning knowledge we already paid for.

Getting a Google API key for the subscription feature turns out to pay for
itself twice over, so the plan leans on it much harder than the feature needs.
subscriptions.list, playlistItems.list and videos.list are 1 unit each against
10,000/day, which means the whole metadata path can move off yt-dlp: exact upload
dates from contentDetails.videoPublishedAt instead of approximate_date (measured
wrong by up to 2 days, and episode numbers are derived from it), durations
without an extraction per video, and no residential-IP request storm when a
channel is added. yt-dlp is then only ever invoked by the proxy, for one video,
because a human pressed play. That is a much better boundary than the one we
have.

The brother's channel is the sole source of truth. An earlier draft modelled
subscriptions as multi-source claims so a manually-added channel could not be
deleted by someone else's list; with one source that is dead weight, and
speculative generality in the code path that deletes things is the wrong place to
spend it. What survives instead is paranoia about removals, which now matter more
rather than less: a 403, a timeout or an empty response is never read as an
unsubscribe, a channel must be absent from three consecutive healthy syncs, and
even then the tree stays on disk and the channel merely goes inactive.

Verified while writing this, rather than assumed:

  - subscriptions.list accepts a channelId filter and is not documented as
    needing an authorized request, so an API key is enough. It returns 403 —
    not an empty list — when subscriptions are private, which is what makes the
    removal safeguards able to tell "he made them private again" apart from "he
    unsubscribed from everything".
  - There is no scraping fallback. /@cflux1030/channels returns 200 but silently
    serves the Home tab: four tabs, zero channel ids in ytInitialData. YouTube
    retired the public subscriptions tab, so the privacy checkbox is mandatory
    rather than merely convenient.
  - @cflux1030 resolves to UCPcTWaLV8zwx4WP4QExHj4Q, so no channels.list call is
    needed at build time.
  - OAuth as a fallback carries a trap worth writing down: a consent screen in
    "Testing" status issues refresh tokens that expire in 7 days, and
    youtube.readonly is not one of the exempt basic scopes.

Catalogue depth is bounded at 3 months or 300 videos per channel, whichever
comes first. The plan says plainly that this bounds the initial build and not the
library, which grows forward indefinitely — roughly 15,000 episodes a year at
20 channels — so the Jellyfin scan-cost measurement in Phase 3 is what tells us
when that becomes a problem. Deliberately not solved now.

One number is still open: subsync_max_new depends on how many channels he
actually follows, which only the day-one API call can tell us. Phase 0 closes it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Tom Flux
2026-08-12 15:26:40 +01:00
co-authored by Claude Opus 5
commit 8cca4fe1e2
3 changed files with 1200 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
FINDINGS.md
+454
View File
@@ -0,0 +1,454 @@
# `ytstream` — results on susan
Date: 2026-08-12. Answers every open question in the `ytstream` handover: §3, §6,
and §7 steps 1–4. Both Jellyfin gates pass (§6 below). §7 step 5 (integrating
with youtube-automate) is **not** done and `/opt/youtube-automate` is unmodified.
Environment note that matters: the PoC calls bare `yt-dlp`. On susan, `PATH`
resolves that to `/usr/local/bin/yt-dlp`, which is **2023.11.16** and has no POT
plugin. The working install is the automation's venv:
```
export PATH=/var/lib/youtube-automate/venv/bin:$PATH # yt-dlp 2026.07.04
# + bgutil-ytdlp-pot-provider 1.3.1
```
That venv's plugin matches the running `bgutil-provider` container (1.3.1) and
retrieves gvs PO tokens correctly. Before this was noticed the container had been
up 15 hours with nothing able to talk to it — no yt-dlp plugin directory existed.
---
## 1. Verified against real videos
Four videos, 2 to 46 minutes. Two are from the live `subs.db`, so they are
representative of what the automation actually handles.
| video | length | TTFB (`--growing`) | ready (default mode) | duration served | truth |
|---|---|---|---|---|---|
| `D6ocsyQLKy8` | 2 min | 7.1s | ~7s | 120.62s | 121s |
| `dQw4w9WgXcQ` | 3.5 min | 7.1s | ~9s | 213.13s | 213s |
| `J9O3sxoMs5U` | 21 min | 6.9s | **20s measured** | 1259.47s | 1259s |
| `NH2MhBFQm9w` | 46 min | 6.5s | ~50s | 2789.45s | 2789s |
Full `ffmpeg -f null -` decode exits 0 in every case, with both system ffmpeg
5.1.7 and Jellyfin's bundled 7.1.3. Codecs are always `h264 1280x720` + `aac`.
"ready" figures other than the 21-minute one are derived from the measured
~8 MB/s pull rate, not separately timed.
**`--load-info-json` reuses the extraction.** This was the load-bearing unknown.
Verbose output shows zero re-extraction markers and no second POT fetch — it goes
straight to `[info] Downloading 1 format(s): 136`. The one-extraction design holds.
**Concurrency is not a problem.** 5/5 clean trials of the exact PoC pattern (one
fresh extraction, two concurrent fetches). §3's worry about the POT provider under
concurrent access looks unfounded.
**Throughput ~8 MB/s**, roughly 60× realtime. 368 MB for 46 minutes of 720p avc1
≈ **480 MB/hour**, about half the handover's 1 GB/hour estimate.
---
## 2. Two bugs found in the format picker — both would have shipped
Both came from the same root cause: YouTube advertises variants with **identical
`abr`**, so `max()` tie-broke on list order.
| video | picked | consequence |
|---|---|---|
| `NH2MhBFQm9w` | `140-drc` | HTTP 403 — DRC variants are advertised but not served |
| `J9O3sxoMs5U` | `140-0` | **German audio on an English podcast** (9-language upload) |
The second is the dangerous one: silent, and it would have played.
Fixed by ranking `language_preference` first (original track = 10, dubs = -1),
then non-DRC, then codec, then bitrate. Language correctness deliberately outranks
the DRC check: a 403 is a loud failure, wrong-language audio is a silent one.
`pick_formats` now also logs the chosen language and warns if a non-original track
is selected on a multi-language upload.
Verified: `140-drc` and `140-0` reproducibly 403 while `140`, `140-7`, `139`,
`251` and `136` all download fine from the same `info.json`.
**Later correction — both bugs had a common upstream cause.** The DRC and dubbed
variants existed *only* because `mweb` was in `player_client`. With
`player_client=default` alone, extraction returns four audio formats (139, 249,
140, 251), no DRC variants and no language variants at all — even on the
nine-language video. So "DRC formats 403" was really "mweb formats 403", and the
German-audio bug could only ever have happened via mweb. `mweb` is now dropped
(§9). The `language_preference` and DRC rules stay in the picker as cheap
insurance if a yt-dlp update changes what `default` returns.
---
## 3. Two defects not mentioned in the handover — both now fixed
### Duration collapses while the file grows
Jellyfin's own ffprobe 7.1.3 on the 46-minute video:
```
during growth : format.duration = 5.92 <- wrong
after complete: format.duration = 2789.45 <- correct
```
§5 says scrubbing won't work while growing. It is worse than that: the **duration
itself** is wrong, which would break Jellyfin's runtime display, seek bar, resume
point and watched state.
This cannot be fixed at the container level. Both attempts failed:
- Patching `mvhd.duration` in place — ignored.
- Inserting an `mehd` box with the correct `fragment_duration` — the box is
written correctly (`mehd v0 fragment_duration=60000`, file still decodes clean),
and ffmpeg **still** ignores it and recomputes from the fragments present.
Also confirmed: ffmpeg writes `mvhd.duration=0` and **no `mehd`** regardless of
whether it knows the input duration — a mux from regular files and one from FIFOs
produce byte-identical headers. `mp4boxes.py` and `mehd_patch.py` in this
directory reproduce both results.
**Fix:** waiting for the mux is now the default. Correct duration, `Accept-Ranges`
and working seeks on the very first request. Costs TTFB — 20s for a 21-minute
video instead of 6.9s. `--growing` keeps the old low-latency behaviour and now
prints a warning. Given yt-dlp runs at ~60× realtime the wait is bounded and
predictable, and it is the same order of magnitude as the 30-second non-seekable
window §5 already accepted.
The Jellyfin gate has since settled this: Jellyfin takes its runtime **only** from
its own probe of the stream, never from the NFO (§6). So `--growing` would hand it
~6 seconds and there is no way to claw the TTFB back. Wait-for-complete is
mandatory, not a preference.
### Range end was ignored
`bytes=1000000-1000999` returned **367 MB** instead of 1000 bytes — the server
parsed only the start offset and always streamed to EOF. This directly undermines
§6's mitigation #2, which depends on serving small bounded probe ranges.
Now parses both ends, honours `bytes=N-M`, `bytes=N-` and the suffix form
`bytes=-N`, clamps an end past EOF, and returns `416` with `Content-Range: */total`
for an unsatisfiable start. `test_range.py` covers all of it — 25 assertions,
no network required.
---
## 4. One correction to §6's premise
The `<fileinfo><streamdetails>` blocks in the existing episode NFOs were written
by **Jellyfin**, not by the automation — `<lockdata>`, `<dateadded>`, `micodec`
and absolute-path `<art>` are Jellyfin's own NFO output, produced after it probed
the real `.mp4` files.
So there was no evidence that pre-seeding `streamdetails` suppresses probing —
mitigation #1 was **untested**, not partly proven. It has since been tested with a
control group and it does nothing at all (§6).
---
## 5. Intermittent 403s — found, and now retried
Two 403s were observed across roughly ten real runs, and they **moved between
formats** — once on video `136`, once on audio `140`, on the same video that had
succeeded minutes earlier.
The first guess was transient rate limiting, because the failures clustered after
~1.5 GB had been pulled from one video. **That guess is wrong.** A retry two
seconds later with a fresh extraction succeeded every single time, and no rate
limit clears in two seconds. See §9 for what was actually measured.
What matters is the consequence: a single 403 on either stream killed the whole
playback with no recovery. It then struck again during the Jellyfin test itself,
so it is now **fixed in `ytstream.py`**: up to `--max-retries` attempts (default
2), each with a fresh extraction, retried only while nothing has been served yet.
That recovered every occurrence seen — including the one during the gate, where
attempt 1 failed and attempt 2 succeeded.
Retry is a mitigation, not a root-cause fix. `ytstream_poc.py` still has no retry;
it is kept as the record of what was originally verified.
---
## 6. Jellyfin: both gates PASS
§7 step 3 (playback) and §6 (scan probing) are now answered against the live
Jellyfin 10.11.4 on susan, using the two channels already subscribed in
`subs.db` — 7 episodes, 2 shows, real metadata.
Library `YouTube (stream)` → `/disks/Plex/_ytstream`, built by
`make_strm_tree.py`. `SaveLocalMetadata=False` on purpose: the real YouTube
library has it **True**, which is why Jellyfin rewrote those NFOs, and leaving it
on here would have overwritten the test NFOs and destroyed the control group.
### §6 scan-time probing — the open risk does not materialise on a normal scan
| operation | probes | pipelines started |
|---|---|---|
| first scan of a new library | **0** | 0 |
| repeat scan, `replaceAllMetadata=false` | **0** | 0 |
| refresh with `replaceAllMetadata=true` | **yes, immediately** | would be all of them |
A normal scan ingests all 7 `.strm` episodes — correct show, season, episode
numbers, aired dates — **without a single request to the proxy**. No scan storm.
Episode `RunTimeTicks` stays null until something plays the item.
The dangerous path is a **full metadata replace**, which a user can trigger from
the UI. Jellyfin probes the `.strm` target immediately. Two corrections to §6's
assumptions while doing so:
- Probes are **open-ended `Range: bytes=0-`**, not the "small ranges from offset
0" §6 assumed. You cannot identify a probe from its Range header, so
mitigation #2 cannot key off request shape. It can still work by serving the
first few MB from a permanent cache — the probe takes what it needs and hangs
up — but the discriminator has to be something else.
- **Refusing a probe causes a retry storm.** Returning 503 produced 9 retries in
~2 seconds from `Lavf/61.7.100`, after which the refresh aborted and left six
of seven episodes with their episode numbers wiped. Whatever the mitigation is,
it must answer, not refuse.
`--max-pipelines` (default 2) is the backstop: a full replace throttles to two
concurrent fetches instead of seven.
### §6 mitigation #1 — pre-seeded streamdetails do nothing
The `alternate` control group settles it. Episodes carrying full
`<fileinfo><streamdetails>` with `<durationinseconds>` behaved **identically** to
those without: `RunTimeTicks` null after a scan in both groups. Jellyfin took
`ProviderIds` and `PremiereDate` from the same NFOs, so they were definitely
being read — it simply does not use NFO stream details or runtime for a `.strm`
item. Mitigation #1 is a dead end, and it is also unnecessary given a normal scan
does not probe at all.
This also settles the `--growing` question from §3: Jellyfin has **no** duration
until it probes the stream itself, so a growing file would hand it ~6 seconds.
**Stay in the default wait-for-complete mode.**
### §7 step 3 playback — passes, and direct-plays
Driven through Jellyfin's own API (`PlaybackInfo`, then `/Videos/{id}/stream`):
| | warm cache | cold, 46-minute episode |
|---|---|---|
| `PlaybackInfo` latency | 0s | **55s — Jellyfin waited, no timeout** |
| `Container` | `mp4` | `mp4` |
| `RunTimeTicks` | 120.62s (truth 120.62) | **2789.45s (truth 2789)** |
| `SupportsDirectPlay` | **True** | **True** |
| streams | h264 1280x720 + aac | h264 1280x720 + aac |
Fetching the episode through Jellyfin's streaming endpoint returned 10,654,544
bytes **byte-identical** to what the proxy produced, decoding cleanly. So
Jellyfin direct-plays it with no transcode — which was the whole point of
preferring avc1+mp4a.
Two things worth knowing:
- A cold 46-minute episode makes the user wait ~55s at the play button. That is
the price of correct duration and seeking. It is bounded and predictable.
- One ffprobe of a 368 MB file generated **481 range requests** to the proxy.
All local and cheap, but it makes the access log noisy and is worth a look if
throughput ever matters.
### Artwork works, and costs nothing extra
Episode thumbnails need **no API call**: the URL is derivable from the video id,
which is already in the filename. `add_thumbnails.py` fetches
`i.ytimg.com/vi/<id>/maxresdefault.jpg` (~200 KB, 1280x720) with `hqdefault.jpg`
(~22 KB) as a fallback, writing `<basename>-thumb.jpg`. Channel avatar and banner
become `poster.jpg` / `fanart.jpg` and do need one yt-dlp call per channel.
Verified in Jellyfin: all 7 episodes carry a 16:9 `Primary` image, both series
have a poster and a backdrop, and an image refresh triggers **0 probes**. Artwork
is entirely independent of the streaming path.
Two things learned while doing it:
- **Jellyfin re-reads an NFO when its mtime changes.** Episode names were
filename-derived until the NFOs were rewritten, after which they picked up the
real titles from `<title>` — full parity with the .mp4 library. So correcting
metadata does not need a full replace (which would probe); just rewrite the NFO
and run a `Default` refresh.
- A `title` collision bug shipped briefly: the tree generator joins `video` and
`channel`, **both of which have a `title` column**, so `row["title"]` was the
video title and both shows got renamed after whichever episode sorted first.
Fixed by passing show fields explicitly. Worth remembering for §7 step 5, where
the same join exists.
### Catalogue size — the current library is small only because of retention
The 7 episodes are not a limit of the design, they are what `subs.db` holds:
`backfill_days = 7` and `retention_days = 9`, both driven by disk. A streaming
library has no such constraint, so it should carry the whole back catalogue.
Actual sizes, from one `--flat-playlist` call per channel:
| channel | videos in catalogue | in subs.db |
|---|---|---|
| Pitch Side | **1249** | 15 |
| The Pyramid Podcast | 3 | 3 |
At 1249 episodes the cost is trivial for `.strm` + `.nfo` (a few MB) and ~250 MB
of thumbnails at maxres. But two things get harder:
1. **`upload_date` is not in a flat listing.** Season and episode numbers are
`year` and `MMDD*10 + ordinal`, so dates are required.
`--extractor-args youtubetab:approximate_date` supplies timestamps in the same
single request, but they are **wrong by up to 2 days** (measured against
subs.db: `J9O3sxoMs5U` reported 08-12, actually 08-10). That yields wrong
episode numbers and, worse, *unstable* ones — a regeneration would renumber
episodes and Jellyfin would lose watched state. If approximate dates are used
they must be persisted on first sight and never recomputed, exactly as
specs.md already mandates for `dir_name`. Accurate dates cost one metadata
extraction per video: ~1249 requests per channel, one-time.
2. **The full-metadata-replace footgun scales with the catalogue.** At 7 episodes
it is an annoyance; at 1249 it would try to fetch every video in the channel.
`--max-pipelines 2` turns a stampede into a slow grind but does not stop it. A
cold-start rate limit should land before any full catalogue does.
### Running it for real
The proxy must outlive a shell. `ytstream.service` is written but **not
installed** — that needs sudo:
```
sudo cp /home/susan/ytstream/ytstream.service /etc/systemd/system/
sudo systemctl daemon-reload && sudo systemctl enable --now ytstream
```
The unit pins `PATH` to the automation venv, which is load-bearing: bare `yt-dlp`
would otherwise resolve to the 2023.11.16 binary in `/usr/local/bin`.
To remove everything: delete the `YouTube (stream)` library in the Jellyfin UI,
then `rm -rf /disks/Plex/_ytstream`.
---
## 7. Files
| file | purpose |
|---|---|
| `ytstream.py` | **the proxy.** Multi-video: `/watch/<video_id>`, `/healthz`. Retry, pipeline cap, LRU cache, `--no-fetch` |
| `ytstream.service` | systemd unit, not installed (needs sudo) |
| `make_strm_tree.py` | builds the .strm/.nfo library from subs.db |
| `add_thumbnails.py` | episode thumbnails + channel poster/fanart, idempotent |
| `ytstream_poc.py` | the original single-video PoC, kept as the record of what was verified |
| `test_range.py` | 25 range-server assertions, no network |
| `test_limits.py` | 18 assertions for the rate limiter, pipeline cap and --no-fetch |
| `run_test.sh` | one end-to-end acceptance run; prints TTFB, codecs, duration |
| `trial_403.sh` | measures the intermittent-403 rate for the PoC's access pattern |
| `mp4boxes.py` | dumps fMP4 boxes and the duration fields |
| `mehd_patch.py` | the `mehd` insertion — kept because the negative result matters |
## 8. Where this stands
The handover's §8 advice was to build the batch downloader first. That is already
done — `/opt/youtube-automate` is live — so this proxy is the second-phase
experiment it was meant to be.
Every unknown in §3 and §7 resolves in the design's favour and both Jellyfin gates
pass. The architecture holds: ffmpeg never touches googlevideo, the mux is a pure
copy, and Jellyfin direct-plays the result with the correct duration.
Three things are known-imperfect rather than unknown:
1. **Cold start costs ~55s on a 46-minute episode** (~21s for 22 minutes).
Inherent to wait-for-complete, and wait-for-complete is mandatory because
Jellyfin takes duration only from its own probe.
2. **A full metadata replace probes every episode.** Now bounded by
`--max-starts` as well as `--max-pipelines` (§9), but still don't run one over
a large library.
3. **The intermittent 403 is still unexplained.** Several plausible causes are
ruled out (§9) and retry recovers it every time, but that is a mitigation.
Not done, and the honest next steps:
- `ytstream.service` is written but **not installed** — that needs sudo. Until
then the library only plays while the proxy runs in a shell.
- Nothing writes `.strm` automatically. `make_strm_tree.py` is a one-shot from
`subs.db`; §7 step 5 is untouched and `/opt/youtube-automate` is unmodified.
- The library covers 7 episodes because that is what `subs.db` holds. A real
deployment should carry the full catalogue — see the catalogue section in §6
for the two things that get harder at 1249 episodes.
- Two libraries coexist by choice: `YouTube` (.mp4) and `YouTube (stream)`
(.strm), same 7 episodes, for comparison.
- tmpfs LRU eviction is implemented but has never fired; the cap was never
reached in testing.
---
## 9. Hardening pass (2026-08-12, after the gates passed)
### The 403 investigation — the obvious fix would have made it far worse
`player_client` was `default,mweb`. Inspecting the `c=` and `pot=` parameters on
the chosen format URLs showed something that looked like an open-and-shut cause:
| `player_client` | video/audio served from | PO token in URL |
|---|---|---|
| `default,mweb` | `ANDROID_VR` | **no** |
| `mweb` | `MWEB` | **yes** |
The obvious conclusion — "we are using token-less URLs, pin the client that gets a
token" — is **wrong**, and testing it was worth the ten minutes:
```
player_client=mweb -> 0 ok / 6 fail (403 on both streams, every trial)
player_client=default -> works
```
mweb fails **both** as a single process and via `--load-info-json`, so it is not a
round-trip problem — mweb URLs are simply rejected right now, PO token and all.
`ANDROID_VR` *without* a token works; `mweb` *with* one does not. Had this been
"fixed" by pinning mweb, every playback would have broken.
Full client sweep on the same video:
| client | result |
|---|---|
| `default` (→ android_vr) | **works** |
| `android_vr` | works |
| `mweb` | 403, every time |
| `web`, `ios`, `web_safari`, `tv` | no usable formats — SABR forced |
So there is no client-level fix available: `default` is the only thing that works,
and it is already what was in use. What changed:
- **`mweb` dropped from `player_client`.** It contributed nothing usable, and it
was the sole source of the DRC and dubbed-language formats behind both original
picker bugs (§2). Removing it eliminates those hazards before the picker sees
them, and cut extraction time from ~5s to **2.3s** — one fewer client API call,
which comes straight off cold-start latency.
- Left as `default` rather than pinned to `android_vr`, so a yt-dlp update can
follow YouTube if android_vr stops working.
**The 403 remains unexplained and still occurs** — one retry fired during the
post-hardening Jellyfin test (`retried: 1, failed: 0`). Retry with re-extraction
recovers it every time observed, so it stays the mitigation. What is now ruled
out: rate limiting (a 2s retry succeeds), `--load-info-json` (fails identically
without it), concurrency (5/5 clean), and PO token absence (token-less URLs are
the ones that work).
### Cold-start rate limiter
`--max-pipelines` caps concurrency, which only slows a runaway metadata refresh
down — over 1249 episodes it would still fetch the whole catalogue, just two at a
time. `--max-starts` (default 20 per `--starts-window`, default 3600s) bounds the
total instead.
The discriminator is that it counts **cold starts only**. Cache hits are never
limited, so re-watching and resuming keep working with the budget exhausted. A
person starts a handful of new videos an hour; a refresh storm hits 20 in seconds.
When the budget is spent the proxy refuses with 503 and logs loudly. From the §6
measurements, a refused probe produces a short libavformat retry burst and then
aborts the refresh — which is the point: an aborted refresh costs some
re-derivable metadata, a runaway one costs the entire catalogue in YouTube
traffic.
`test_limits.py` covers the limiter, the concurrency cap and `--no-fetch` with the
pipeline runner stubbed out — 18 assertions, no network.
### Known gap
Sessions live in memory, so a proxy restart orphans whatever is already on tmpfs
and the next request re-fetches. Harmless on this host (tmpfs is cleared on
reboot anyway) but worth knowing if a service restart ever looks like a stall.