ytstream/plan.md

# `ytstream` — implementation plan

**Target machine:** `susan`
**Status:** **deployed and running.** Both systemd units are installed and active, 10 of the 119
mirrored channels are approved, and 251 episodes are live in Jellyfin with correct metadata and
verified DirectPlay. 345 tests pass. What remains is the cron entries, curating the rest of the
subscription list, and the cut-over in §12.

The streaming PoC measurements this plan was designed around are in **`FINDINGS.md`** alongside this
file. What the build changed is in **§17**; what deployment changed is in **§18**.

**Repo:** `/opt/ytstream`, pushed to `/disks/git-repos/ytstream.git`, branch `main`. The PoC
scaffolding under `/home/susan/ytstream` is superseded; the proxy now lives in `proxy/` and the
PoC-era media tree was moved to `/disks/Plex/_cache/ytstream-poc-tree-backup`.

**Relationship to `youtube-automate`:** ytstream **replaces** it. The two are entirely separate
trees, databases, services and Jellyfin libraries, and they will run side by side only for as long
as it takes to satisfy §13. `youtube-automate` is then decommissioned (§12). Nothing in
`/opt/youtube-automate` is modified by this work.

---

## 1. What we're building

A DVR-shaped YouTube library for Jellyfin that **stores no video bytes**.

`youtube-automate` downloads each video to disk and points Jellyfin at the file. ytstream writes a
~50-byte `.strm` file containing a URL, and materialises the actual video only when somebody
presses play — fetched on demand by a local proxy, held on tmpfs, dropped when the cache fills.

### Explicitly in scope

- Mirroring **one YouTube account's subscriptions** as Jellyfin TV series, one episode per video —
  that account is the source of truth for what exists (§4)
- **Automatic subscription pickup from a YouTube account's public subscription list** (§4) — the
  new requirement
- A **rolling 30-day window** per channel: videos older than that are removed from disk, so the
  library stays small and bounded forever (§5)
- Just-in-time streaming via the proxy, with per-video caching
- NFO metadata, episode thumbnails, channel poster/fanart
- An admin UI for subscriptions, status and settings
- Running as real systemd services with real logs and real alerting

### Explicitly out of scope — do not build these

- Any form of transcoding. susan is a dual Westmere Xeon with **no AVX**; software transcode is off
  the table. The proxy produces `-c copy` fMP4 that Jellyfin direct-plays, and that is the only
  supported path.
- Downloading and keeping video files. If we want a permanent copy of something, that is a
  different tool.
- Playlists, Shorts, livestreams, comments, community posts, memberships.
- Anything that runs on `victoria` (the Linode). susan's residential IP is a load-bearing part of
  not getting flagged by YouTube.
- A YouTube *account* login / cookies. PO tokens only, exactly as `youtube-automate` does it.

---

## 2. This is a fork, not a green-field rewrite

`youtube-automate` is 3,543 lines and most of it is correct, verified, and has nothing to do with
downloading. Rewriting it from scratch would mean re-deriving the RSS filtering, the episode
numbering, the NFO schema and the auth code — all of which were validated on the live machine on
2026-08-11 and are documented in `/opt/youtube-automate/specs.md`.

So: copy the tree, then add, delete and replace.

| `youtube_automate` module | Fate in `ytstream` | Note |
|---|---|---|
| `discovery.py` | **Lift, then extend** | UULF-feed polling is the cheapest correct incremental source and stays — and at 119 channels it still completes in 8 s (§16). Gains an API-backed 30-day backfill (§3). |
| `naming.py` | **Lift unchanged** | Season = upload year, episode = `MMDD*10 + ordinal`. Keep byte-identical so both trees sort the same during the overlap. |
| `nfo.py` | **Lift, small change** | Drop `<fileinfo><streamdetails>` — measured to accomplish nothing (FINDINGS §6). Keep `<durationinseconds>`. |
| `channels.py` | **Lift, minus artwork fetch** | Artwork moves to the API/`i.ytimg.com` path already built in `add_thumbnails.py`. |
| `db.py`, `settings.py`, `util.py`, `config.py` | **Lift** | New DB file and new schema version (§7). |
| `web/` (auth, server, templates) | **Lift** | 838 lines of working scrypt auth + admin UI. New routes for subscription sources. |
| `jellyfin.py` | **Lift, harden** | Library refresh must never be `replaceAllMetadata` (§5). |
| `doctor.py` | **Lift, extend** | Add checks for API key validity, proxy health, `.strm` orphan count. |
| `ytdlp.py` | **Lift, narrow** | Only the proxy calls yt-dlp now. |
| `download.py` (330 lines) | **Delete** | Replaced by `strm.py`, which writes a text file. |
| `reap.py` (150 lines) | **Lift, simplify** | Originally slated for deletion — the 30-day window (§5) brings it back. Same aging-out logic and tombstone semantics, minus the disk-cap machinery, and deleting a `.strm` instead of a 500 MB mp4. |
| `runner.py` | **Rewrite** | The run loop changes shape: sync subscriptions → poll → materialise → refresh. |
| — | **New: `strm.py`** | Writes `.strm` + `.nfo` + thumbnail for one video. |
| — | **New: `subsync.py`** | The brother-subscription puller (§4). |
| — | **New: `api.py`** | YouTube Data API v3 client (§3). |
| — | **New: `proxy/`** | `/home/susan/ytstream/ytstream.py`, moved in and split up. |

Everything lifted keeps its tests. `youtube-automate` has 12 test modules; they come across too.

---

## 3. The pivot: the YouTube Data API becomes the metadata source

The subscription feature (§4) forces us to get a Google API key. Once we have one, it is worth
noticing what else it buys, because it resolves the two problems that would otherwise make the
full back catalogue impractical.

**Verified quota costs** (`developers.google.com/youtube/v3/determine_quota_cost`): a new project
gets **10,000 units/day** shared across everything except `search.list` and `videos.insert`, which
have their own 100-call/day buckets. `subscriptions.list`, `playlistItems.list`, `videos.list` and
`channels.list` are **1 unit each**, and each returns up to 50 items.

That changes the arithmetic completely. All figures below use the measured shape of the real account:
**119 subscriptions, 441 long-form videos in a 30-day window** (§16).

| Job | Method | Calls | Units |
|---|---|---|---|
| Sync subscriptions, hourly | `subscriptions.list` | 3 pages × 24/day | **72/day** |
| Initial backfill, all 119 channels | `playlistItems.list` @50 | 119 | 119 once |
| Durations for those 441 videos | `videos.list` @50 ids | 9 | 9 once |
| Steady-state incremental discovery | **RSS feed** | 119/hour | **0** |
| Durations for the day's ~15 new videos | `videos.list` @50 ids | 1 | 1/day |

**Initial build: ~130 units. Steady state: ~75 units/day**, nearly all of it the hourly subscription
poll — which is 3 pages rather than 1 precisely because there are 119 subscriptions. Against
10,000/day that is under 1%, so quota is not a constraint worth designing around, provided we never
touch `search.list` (100 calls/day, and we have no use for it).

If it ever mattered, the lever is obvious: the subscription list changes daily at most, so polling it
hourly is already generous and dropping to every 4 hours would cut the bill by 75%.

### Why this matters more than the quota

**It takes yt-dlp out of the cataloguing path entirely.** `youtube-automate` needs a yt-dlp
extraction per video to learn an upload date whenever it backfills past the RSS feed, at roughly 2.3s
each. That is the "scan storm" failure mode the original handover warned about: requests to YouTube
from a residential IP, in bulk, for videos nobody asked to watch.

Be honest about the size of the win now that the window is 30 days: at 441 videos, the yt-dlp route
would be ~441 extractions ≈ 17 minutes, not the hours a 90-day window implied. The
API is still the right answer — it is documented, keyed, quota-metered, indifferent to PO tokens and
SABR, and cannot be rate-limited by YouTube's anti-bot heuristics — but it is now buying
*correctness* (exact dates, real durations) far more than it is buying *safety*.

So the split becomes:

- **Cataloguing** (what exists, when, how long, what it's called) → **YouTube Data API + RSS**. No
  yt-dlp, no PO token, no IP-flagging risk.
- **Playback** (actual media bytes) → **yt-dlp inside the proxy**, one video at a time, only when a
  human pressed play.

That is a much better boundary than the current one, and it is the main reason to build ytstream as
a new service rather than patch `youtube-automate`.

### It also fixes exact upload dates

Measured during the PoC: `--flat-playlist` reports `timestamp: None`, and
`youtubetab:approximate_date` is **wrong by up to 2 days**. Since season/episode is derived from
the upload date, an approximate date means episodes numbered into the wrong day — and Jellyfin
caches episode numbers, so fixing it later is a metadata-wipe operation. `playlistItems.list`
returns `contentDetails.videoPublishedAt` as an exact RFC-3339 timestamp. Use it.

### UULF works through the API — verified

`youtube-automate` polls the **UULF** playlist (`UU` with `LF` spliced in), which is undocumented but
excludes Shorts and livestreams at the cheapest possible point (specs.md §4). Whether
`playlistItems.list` would accept an undocumented playlist id was the plan's main open technical
question. **It does** — verified 2026-08-12 against Pitch Side, returning exact
`contentDetails.videoPublishedAt` timestamps:

| Playlist | `totalResults` | |
|---|---|---|
| `UULFjCJ2LaOIsPzOoXUTMDI3wg` | **1,249** | long-form only |
| `UUjCJ2LaOIsPzOoXUTMDI3wg` | **2,724** | everything |

So UULF excludes 54% of that catalogue, and the existing Shorts/livestream filtering carries over
unchanged with no fallback needed. The `UU`-plus-duration-filter path in earlier drafts is not
required — though it remains the correct contingency if YouTube ever retires UULF, since `UU` is the
documented one.

Corroborating, from the `videos.list` check: of 50 consecutive `UU` uploads, **38 were ≤120 s**. The
`min_duration_seconds = 120` default is filtering a real majority, not an edge case.

---

## 4. Subscription mirroring — the new requirement

**Goal:** the brother subscribes to a channel on YouTube, and it appears in Jellyfin without anyone
touching an admin page.

### The account to mirror — resolved

| | |
|---|---|
| Handle | `@cflux1030` |
| Channel id | **`UCPcTWaLV8zwx4WP4QExHj4Q`** |
| Display name | `C Flux` |

Resolved via yt-dlp on 2026-08-12, so no `channels.list` call is needed at build time — seed it as
the single `source` row.

**This account is the sole source of truth.** There is no second source and no parallel manual
subscription workflow: what he follows on YouTube is what exists in Jellyfin. Tom's own
subscriptions are explicitly not part of this. That is a simplification (§4.3) but it moves all the
weight onto the removal safeguards (§4.4), because nothing else protects a channel any more.

### It works, with one condition on his side

Confirmed against the API reference: `subscriptions.list` accepts a `channelId` filter —
*"The API will only return that channel's subscriptions"* — and unlike `mine`,
`mySubscribers` and `myRecentSubscribers` it is **not** documented as requiring an authorized
request. So a plain API key is enough. The condition is that his subscriptions must be public:
the implementation guide states the API returns **403** if the channel *"does not publicly expose
its subscriptions and the request is not authorized by the channel's owner"*, and the errors table
lists `subscriptionForbidden` (403) — *"The requester is not allowed to access the requested
subscriptions."*

This is a good failure mode: **403, not an empty list.** We can tell "he made his subscriptions
private again" apart from "he has no subscriptions", which matters a lot for §4.4.

**There is no scraping fallback — verified.** `https://www.youtube.com/@cflux1030/channels` returns
200 but silently serves the Home tab: the rendered `ytInitialData` lists exactly four tabs (Home,
Videos, Playlists, Search) and contains **zero** channel ids. YouTube retired the public
subscriptions tab, so the API is the *only* route to this list. That makes the privacy checkbox
genuinely mandatory rather than merely the convenient path, and it means a 403 has no workaround
short of §4.2.

### 4.1 Setup — what has to happen once

**Google side** (Tom, ~5 minutes, free, no billing account required):

1. `console.cloud.google.com` → new project, e.g. `ytstream`.
2. APIs & Services → Library → **YouTube Data API v3** → Enable.
3. Credentials → Create credentials → **API key**.
4. Restrict the key: Application restrictions → *None* (it is called from a server, so referrer and
   Android/iOS restrictions do not apply; an IP restriction is optional and breaks if susan's
   residential IP rotates). API restrictions → **YouTube Data API v3 only**.
   **Do step 2 before this step.** YouTube Data API v3 does not appear in the API-restriction picker
   until it is enabled on the project, so restricting first produces a key that blocks the one API it
   exists for — see §16, which is exactly what happened.
5. Paste it into the ytstream admin UI. It is stored in the `setting` table like
   `jellyfin_api_key` already is — never in the repo, never in a systemd unit.

No OAuth consent screen. No app verification. No user-facing consent flow.

**Brother's side** (one checkbox, and it is the only thing he has to do):

- youtube.com → Settings → **Privacy** → uncheck **"Keep all my subscriptions private"**.

**Verify immediately** — before writing any of `subsync.py` — that his account actually returns
data:

```sh
curl -s "https://www.googleapis.com/youtube/v3/subscriptions?part=snippet\
&channelId=UCPcTWaLV8zwx4WP4QExHj4Q&maxResults=50&key=$KEY" \
  | python3 -c 'import json,sys; d=json.load(sys.stdin); \
      print(d.get("pageInfo")); \
      [print(i["snippet"]["resourceId"]["channelId"], i["snippet"]["title"]) \
       for i in d.get("items",[])]'
```

A 200 with `items[]` confirms the design on the real account. A 403 means the privacy checkbox has
not taken effect — and per the finding above, there is no way around it.

**Verified 2026-08-12: 200 with items.** The feature works as designed.

> **119 subscriptions** actually returned, across 3 pages.
> `pageInfo.totalResults` claims **127** — see §15; trust the fetched list, not the count.
> → `subsync_max_new` = **25**.

### 4.2 Fallback if `channelId` turns out not to work

Ranked, if and only if the day-one curl fails. Note that all three require him to do something —
there is no silent workaround, because the public HTML route is gone.

1. **OAuth with `mine=true`.** Robust and privacy-setting-independent, but carries a trap:
   confirmed in Google's OAuth 2.0 docs, a project whose consent screen publishing status is
   **"Testing"** is *"issued a refresh token expiring in 7 days"* unless the only scopes are basic
   profile ones. `youtube.readonly` is not, so the token would die weekly. Avoiding that means
   moving the app to "In production", which for a sensitive scope means Google verification — a
   disproportionate amount of process for one brother. Only do this if option 3 is unacceptable.
2. **Google Takeout subscription CSV**, dropped into the admin UI. Manual, but a 30-second job
   whenever he adds channels, and zero moving parts.
3. **Just tell him to use the admin UI.** It already exists and it is one text box.

Design `subsync.py` so the source of a channel list is pluggable (§4.3 makes this fall out
naturally), so swapping between these is a small change rather than a rewrite.

### 4.3 One source of truth — the sync is authoritative

His subscription list *is* the subscription list. So the model is simply "make the DB match the API
response", and it needs no reconciliation machinery:

- A channel in his list is subscribed. A channel that leaves it is unsubscribed, subject to §4.4.
- Two columns on `channel` carry what the sync needs: `source` (provenance) and `missing_syncs` (the
  §4.4 counter). No join table.

An earlier draft modelled this as multi-source *claims*, so that a manually-added channel could not
be deleted by someone else's list. With one source that machinery is dead weight, and speculative
generality in the part of the system that deletes things is the wrong place to spend it. If a second
account is ever mirrored, `source` is already there to key on and the claims model can come back
then.

**One escape hatch survives:** `source = 'manual'` marks a channel the sync will never remove. It is
for pinning something during debugging, not a workflow, and it is not exposed as "subscribe to a
channel" in the UI — the way to add a channel is to subscribe to it on YouTube.

**The cost of this choice, stated plainly:** if he unsubscribes from a channel, its series stops
updating and disappears from the library view. That is the correct behaviour for a mirror, and it is
exactly why §4.4 refuses to act on a single bad response.

### 4.4 Removals must be slow and loud; additions must be capped

Two failure modes here are genuinely destructive, and both are cheap to defend against.

**Runaway additions.** If he has 400 subscriptions, the first sync queues 400 channels — ~8,000
episodes at the measured rate in §5, and 400 backfills' worth of API calls and file writes in one
run. The 30-day window makes this survivable where the earlier 90-day draft did not, but it is still
not something to discover by accident. So:

- A sync that would add more than **`subsync_max_new`** channels adds none of them. It records them
  as `pending_approval`, alerts, and waits for a click in the admin UI.
- The cap applies per sync run, so ordinary drip-feed additions never trip it.

**Separate the first sync from steady state.** Otherwise the cap always trips on day one, whatever it
is set to, and the guard trains everyone to ignore it. This is now certain rather than hypothetical:
the measured list is **119 channels**, so the first sync trips any sane cap. So a source's **first**
sync is an explicit bulk import — the admin UI shows all 119 with per-channel checkboxes and the
episode count each implies, and nothing is subscribed until someone confirms. It is also the natural
place to *not* take all 119: a personal subscription list accumulated over years is not automatically
a list of things worth putting in a media library.

From the second sync onwards the cap is a runaway guard. **`subsync_max_new` = 25**, from
`max(10, ceil(119 × 0.2))` — a genuine burst gets through, an order-of-magnitude jump does not.

**Runaway removals.** A transient 403, a network blip, or him re-ticking the privacy box all look
like "he unsubscribed from everything". Deleting 119 channels' worth of Jellyfin metadata on that
basis would be unrecoverable in any pleasant way. So:

- **A 403, a 5xx, a timeout, or a zero-item 200 is never treated as a removal.** It increments the
  source's failure counter, alerts, and changes nothing. A genuinely empty list is
  indistinguishable from a broken one in consequence, and we prefer the harmless reading.
- A channel missing from an otherwise-healthy response increments `missing_syncs`. Only at
  **`subsync_missing_threshold`** (default **3** consecutive syncs, so ~3 hours) is it unsubscribed.
- **Unsubscribing deletes the channel** — its directory, its `.strm` files, its NFOs and artwork, and
  its rows. Re-subscribing rebuilds it from scratch, which under the 30-day window (§5) is one
  `playlistItems.list` page and ~20 files: cheap enough that keeping a dormant copy around would be
  the more complicated choice. The Jellyfin series disappears, taking its watch state with it.
- `source = 'manual'` channels are exempt from all of the above.

Note how the two halves interact: because the *action* is now destructive and irreversible-ish, the
*detection* rules above are what stand between a transient 403 and a wiped library. They are not
belt-and-braces. Do not weaken the three-sync threshold to make testing more convenient — make the
threshold configurable and set it to 1 in tests.

**Ordering:** the sync runs *before* the poll in the same run, so a channel added at 14:00 has its
catalogue built in the same pass.

### 4.5 Alerting

Existing convention on susan is `runitor` + healthchecks.io at `hc.jihakuz.xyz`, one UUID per job.
Follow it. A sync that hits 403, trips the add cap, or accumulates failures must exit non-zero so
the check goes red — a silently-broken subscription mirror is the worst possible outcome, because
nothing appears to be wrong until someone asks why a channel never showed up.

### 4.6 The brother gets admin access, on the existing shared password

He gets access, which is right — the approval queue (§4.4) and the "channel went missing" alerts are
about *his* subscriptions, so they should be his to action rather than landing on Tom every time.

**Auth does not change.** One shared `admin_password_hash` in the `setting` table, exactly as
`youtube-automate` already does it, matching how every other service on susan is shared. An earlier
draft proposed a two-row `user` table; it is not worth the code. The two things given up are worth
naming once and then forgetting: neither party's access can be revoked without changing the other's
password, and the approval log records *that* something was approved rather than by whom.

`web/auth.py` therefore carries over **unchanged**, and it is adequate for an internet-facing login:
scrypt hashing and per-address failed-login throttling that reads `X-Forwarded-For` (which nginx must
set — it already does, see §12 step 4).

---

## 5. Scale — a rolling 30-day window

> **`retention_days = 30`.** A video older than 30 days has its `.strm`, `.nfo` and thumbnail
> deleted and its DB row tombstoned. Backfill on subscribe reaches back the same 30 days. One
> number governs both ends, so the library cannot grow.

This supersedes the earlier "3 months or 300 videos" answer, which was about *backfill depth* while
this is about *retention*; keeping both would mean backfilling 90 days and then immediately deleting
two thirds of it. If the intent was really "go back 3 months, keep everything", set
`retention_days = 90` and this section still holds — only the numbers move.

### What this actually costs — measured across the real subscription list

Not extrapolated. On 2026-08-12 I pulled all 119 subscriptions and probed every one of their UULF
feeds:

| | |
|---|---|
| Channels subscribed | **119** (117 with working feeds; 2 return HTTP errors) |
| **Long-form videos in the last 30 days, all channels** | **441** |
| Median per channel | **1** |
| Channels with **zero** long-form uploads in 30 days | **52 of 117** |
| Busiest | penguinz0 57, Daggerwin 30, GothamChess 30, FORMULA 1 28 |
| Poll cycle for all 119 feeds | **7.9 s** wall, 8 threads (15 feeds/s) |

So the whole library is **~441 episodes, ~1,300 files** — smaller than the 20-channel guess this
section previously carried, despite six times as many channels, because the distribution is wildly
skewed. Jellyfin will not notice this. Neither will the network: an hourly poll of all 119 feeds costs
eight seconds.

UULF is doing substantial work. For Pitch Side, `playlistItems.list` reports **1,249** items in UULF
against **2,724** in UU — it excludes 54% of the catalogue — and of 50 consecutive UU uploads, **38
were ≤120 s**. The Shorts majority is real, and it is exactly what nobody wants as Jellyfin episodes.

### The skew creates a problem the window alone does not solve

**52 of 117 channels would produce an empty series.** A channel that uploads every six weeks has
nothing inside a 30-day window, so it appears in Jellyfin as a show with no episodes — and worse, it
would flicker in and out of existence as its one video ages past the line. With a median of 1 video
per channel per month, this is the common case, not an edge case.

Recommended fix, and it is a small one: make retention **`max(30 days, the N most recent videos)`**
via a `min_keep_videos` setting, default **5**.

- A channel uploading twice a year keeps its last 5 videos permanently visible instead of showing an
  empty shelf.
- A channel uploading daily is unaffected — 30 days already exceeds 5 videos.
- Cost: `441 + (52 × 5) ≈ 700` episodes. Still nothing.
- It removes the flicker, because a video only leaves once 5 newer ones exist.

The alternative — per-channel `retention_days` overrides — is already in the schema but means hand-
tuning 119 channels, which nobody will do. **Also materialise a channel's directory only once it has
at least one in-window video**, so a genuinely dead channel does not create an empty shelf at all.

### A useful side effect: RSS nearly covers the whole window

The UULF feed returns 15 entries, and for Pitch Side those 15 span **23.3 days** — because the feed
holds 15 *long-form* items, not 15 uploads. At these rates a single free RSS fetch covers most of a
30-day window on its own.

That narrows what the API is for (§3): subscription reading, durations, and topping up the few days
RSS does not reach on a fresh subscribe. It does not eliminate it — RSS carries no duration at all,
and a busier channel would truncate sooner — but nobody should be surprised when the API turns out
to be handling a few dozen calls a day rather than thousands.

### Aging out has two consequences worth stating

1. **Tombstones are mandatory, not optional.** Delete a `.strm` and the next poll finds the video in
   the feed again and re-materialises it, forever. `youtube-automate` already solves this: the DB row
   survives deletion in a terminal state (`aged_out`) and discovery skips it. Lift that behaviour
   exactly; it is the single most likely thing to get wrong here.
2. **Jellyfin loses watch state for aged-out episodes.** Removing the media file removes the item,
   and with it play counts and resume positions. For a 30-day window on a subscription feed that is
   acceptable — this is a "what's new" library, not an archive — but it should be a deliberate choice
   rather than a surprise. Anything worth keeping permanently wants a different tool (§1).

1. **Jellyfin scan cost is now a non-issue, but still worth one measurement.** Every episode is a
   `.strm` + a `.nfo` + a thumbnail. Normal scans were measured to make **0 media probes** (FINDINGS
   §6), which is what makes this viable at all. At ~1,200 files the stat-and-parse cost will not be
   noticeable. **Record it in Phase 3 anyway**, as seconds-per-1,000-episodes, because it is the
   number that would let anyone judge a future change to `retention_days` without re-deriving it.
2. **`replaceAllMetadata` remains the one real hazard.** It is the one operation verified to probe
   media, and at ~400 items it means ~400 cold starts. The cold-start rate limiter (20/hour) contains
   the damage to YouTube's side, but the library-side result is items whose metadata got wiped and
   not re-derived. Defences, all of them:
   - the proxy 503s past the budget (built, tested — `test_limits.py`)
   - `ytstream` never itself issues a refresh with `replaceAllMetadata=true`
   - the admin UI documents "do not click Replace all metadata" in the place where someone would be
     tempted to
   - **NFOs are the source of truth**, so recovery is a re-materialise pass over the tree, not a
     re-fetch from YouTube. Make sure that pass exists and is one CLI command.
3. **Episode numbering holds, and tombstones are what make it hold.** `MMDD*10 + ordinal` clamps at
   10 uploads/channel/day and is computed against the DB, not the batch. Since aged-out rows stay in
   the DB, the ordinal for a given day never shifts as videos are deleted — which is the second
   reason the tombstones are load-bearing. Season = upload year means a 30-day window usually spans
   one season and occasionally two, which needs no special handling.
4. **The `.strm` URL is baked into every file.** Changing the proxy's host or port means rewriting
   all of them. Cheap (it is a tree walk) but it must be a supported CLI command, not a `sed`
   someone invents under pressure. Settle the URL now: `http://127.0.0.1:8099/watch/<id>`. Jellyfin
   is the only client that ever reads it, it runs on the same host, and 127.0.0.1 means the proxy is
   unreachable from the network by construction.
5. **Backfill stays resumable** (`channel.backfill_cursor`, commit per page of 50) even though a
   30-day backfill is now one or two pages. It costs a column and it is the difference between a
   crashed first sync resuming and starting over across 119 channels.

---

## 6. Components

Four processes. Two are long-running services, two are cron jobs.

### 6.1 `ytstream-proxy` — systemd service

The PoC, productionised. `/home/susan/ytstream/ytstream.py`, 31 KB, currently running by hand.

Already built and verified: `/watch/<id>` and `/healthz`, single extraction reused by both
yt-dlp legs via `--load-info-json`, FIFO→ffmpeg `-c copy` fMP4 on tmpfs, correct HTTP range
handling (25 assertions), LRU cache, concurrency cap, retry-on-403, cold-start rate limiter
(18 assertions), counters.

Outstanding before it is a service:

- **Install the unit.** `ytstream.service` is written but not installed; susan has **no
  passwordless sudo**, so this goes in `deploy/deploy.sh` for the operator to run once. The unit
  pins `Environment=PATH=/var/lib/ytstream/venv/bin:...` (load-bearing — the venv yt-dlp is the
  only one with the POT plugin), `User=susan Group=mediaserver UMask=0002`, and
  `Requires=docker.service` for the POT provider container.
- **Startup sweep of the work dir.** Sessions live in memory; a restart orphans whatever is in
  `/dev/shm/ytstream`. Wipe it on start.
- **Prove LRU eviction.** It has never fired in testing. Add a test with a tiny `--cache-gb`.
- **Decide the tmpfs budget.** `/dev/shm` is 24 GB of 47 GB RAM; throughput measured at ~480
  MB/hour of video. `--cache-gb 8` holds ~16 hours of content, which is ample. Do not raise it
  without thinking about what else on susan wants RAM.
- **Log to journald** rather than a scratch file, and keep the access log behind a flag.

**Unresolved, and staying that way:** the intermittent 403 has no known cause. Ruled out during the
hardening pass: rate limiting, `--load-info-json` round-tripping, concurrency, and — the
plausible-sounding one — missing PO tokens. Pinning `player_client=mweb` to get token-bearing URLs
measured **0 ok / 6 fail**, while token-less `ANDROID_VR` URLs work; had that "fix" shipped it would
have broken every playback. Retry with a fresh extraction is the mitigation, it works, and it fires
roughly once per dozen cold starts. Keep the `retried` counter visible so a change in that rate is
noticeable.

### 6.2 `ytstream-admin` — systemd service

Lifted from `youtube_automate/web/` unchanged: shared-password scrypt auth (§4.6), per-address
failed-login throttling reading `X-Forwarded-For`, behind nginx on `127.0.0.1:8086` (8085 is
`youtube-automate`'s during the overlap).

New routes beyond what exists: subscription **sources** (add/remove a mirrored YouTube account, show
its last sync, its failure count, its `pending_approval` queue with an approve/reject action), the
first-sync bulk-import screen, per-channel backfill-depth controls, account management for the two
users, and the API key field.

**Two people use this**, so the copy matters more than it did with a single operator: the
"do not click Replace all metadata" warning (§5) has to be where someone would be tempted, and the
approval screen has to show what a channel implies — an episode count — before it is approved.

### 6.3 `ytstream run` — hourly cron

`sync sources → poll feeds → materialise → refresh Jellyfin`. Under `flock` on
`/var/lib/ytstream/run.lock`, under `runitor` with its own healthchecks UUID, via
`sg mediaserver "..."` to match the existing convention.

### 6.4 `ytstream backfill` — manual / one-off per channel

The expensive path (§3, §5). Deliberately not on a timer: it runs when a channel is first
subscribed, and when someone explicitly asks for a deeper catalogue.

---

## 7. Data model

`/var/lib/ytstream/ytstream.db`. Start from `youtube-automate`'s schema — `channel`, `video`,
`setting` are the right shape — with these changes:

**`channel`** — add `backfill_cursor TEXT` (resumable backfill) and `uploads_playlist TEXT` (whether
UULF or UU won, §3). **Keep `retention_days`** as a per-channel override of the 30-day window (NULL =
use the global setting) — `youtube-automate` already has this column and the `rescan` semantics that
go with it. No `active` column: unsubscribing now deletes rather than deactivating (§4.4).

**`video`** — drop `size_bytes`, `downloaded_at`, `attempts`, `last_error`; none of them mean anything
when there is no download. **Keep `deleted_at`** — it is the aging-out timestamp now (§5). Keep
`duration` (from `videos.list`; it feeds `<durationinseconds>`). Add `published_at TEXT` (exact
RFC-3339 from the API) alongside the date-only `upload_date` that naming uses.

`state` becomes `{listed, materialised, skipped_short, skipped_live, skipped_old, aged_out}`. No
`pending`/`downloading`/`failed`, because materialising a `.strm` cannot meaningfully fail. The two
terminal states matter and are not the same thing:

- **`skipped_old`** — discovered already outside the window; never materialised. `youtube-automate`'s
  `rescan` can revive these if `retention_days` is raised.
- **`aged_out`** — was materialised, then deleted by the sweep. **Never revived**, or raising
  `retention_days` would resurrect months of episodes into Jellyfin as "new". These are tombstones
  and discovery must skip them unconditionally (§5).

**`channel`, continued** — two more columns instead of the join table an earlier draft had (§4.3):
`source TEXT NOT NULL` (`'youtube'` for synced, `'manual'` for the pinning escape hatch) and
`missing_syncs INTEGER NOT NULL DEFAULT 0` (the §4.4 counter).

**`source`** — new, and in practice **exactly one row**: `key`, `label`, `channel_id`, `enabled`,
`last_sync_at`, `last_sync_ok`, `consecutive_failures`, `last_error`. A one-row table rather than six
`setting` keys, because these are fields of one thing and they change together.

**`pending_approval`** — new, the §4.4 add-cap and first-sync import queue: `source`, `channel_id`,
`title`, `seen_at`, `resolved_at`, `resolution` (`approved`/`rejected`).

**`setting`** — carry over what still applies (`jellyfin_url`, `jellyfin_api_key`,
`pot_provider_url`, `max_height`, `min_duration_seconds`, `admin_password_hash`, `session_secret`,
`last_run_at`), including **`retention_days`** — now **30**, and governing both the aging-out sweep
and the backfill reach (§5). Drop `disk_cap_gb`, `write_subs`, `sub_langs`, `sponsorblock_mark`,
`max_attempts`, `backfill_days` — download-era concepts. Add `youtube_api_key`, `subsync_max_new`
(**25**, derived in §4.4), `subsync_missing_threshold` (3), `proxy_base_url`
(`http://127.0.0.1:8099`), `backfill_max_videos` (**300**) purely as a runaway guard on a channel that
turns out to upload 50 times a day, and **`min_keep_videos` (5)** — the fix for the 52 empty series
measured in §5.

**No migration from `subs.db`.** It holds 2 channels and 18 videos. Re-subscribe by hand and let
the backfill do the rest; a migration script would be more code than the data is worth.

---

## 8. On-disk layout and the `.strm` contract

Media root: **`/disks/Plex/_ytstream/`** — already created and already wired to the *YouTube
(stream)* Jellyfin library, with 2 channels in it from the PoC. Keep it. (The name has a leading
underscore matching `_ingest`/`_cache` on that volume; `/disks/Plex/YouTube` stays with
`youtube-automate` until §12 retires it.)

```
/disks/Plex/_ytstream/
└── Pitch Side/
    ├── tvshow.nfo
    ├── poster.jpg                                  # channel avatar
    ├── fanart.jpg                                  # channel banner
    └── Season 2026/
        ├── Pitch Side - S2026E8120 - Title [dQw4w9WgXcQ].strm
        ├── Pitch Side - S2026E8120 - Title [dQw4w9WgXcQ].nfo
        └── Pitch Side - S2026E8120 - Title [dQw4w9WgXcQ]-thumb.jpg
```

Naming, sanitisation and `dir_name` stability rules are `youtube-automate`'s, unchanged
(specs.md §5) — identical rules in both trees is what makes the overlap period sane.

**`.strm` contents:** one line, no trailing newline required:

```
http://127.0.0.1:8099/watch/dQw4w9WgXcQ
```

**NFO:** `episodedetails` with `title`, `season`, `episode`, `aired`, `plot`, `runtime`,
`durationinseconds`, `uniqueid type="youtube"`. **No `<fileinfo><streamdetails>`** — pre-seeding it
was measured to change nothing about whether Jellyfin probes (FINDINGS §6), so it is dead weight.

**No `.work/` dir under the media root.** The proxy's scratch space is `/dev/shm/ytstream`, outside
the library entirely, which is strictly better than the dot-dir-plus-`.ignore` belt-and-braces that
`youtube-automate` needs.

---

## 9. Environment and paths

Verified on the machine, 2026-08-12.

| Purpose | Path |
|---|---|
| Source (bare repo) | `/disks/git-repos/ytstream.git` — **exists but is not yet `git init --bare`** |
| Checkout | `/opt/ytstream` — exists, `susan:automation` `0770` |
| Entry point | `/usr/local/bin/ytstream` (needs root, → `deploy.sh`) |
| Virtualenv | `/var/lib/ytstream/venv` |
| State DB | `/var/lib/ytstream/ytstream.db` |
| Lock file | `/var/lib/ytstream/run.lock` |
| Media root | `/disks/Plex/_ytstream/` |
| Proxy scratch | `/dev/shm/ytstream` (tmpfs, 24 GB available) |
| Admin UI | `127.0.0.1:8086`, nginx → `tube.jihakuz.xyz` (§12) |
| Proxy | `127.0.0.1:8099`, not exposed |

Facts that constrain the design:

- **Ownership:** the media tree convention is `susan:mediaserver`, dirs `0770`, files `0664`.
  `/disks/Plex` is `0770` with no setgid; Jellyfin reaches the tree only through its `mediaserver`
  supplementary group. Both units run `User=susan Group=mediaserver UMask=0002`; cron goes through
  `sg mediaserver`. `/disks/Plex/_ytstream` already has the setgid bit — keep it.
- **No passwordless sudo.** Everything touching `/usr/local/bin`, `/etc/systemd/system` or
  `/etc/nginx` belongs in `deploy/deploy.sh`.
- **yt-dlp:** `/usr/local/bin/yt-dlp` is a **2023.11.16** binary, far too old to work. Nothing
  references it; leave it alone. ytstream gets its own venv with `yt-dlp[default]` (the extra that
  ships `yt-dlp-ejs`, mandatory for JS challenge solving), `bgutil-ytdlp-pot-provider==1.3.1`
  matching the container tag, and `curl-cffi<0.16`. Note that during the overlap **two** venvs will
  hold yt-dlp; keep both current, and do not let a bare `yt-dlp` on `PATH` be what either service
  resolves.
- **POT provider:** container on `127.0.0.1:4416`, `--restart unless-stopped`, Docker enabled at
  boot. Shared with `youtube-automate` during the overlap and inherited afterwards. It is the
  reason the proxy unit declares `Requires=docker.service`.
- **Client selection is settled — do not re-litigate it.** `CLIENT_ARGS =
  "youtube:player_client=default"`. `mweb` is deliberately absent: its formats 403 on every
  attempt, and it was the sole source of the DRC and dubbed-language variants behind both original
  format-picker bugs. `web`/`ios`/`web_safari`/`tv` are SABR-only and yield nothing usable. Left as
  `default` rather than pinned to `android_vr` so a yt-dlp update can follow YouTube.
- **Jellyfin 10.11.4**, native systemd, `0.0.0.0:8096`, API key already in the DB.

---

## 10. Playback UX — state it plainly

Measured: wait-for-complete is ready in **20 s** for a 21-minute video and **~55 s** for a
46-minute one, at ~8 MB/s (~60× realtime). `--growing` mode gives TTFB of **6.5–7.1 s** but ffmpeg
writes `mvhd.duration=0` and no `mehd` box regardless of what it knows about the input — verified,
including an attempt to patch the boxes afterwards, which ffmpeg ignores — so the player sees a
video of unknown, growing length. Seeking and progress bars misbehave.

**Ship wait-for-complete.** A ~20–55 s spinner that then behaves like a normal file beats instant
playback with a broken timeline. Keep `--growing` as a flag for experimentation. Revisit only if
someone complains, and if they do, the fix is a pre-warm on library browse rather than a change to
the mux.

---

## 11. Configuration and secrets

Secrets live in `/var/lib/ytstream/ytstream.db`, mode `0640` `susan:automation`, and are entered
through the admin UI. Never in the repo, never in a systemd unit, never in a cron line. This matches
`youtube-automate`, where `jellyfin_api_key` and `session_secret` already live that way.

Four keys, all in `setting`: `youtube_api_key` (new), `jellyfin_api_key`, `session_secret`, and
`admin_password_hash` — one shared admin password, as everywhere else on susan (§4.6).

The YouTube API key is restricted to the YouTube Data API v3 and has read-only reach over public
data. Worst case on leak is quota exhaustion; rotation is a two-minute job in the console.

**The Phase 0 key is treated as disposable and rotated once the service is built** (Tom's call, and
the right one — it has been pasted into a terminal and a chat transcript while debugging). That makes
one thing a requirement rather than a nicety: **rotating the key must not need a restart or a code
change.** Read it from the `setting` table at the point of use, not once at process start, so the
admin UI's "save" is the whole rotation procedure.

---

## 12. Decommissioning `youtube-automate`

Not until §13 phase 5 passes. Order matters — the point is that every step is reversible until the
last one.

1. **Stop new work.** Comment out the `17 * * * *` cron entry. Leave the service running so the
   admin UI still answers.
2. **Watch for a week** with both libraries live in Jellyfin. This is the real acceptance test:
   does anyone reach for the old library?
3. **Retire the Jellyfin library.** Remove *YouTube* (`/disks/Plex/YouTube`) via
   `/Library/VirtualFolders`. Rename *YouTube (stream)* → *YouTube*. **Note from the PoC:**
   deleting and recreating a library at the same path returns the **same** `ItemId` and reuses the
   old items — if a clean slate is ever needed, use a fresh path, as `_ytstream` already is.
4. **Free the hostname.** `tube.jihakuz.xyz` is served by a leftover TubeArchivist `server` block
   inside `sites-available/jihakuz.xyz`, which owns the Let's Encrypt cert and wins because nginx
   takes the first matching block. Repoint it at 8086 the same way `deploy/fix-nginx-tube.sh` did
   for 8085; do not install a competing vhost file.
5. **Disable the service.** `systemctl disable --now youtube-automate.service`; remove the unit.
6. **Reclaim the bytes.** Estimated 5–10 GB under `/disks/Plex/YouTube`. Delete only after step 2
   has actually elapsed.
7. **Keep, do not delete:** `/var/lib/youtube-automate/subs.db` (copy it aside — it is the only
   record of what was subscribed and when), `/opt/youtube-automate` and its bare repo, and
   `specs.md`, which remains the reference for every rule ytstream inherited.

`update-ytdlp.sh` (Mondays 04:40) must be **repointed, not removed** — it becomes ytstream's, and it
is the thing that keeps playback working as YouTube changes.

---

## 13. Build order

Each phase ends in something checkable. Do not start the next one until it does.

**Phase 0 — verify the assumptions the plan rests on. ✅ COMPLETE 2026-08-12.** All three API checks
pass; `tools/verify_api.py` reproduces them in one command. Results and the numbers they changed are
in §16.

**Phase 1 — skeleton and lift. ✅ COMPLETE 2026-08-12.** Forked, renamed, new schema, 337 tests
green. Lifted: `naming`, `nfo`, `db`, `settings`, `util`, `config`, `channels`, `jellyfin`, `doctor`,
`ytdlp`, `web/`. Deleted `download.py`. See §17 for what changed on the way through.

**Phase 2 — the proxy as a service. ⏳ CODE DONE, INSTALL PENDING.** Moved to
`proxy/ytstream_proxy.py`; its two standalone test scripts are now `tests/test_proxy.py`, driving the
real `make_handler(mgr, …)` so routing and video-id validation are covered too. `deploy/` carries both
systemd units, `bootstrap.sh` and `deploy.sh`.
→ *Blocked on root: the operator must run `sudo /opt/ytstream/deploy/deploy.sh`. Until then the
PoC-era proxy on 8099 is what serves playback.*

**Phase 3 — catalogue and retention. ✅ CODE COMPLETE, verified against the live API.** Pitch Side
backfilled to **20 episodes** (the §5 estimate was ~18), Asianometry to 6, in **6.8 s** for a full
channel. Titles, exact dates and durations all correct; a generated `.strm` fetched through the proxy
returns h264 720p + aac and honours ranges; the NFO's `durationinseconds` matches the API to the
second.
→ *Still outstanding: the Jellyfin scan-time measurement for §5, which needs the tree at the real
media root, which needs Phase 2 installed.*

**Phase 4 — subscription sync. ✅ CODE COMPLETE.** `subsync.py`, the first-sync import, the add cap,
the missing-threshold, deletion on unsubscribe, and the `/pending` admin page with source management,
sync-now, and multi-select approve/reject — all four new routes CSRF-guarded, verified over real HTTP.
Against the live account the first sync queued **119 channels and added none**; approving three at
once added three.

The destructive half is covered by tests rather than by having done it to the real account: a forced
403, a forced network error and a forced empty response each leave the database untouched, absence is
counted across three healthy syncs before deletion, and `manual` channels are exempt.
→ *Still outstanding: the healthchecks UUIDs in `deploy/crontab.fragment`, and watching a real
subscribe-then-unsubscribe cycle once the services are installed.*

**Phase 5 — cut over.** §12 steps 1–2, run for a week.
→ *Done when: nothing has broken and nobody has used the old library.*

**Phase 6 — decommission.** §12 steps 3–7.

---

## 14. Decisions — answered 2026-08-12

1. **Account to mirror:** `@cflux1030` → `UCPcTWaLV8zwx4WP4QExHj4Q` ("C Flux"), resolved and
   recorded in §4. **Sole source of truth** — Tom's own subscriptions are not part of this at all,
   which removed the multi-source claim model from §4.3 and put the weight on §4.4 instead.
2. **Retention:** a **rolling 30-day window** — anything older is deleted from disk and tombstoned
   (§5). This supersedes the earlier "3 months or 300 videos" answer, which was about backfill depth;
   holding both would mean backfilling 90 days and deleting two thirds of it immediately. If the
   intent was "reach back 3 months and keep it", `retention_days = 90` is a one-line change and §5
   still holds. Measured consequence: **441 episodes total** across 119 channels, not 20,000.
3. **`subsync_max_new` = 25** — closed by Phase 0. Measured 119 subscriptions, and
   `max(10, ceil(119 × 0.2)) = 25` (§4.4).
4. **Brother gets admin access:** yes, on the **existing shared password** — no `user` table, no
   per-account credentials (§4.6). `web/auth.py` carries over untouched.
5. **Unsubscribe deletes the channel** rather than deactivating it, since re-subscribing is one API
   page and ~20 files under the 30-day window (§4.4). This makes the removal *detection* safeguards
   load-bearing rather than precautionary.

---

## 15. Gotchas carried forward

Things already paid for once. All of these are verified.

- **A normal Jellyfin scan makes 0 media probes**, which is the single fact this design depends on.
  `replaceAllMetadata=true` **does** probe. Pre-seeded `streamdetails` do not prevent it.
- **`--load-info-json` reuses the extraction** — confirmed twice, once during the original PoC and
  again while hunting the 403. Both yt-dlp legs run from one `-J`.
- **ffmpeg will not write a duration into a fragmented MP4**, and will not accept one patched in
  afterwards. Hence wait-for-complete (§10).
- **Two picker bugs, both silent, both from `mweb`:** `140-drc` was chosen over `140` on an `abr`
  tie and 403'd; `140-0` was chosen on a 9-language video and produced German audio. The audio sort
  key ranks `language_preference` **first**, deliberately above the DRC check — a 403 is a loud
  failure, wrong-language audio is a silent one that would have shipped.
- **`row["title"]` ambiguity renamed every series once.** The channel/video join has `title` on both
  sides. Pass show fields explicitly, never `row["title"]`. Caught only by accident.
- **Jellyfin's `/Items/{id}` needs user context** — use the `/Items?ids=` form.
- **`youtubetab:approximate_date` is wrong by up to 2 days.** Do not derive episode numbers from
  it; use `playlistItems.list` (§3).
- **YouTube's public subscriptions tab is gone.** `/@handle/channels` returns 200 and silently
  serves Home — four tabs, zero channel ids in `ytInitialData`. Verified 2026-08-12. Nobody should
  spend an afternoon trying to scrape it; the API is the only route (§4).
- **`pkill -f 'ytstream.py'` kills the shell that runs it**, because the command string contains its
  own pattern. Bracket it: `pkill -f 'ytstrea[m].py'`.
- **Google returns 403 `forbidden` for two unrelated setup mistakes**, and the useful signal is in
  `error.details[].reason`, not `error.errors[].reason`. `SERVICE_DISABLED` means the API is not
  enabled on the project (and carries an `activationUrl` naming the project number);
  `API_KEY_SERVICE_BLOCKED` means the API is enabled but *this key's* restrictions exclude it. They
  are fixed on different console screens. `tools/verify_api.py` distinguishes them and prints the fix.
- **Console changes take up to ~2 minutes to propagate.** Measured: a key restriction change was still
  returning `API_KEY_SERVICE_BLOCKED` at 0 s, 30 s and 60 s, and succeeded at **90 s**. Do not
  conclude a setting is wrong until a couple of minutes have passed — retry before re-editing.
- **`subscriptions.list` `pageInfo.totalResults` overcounts.** It reported **127** while full
  pagination returned **119** distinct channels, and 2 of those 119 have feeds that HTTP-error —
  terminated or private channels still counted as subscriptions. Never treat `totalResults` as the
  list length; page to exhaustion and count what arrives. Anything sized off it (a progress bar, a
  cap, an "is the list complete" check) will be wrong.

---

## 16. Phase 0 results — COMPLETE, 2026-08-12

All three assumptions the plan rested on are verified against the live API. Nothing in the design had
to change; several numbers did.

### The three API checks

| # | Assumption | Result |
|---|---|---|
| 1 | `subscriptions.list?channelId=` returns his subscriptions with only an API key | **PASS.** 200, **119** channels across 3 pages. The feature is possible exactly as designed. |
| 2 | `playlistItems.list` accepts an undocumented `UULF…` playlist id | **PASS.** 1,249 items with exact `videoPublishedAt`. No `UU` fallback needed (§3). |
| 3 | `videos.list` returns parseable `contentDetails.duration` | **PASS.** 50 ids in one call, 50 returned, 0 unparseable. |

Reproduce with `python3 tools/verify_api.py --key <key>` (~5 quota units).

### The numbers that came out of it

- **119 subscriptions**, not the ~20 the plan assumed — and `pageInfo.totalResults` claims 127 (§15).
- **441 long-form videos across all 119 channels in the last 30 days.** That is the library size.
- **Median 1 video per channel per 30 days**, and **52 of 117 channels uploaded nothing** — the
  distribution is severely skewed, which is what produced the `min_keep_videos` recommendation in §5.
- **UULF excludes 54%** of Pitch Side's catalogue (1,249 of 2,724), and **38 of 50** consecutive `UU`
  uploads were ≤120 s. The Shorts filter earns its place.
- **All 119 feeds poll in 7.9 s** (8 threads). Hourly polling is free.
- `subsync_max_new` = **25**.

### Also done

- **Bare repo initialised.** `/disks/git-repos/ytstream.git` was an empty directory; it is now a real
  `--bare --shared=group` repo with `HEAD` → `refs/heads/main`, matching `youtube-automate.git`.
  `/opt/ytstream` is a checkout with `origin` pointed at it.
- **The mirrored channel is resolved:** `@cflux1030` → `UCPcTWaLV8zwx4WP4QExHj4Q` ("C Flux"). No
  `channels.list` call needed at build time.
- **There is no scraping fallback** — `/@cflux1030/channels` serves the Home tab with zero channel ids
  (§15), so the privacy setting is mandatory. It is now confirmed to be off, since the API returns his
  list.
- **`tools/verify_api.py`** written, self-tested, and hardened against the two setup failures below.

### Getting the key working took three attempts, and the errors were misleading

Recorded because the next person to set up a Google Cloud project will hit the same wall. All three
states return HTTP 403 `forbidden`, and the distinguishing signal is `error.details[].reason`:

1. **`SERVICE_DISABLED`** — YouTube Data API v3 not enabled on project `510818173753`. Carries an
   `activationUrl` naming the project.
2. **`API_KEY_SERVICE_BLOCKED`** — API enabled, but the *key's own* API restrictions exclude it. A
   different console screen entirely. The ordering is the trap: YouTube Data API v3 is absent from the
   key's restriction picker until the API is enabled on the project, so restricting first yields a key
   that blocks the only API it exists for. §4.1 step 4 now says enable-then-restrict.
3. **Propagation delay** — after the restriction was fixed, calls still failed at 0 s, 30 s and 60 s,
   and succeeded at **90 s**. Nothing was wrong at that point except impatience.

### What remains before Phase 1

Nothing. Phase 0 is closed. The one open decision is §14 item 2 — whether `min_keep_videos` (§5) is
wanted, which is a five-minute change either way and does not block starting Phase 1.

---

## 17. What the build changed — 2026-08-12

Three bugs, two of which only real data would have found. Recorded because each one
is a trap the next change could walk back into.

### The prune boundary — caught by a test

`strm.remove` pruned empty directories up to the media root, so a channel directory
whose `tvshow.nfo` happened to be missing would be deleted along with its last
season. It only *looked* safe because `tvshow.nfo` normally stops the walk. The
boundary is now the channel directory explicitly, and a test asserts the channel
directory survives.

### Titles were missing on the oldest episodes of every backfill

The backfill inserted rows with no title and left the RSS poll to fill them in.
That works only if RSS reaches as far back as the retention window, and it does
not: the feed returns 15 entries, which for Pitch Side spans 23 days against a
30-day window. **Five of twenty episodes were named after their video id.**

`playlistItems.list` now requests `snippet` as well as `contentDetails`. Both parts
cost the same single quota unit together as either does alone, and `snippet.title`
arrives alongside the exact publish date. Note the trap next door:
`snippet.publishedAt` is when the video was *added to the playlist*, not when it was
published — only `contentDetails.videoPublishedAt` is the real thing.

### The fallback title was written back to the database — the worse half

`strm.materialise` used `video["title"] or video["video_id"]` for the filename and
then stored *that* as the title. So an untitled row became a row whose title was its
own video id, which is not empty, which permanently disabled the repair path that
fills titles in from a later feed poll. The two bugs compounded: the first created
badly-named episodes and the second made them permanent.

Now the fallback is used for the filename only, and a title that arrives late also
deletes the badly-named files and re-queues the video so it is rewritten under its
real name.

### Also worth knowing

- **`_form()` collapsed repeated fields to the last value.** The approval queue is a
  form of checkboxes all named `id`; through `_form()` it would have silently
  approved only the last box ticked. Added `_form_list()`, and verified over real
  HTTP that approving three at once adds three.
- **`min_keep_videos` shipped at 5** rather than being left open (§14 item 2). Without
  it, 52 of 117 measured channels are empty Jellyfin series that flicker in and out
  as their single video crosses the window. Set it to 0 to get pure 30-day retention.
- **Two settings validators earn their keep**: `subsync_missing_threshold` rejects 0
  at the form, and `subsync.sync_source` clamps it to 1 anyway — a stored zero would
  mean "unsubscribe before any absence has been confirmed".

### Measured during the build

| | |
|---|---|
| Tests | **337**, no network, no yt-dlp, no Jellyfin |
| First sync of the real account | 119 queued, **0 added** |
| Pitch Side backfill | **20 episodes** (§5 predicted ~18) in **6.8 s** |
| Asianometry backfill | 6 episodes |
| Generated `.strm` played through the proxy | h264 720p + aac, ranges honoured |
| NFO `durationinseconds` vs API truth | 889 vs 889 |

---

## 18. What deployment changed — 2026-08-12

Installing it found six things the tests could not. Two were my bugs in the deploy
scripts, one was a missing dependency, and three were Jellyfin behaviours that only
appear against a real library.

### The deploy scripts had a circular dependency

`deploy.sh` installed and started the units, then told the operator to run
`bootstrap.sh` — but `bootstrap.sh` refused to run until `/var/lib/ytstream`
existed, and only `deploy.sh` creates it. Neither could go first. The units started
against a venv that did not exist, failed `203/EXEC`, and restart-looped 17 and 21
times until the venv appeared.

`deploy.sh` now creates the state directory, calls `bootstrap.sh` itself via
`runuser` so the venv is not left root-owned, and refuses to start the units at all
if the venv is still missing. One command, correct order.

### Deno was missing, and `doctor` caught it

**Mandatory, not optional.** Without a JS runtime yt-dlp cannot solve the `n`
challenge; youtube-automate measured the consequence on this machine as 22 formats
instead of 29 and throttled downloads. The new venv had `yt-dlp-ejs` but no `deno`,
and nothing else would have noticed until playback quietly degraded.
`bootstrap.sh` now installs it — preferring a copy from the youtube-automate venv
while that still exists, falling back to the GitHub release — and verifies yt-dlp
reports `JS runtimes: deno`.

### Jellyfin ignores `<runtime>` and `<durationinseconds>` for episodes

It reads the rest of the NFO — `aired` and the `youtube` provider id both arrive —
but runtime comes only from a media probe, so a `.strm` episode shows **no duration
until it has been played once**. Not fixable from our side: the only way to supply
one is to probe, which means fetching every episode, which is the one thing this
design exists to avoid. Accepted limitation, stated here so nobody re-litigates it.

### Episodes had no plot at all

`strm.materialise` passed `plot=None`, so every synopsis was empty — while both
sources hand us descriptions for free. Now plumbed through: RSS carries
`media:group/media:description`, and `videos.list` carries `snippet.description` in
the call already being made for durations, so the ~40% of episodes older than RSS
reaches still get one. That needed a **schema v2 migration**; v1 was left exactly as
it shipped so a fresh install and a migrated one end up identical, which is asserted
by a test.

### `materialise --all` created duplicates instead of repairing

The documented recovery path from a metadata wipe was itself broken. Episode numbers
were re-derived on every run, and `next_episode()` excludes the row it is numbering,
so re-materialising a day's videos in a different order renumbered them — new
filenames, old files left behind. One run left **102 orphaned NFOs against 251
episodes**.

Two fixes: an episode number, once assigned, is now permanent and reused from the
row; and materialising a video that already has a different `rel_path` removes the
old files first. Running `materialise --all` twice in a row is now a no-op, verified
on the live tree and pinned by tests.

### There is a safe metadata refresh, and this is it

§5 says never to use `replaceAllMetadata=true`, and that stands. But a plain
`/Library/Refresh` does **not** reliably re-read a rewritten NFO — after rewriting
all 251, fifty kept their old empty metadata. The middle ground works:

```
POST /Items/{id}/Refresh?metadataRefreshMode=Default&imageRefreshMode=Default
                        &replaceAllMetadata=false&recursive=true
```

Measured across the whole 251-episode library: **plots and aired dates went from 201
to 251, and the proxy served 0 requests.** The safety is entirely in
`replaceAllMetadata=false` — with it `true`, Jellyfin discards what it has and
re-derives from the media. Exposed as `ytstream refresh-metadata`, and run
automatically after `materialise --all`.

### Measured on the deployed service

| | |
|---|---|
| Channels approved (of 119 queued) | 10 |
| Episodes materialised | **251** in 70 s |
| Filtered as Shorts / livestreams | 2 / 4 |
| Tree size | 49 MB for 753 files |
| **Jellyfin full scan** | **251 episodes in ~119 s** (~8 min per 1,000) |
| **Media probes during that scan** | **0** |
| Playback via Jellyfin `PlaybackInfo` | DirectPlay h264 720p + aac, 2049 s runtime |
| Episodes with plot / aired after refresh | 251 / 251 |
| `doctor` | all fatal checks pass |

## 19. First play was broken, and how — 2026-08-13

The day after deployment, playback failed in Jellyfin. The service was healthy the
whole time: both units active, `doctor` green, `/healthz` reporting `failed: 0`.

**Cached videos played; uncached ones did not.** Yesterday's "DirectPlay verified"
was measured only on videos already pulled during testing, so the first-play path
had never actually been exercised end to end. That is the hole in §18's evidence.

### The mechanism

In wait-for-complete mode the handler blocked on `sess.final.wait(wait_timeout)`
before sending anything — not the body, not even response headers. The wait is the
whole download and mux:

| upload | cold time to first byte |
|---|---|
| 8 min | ~10 s |
| 6.8 min *(one transient failure + retry)* | >12 s |
| 22 min | 47 s |
| 46 min | **79 s** ← what the user hit |
| 66 min | 156 s |

No player waits that long, so the socket sat silent until the client gave up. The
proxy counted it a success, which is why nothing looked wrong from inside.

### The fix: bound the wait, then stream

The output is already a fragmented MP4 (`frag_keyframe+empty_moov`), so it is
readable while being written. The only thing a *finished* file buys is a correct
duration and working seeks — ffmpeg patches the real duration into the moov on
close, and ignores both `mvhd.duration` and an injected `mehd` before then
(confirmed: a growing file probes 197 s → 1185 s → 2378 s → 4127 s against a true
4128 s).

So the wait is now capped by `--first-byte-grace` (default 12 s, explicit in the
unit). Whatever has not muxed by then is streamed as it is written.

| | before | after |
|---|---|---|
| TTFB, 66-min upload cold | 156 s (silent) | **12.0 s** |
| TTFB, same video cached | ~0 | ~0 |
| Range request on a complete file | 206 + `Content-Range` | unchanged, 1.7 ms |

`--first-byte-grace` is a **cap, not a prediction**. Completion time ranged 10–156 s
and does not track duration closely: YouTube's per-format throttling varies, and one
transient failure plus a retry costs ~5 s of extraction before any byte is pulled.
Raising it to `--wait-timeout` restores the old finished-file-or-504 behaviour, which
is the bug. The unit carries a comment saying so.

### Cost, stated plainly

A first play that misses the grace has **no seek bar and no duration** for that
watch. The stream is chunked with no `Content-Length`, so the client cannot seek
even after the mux lands mid-play; it has to re-request, which happens on the next
play. Every subsequent play of that video is perfect and instant. This is a real
regression against a *hypothetical* fast first play, and a large improvement over
an error.

### A second bug found while fixing the first

The streaming loop waited on `sess.complete` (finished **and** good) rather than
`sess.final` (finished). A producer that died after writing some bytes therefore
never satisfied the wait: the loop sat in `_wait_for_bytes` for the full 45 s
`STALL_TIMEOUT` and then dropped the connection. Now split into `finished` for every
wait and `sess.complete` only for the ranges decision. Caught by a new test, not by
inspection.

### Also corrected

The too-old-yt-dlp warning pointed at `/var/lib/youtube-automate/venv/bin` — the tree
§12 decommissions. It now names ytstream's venv, and says the unit pins PATH so a
manual run has to as well. That warning is what a future debugger reads at 2am.

The `/healthz` payload gained `mode` (`wait-then-stream` / `growing`) and
`first_byte_grace_s`, because the difference between "plays" and "playback error" was
invisible without reading the unit file.

Eight new tests in `test_proxy.py` cover: a slow mux streaming instead of blocking, a
fast mux keeping ranges and seeking, `--first-byte-grace` at `--wait-timeout`
restoring strict mode, a failed producer with bytes being served but 502 in strict
mode, a failed producer with no bytes always 502, `--growing` meaning no wait, and
`/healthz` reporting the mode. 353 pass.

### A restart used to strand the cache

The session map is in memory only, so every session directory left in the work root
after a restart is unreachable (nothing can find it) *and* unevictable (the cache
budget only sums tracked sessions). The work root is a tmpfs, so that is leaked RAM
until the next reboot — the restart that shipped the fix above would have stranded
1.56 GB. `reset_work_root()` now clears it at startup and logs what it reclaimed.
Session directories only; a stray file in the work root is left alone.

## 20. Decommissioning, done and outstanding — 2026-08-13

Started the same day the TTFB bug (§19) was fixed, which is earlier than §12 step 2
intended: that step says run both for a week, precisely so a bug like §19 surfaces
while the old service is still there to fall back on. Everything below is therefore
reversible, and the two irreversible steps are deliberately left undone.

### The old service was smaller than assumed

It tracked **2 channels** (Pitch Side, The Pyramid Podcast), 32 video rows, 9 files
on disk, 2.0 GB — not the 5–10 GB §12 step 6 estimated. All 32 fall inside
ytstream's 30-day window, so nothing in the old library is content ytstream cannot
reach.

**Pitch Side was already mirrored; The Pyramid Podcast was not** — it sat unresolved
in the approval queue, so decommissioning without checking would have silently
dropped one of the two channels the old service existed to follow. Approved, and it
backfilled 4 episodes. 11 channels now.

### Done

| step | what |
|---|---|
| §12.1 | Cron handed over: `youtube-automate run` → `ytstream run`, and `update-ytdlp.sh` repointed |
| §12.7 | `subs.db` copied to `/var/lib/ytstream/youtube-automate-subs.db.archived-20260813` |
| — | The Pyramid Podcast carried over |

**The two healthchecks UUIDs are inherited, not new,** and the schedules are
unchanged (`:17` hourly, Mondays `04:40`). A check may be configured with a cron
expression rather than a simple period, so moving to the `:23`/`04:50` slots the old
fragment proposed could have alerted on a job that ran fine. This also means no new
UUIDs were needed — the placeholder problem from §18 is gone. The checks are still
*named* after youtube-automate in the hc UI; renaming them there changes nothing.

First real proof of the rolling window, from that first run: **5 videos uploaded
2026-07-13 aged out** at 31 days, with 7 new ones discovered and materialised, in
7.5 s.

### Left for a human

`deploy/decommission.sh` does §12 steps 4 and 5 (nginx repoint to 8086, disable the
service) and **refuses to run until an admin password is set** — repointing a public
hostname at a UI that fails closed, as this one does with no password, produces a
site nobody can log into and an evening spent working out why.

Not scripted, because each destroys something:

* **`ytstream set-password`** — interactive, and blocks the above.
* **Jellyfin** (§12.3): remove *YouTube*, rename *YouTube (stream)* → *YouTube*.
  Nothing in the code matches on the library *name* — `find_library()` matches on
  path and `LIBRARY_NAME` is only `create_library`'s default — so the rename is safe
  and the constant can stay as it is.
* **`/disks/Plex/YouTube`** (§12.6), 2.0 GB.
* **`/opt/youtube-automate`, its repo, `subs.db`, `specs.md`** — keep (§12.7).

### §12.3 is scripted now, and §12.7 changed

`deploy/retire-jellyfin-library.py` does the Jellyfin step. It matches libraries by
**path, never by name**, and reads the name to delete back from the API instead of
assuming it — the delete endpoint takes a name, matches loosely on some versions,
and `YouTube` is a prefix of `YouTube (stream)`. It refuses to retire the old
library unless ytstream's has episodes, because doing it with a broken replacement
leaves no YouTube library at all. Dry run by default; `--yes` applies.

What it costs, stated in the script itself: Jellyfin's watch history and resume
positions for the deleted library go with it. The files do not.

**§12.7 revised — `/opt/youtube-automate` can go after all.** Verified 2026-08-13:
the working tree is clean, everything is pushed to
`/disks/git-repos/youtube-automate.git` (612 KB), and both `specs.md` and
`specs.handover-original.md` are tracked, so the reference material survives in the
bare repo. Nothing in ytstream's code references the old tree — only comments and
`decommission.sh`, which names the *service*.

Still worth keeping out of `rm`: `/var/lib/youtube-automate` (170 MB) holds the old
venv and `subs.db`, and `subs.db` is *not* in the repo — it is state, not code. It is
already copied to `/var/lib/ytstream/youtube-automate-subs.db.archived-20260813`, so
that directory is now safe to delete too, just not before checking that copy exists.

### Jellyfin's rename endpoint takes a name, not an id — and re-ids the library

`POST /Library/VirtualFolders/Name` is the odd one out in that controller: most of
`/Library/VirtualFolders/*` takes an `id`, and this one takes `name`. Passing an id
returns a bare `HTTP 400: Error processing request.` with nothing to say why. Verified
against Jellyfin 10.11.4, 2026-08-13: `name=…&newName=…` → `204`.

Renaming **changes the library's ItemId**, because Jellyfin derives it from the name:
`98e74a0c…` became `34f331a8…` — which was the *deleted* library's id, since that one
had the name we renamed to. Consequences, checked rather than assumed:

* `find_library()` matches on **path**, so ytstream is unaffected. `doctor` reports
  `library 'YouTube'` and `refresh-metadata` re-read all 257 NFOs with **0 proxy
  requests** immediately after the rename.
* Anything that ever caches a Jellyfin ItemId across a rename will break. Nothing
  does today. Do not add one.

End state: one library, `YouTube`, at `/disks/Plex/_ytstream`, 11 series and 257
episodes. `/disks/Plex/YouTube` is no longer a library; its 9 files are still on disk.

## 21. Decommission complete, and the catalogue opened up — 2026-08-13

§12 is done. Verified rather than assumed:

| | |
|---|---|
| `youtube-automate.service` | inactive, disabled (unit left in place) |
| `tube.jihakuz.xyz` | nginx → `127.0.0.1:8086`, ytstream's admin UI |
| Admin password | set |
| Jellyfin | one library, `YouTube`, at `/disks/Plex/_ytstream` |
| `/opt/youtube-automate` | deleted — verified fully pushed to its bare repo first |
| `/disks/Plex/YouTube` | deleted, 2.0 GB reclaimed |
| Cron | `ytstream run` hourly, `update-ytdlp.sh` Mondays, inherited UUIDs |

`/var/lib/youtube-automate` (170 MB: old venv + `subs.db`) is still on disk. `subs.db`
is archived to `/var/lib/ytstream/`, so it can go whenever — it is the one thing that
was never in the git repo, being state rather than code.

### All 119 subscriptions approved

The 108 queued channels were approved rather than curated, because the brother's
subscription list is defined as the source of truth (§4): hand-picking a subset means
he subscribes to something and nothing happens, which is the feature not working.
Reversible per channel with `ytstream unsubscribe`.

`approve --all`: **108 added, 0 failed, 70 s.**

## 22. Serving a partial file to Jellyfin is worse than making it wait — 2026-08-13

§19 capped the wait for a complete mux at 12s and streamed the partial file after
that, to stop long videos failing to start. That was the wrong trade, and a real
play found it within the hour: a 2-hour upload took **3 minutes to start, played 6
seconds, and stalled permanently**.

### What Jellyfin actually did

```
ffmpeg -analyzeduration 200M -probesize 1G -i http://127.0.0.1:8099/watch/JYZYnkXMxdU
       -codec:v:0 libx264 -preset veryfast -maxrate 4830438 ... -f hls
```

It **transcoded**, after dragging up to a gigabyte through the proxy to probe. The
Jellyfin log closes it out: `Playback stopped ... Stopped at "6016" ms`.

### Why — and it is the container, not the client

A fragmented MP4 written with `empty_moov` **has no duration in its header**. The
only way to obtain one is to sum every fragment, so probing a *growing* file means
reading all of it — hence `-probesize 1G`. Jellyfin therefore cannot establish
duration, codec or bitrate, stops trusting direct play, and transcodes a stream it
also cannot seek.

The same video once complete, via `PlaybackInfo`:

| | |
|---|---|
| `SupportsDirectPlay` | **True** |
| `RunTimeTicks` | 1278 s (database says 1279) |
| `Bitrate` | 3,290,533 |
| Streams | h264 720p 3.16 Mbps + aac 128 kbps |

The transcode had been targeting `-maxrate 4830438` — **4.83 Mbps, above the
source's 3.29**. Jellyfin was re-encoding a stream that already fitted, purely
because it could not measure it. ffmpeg patches the real duration into the moov on
close, which is what makes the complete file cheap to probe and safe to direct-play.

### The decision

`FIRST_BYTE_GRACE` now defaults to **infinite** — wait for the complete mux,
bounded by `--wait-timeout` (raised to 600s to cover a 2-hour, 2.4 GB upload). A
cold long video is slow to start, and that is accepted deliberately:

* the fetch outlives the request, so a **retry is instant**;
* the failure is a stall the user can retry, not a runaway transcode that wastes a
  gigabyte of transfer and cannot succeed.

Both failure modes are now recorded at the constant, in the order they were
measured, so nobody re-tries the 12s cap and rediscovers this. `/healthz` reports
`mode: wait-for-complete` and `first_byte_grace_s: null` — `null` rather than a
number because `json.dumps` renders an infinite float as `Infinity`, which is not
valid JSON.

### Still unresolved

**A cold long video does not start on the first press.** The grace was the wrong
fix for it, and the right one is not in yet. The options, none free:

1. **Pre-warm** the newest episode per channel after each run — turns the common
   case ("watch the latest") instant, at the cost of fetching videos nobody asked
   for, which is the premise the whole design rejects. Bounded, though: 119 videos.
2. **Faster pull.** `yt-dlp -o -` uses a single connection; measured 4.3 MB/s on a
   525 MB upload. If concurrent ranged fetches work on these formats, a 2-minute
   wait could become 20 seconds and the problem mostly disappears.
3. **Give Jellyfin the metadata up front** so it never probes: duration, codec and
   bitrate are all known before a byte is fetched. `<streamdetails>` in the NFO was
   measured not to affect scan probing, but its effect on `PlaybackInfo`
   specifically has not been tested — and that is the one that decides transcoding.

Option 2 is the one to measure first: it is the only one that costs nothing and
helps every case.