tom/ytstream
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Phase 0: rolling 30-day window, delete on unsubscribe, shared password

Browse Source 
Phase 0 is done apart from one external dependency, and what it measured
changed the shape of the plan more than the three answers it was meant to
confirm.

Measured the actual upload rates from the live UULF feeds rather than guessing
from catalogue size, and the numbers make the whole scale section boring. Pitch
Side publishes 0.60 long-form videos a day, The Pyramid Podcast 0.77 — so a
30-day window is about 20 episodes per channel and roughly 400 across a
20-channel library, not the 20,000 the previous draft braced for. The UULF feed
is also doing more work than expected: it excludes 60-74% of what these channels
publish, all of it Shorts and livestreams nobody wants as Jellyfin episodes.

A side effect worth recording: Pitch Side's 15-entry UULF feed spans 23.3 days,
because the feed holds 15 long-form items rather than 15 uploads. Free RSS
therefore covers most of a 30-day retention window on its own, which narrows
what the API is for. It still earns its place — RSS carries no duration, and
subscription reading has no alternative at all — but the honest version is that
it now buys correctness far more than it buys safety, and the plan says so
instead of keeping the more flattering 4-hours-of-yt-dlp argument.

Three decisions folded in.

Retention is a rolling 30 days, superseding the earlier "3 months or 300
videos". Those answered different questions — backfill depth versus retention —
and holding both would mean backfilling 90 days and deleting two thirds of it on
the next sweep. One number now governs both ends, so the library cannot grow.
This resurrects reap.py, which the previous draft deleted, and makes tombstones
load-bearing in two ways: without them the next poll re-materialises everything
the sweep just deleted, and episode ordinals for a given day would shift as
videos disappear. aged_out is deliberately never revivable, or raising
retention_days would resurrect months of episodes into Jellyfin as new.

Unsubscribing now deletes the channel rather than deactivating it, which is
reasonable when rebuilding costs one API page and ~20 files. The consequence is
that the removal-detection rules stop being precautionary and become the only
thing standing between a transient 403 and a wiped library, so the plan now says
that explicitly next to them.

Auth reverts to the single shared admin_password_hash, matching every other
service on susan. web/auth.py carries over untouched. What that gives up —
independent revocation, and knowing who approved what — is named once and then
dropped.

Added tools/verify_api.py so the remaining Phase 0 work is one command. It runs
all three API checks, paginates the subscription list, prints the
subsync_max_new value derived from totalResults, distinguishes 403
subscriptionForbidden from every other failure with the fix in the message, and
exits non-zero so it can be gated on. The ISO-8601 duration parser is
unit-checked.

Blocked on: there is no Google API key anywhere on this machine (searched
/opt/*, /home/susan, the settings table and the config trees for AIza-shaped
strings). Creating one needs a browser and a Google account. That plus the
brother unchecking "Keep all my subscriptions private" are the two external
prerequisites, and §16 records exactly what each unverified assumption costs if
it turns out false.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Tom Flux
2026-08-12 15:37:53 +01:00
co-authored by Claude Opus 5
parent 8cca4fe1e2
commit 339232c7f9
2 changed files with 438 additions and 132 deletions
Download Patch File Download Diff File Expand all files Collapse all files
plan.md
+213 -132
View File
@@ -30,8 +30,8 @@ presses play — fetched on demand by a local proxy, held on tmpfs, dropped when
that account is the source of truth for what exists (§4)
- **Automatic subscription pickup from a YouTube account's public subscription list** (§4) — the
new requirement
- A bounded back catalogue per channel — **3 months or 300 videos, whichever comes first** — that
then grows forward indefinitely (§3, §5)
- A **rolling 30-day window** per channel: videos older than that are removed from disk, so the
library stays small and bounded forever (§5)
- Just-in-time streaming via the proxy, with per-video caching
- NFO metadata, episode thumbnails, channel poster/fanart
- An admin UI for subscriptions, status and settings
@@ -72,7 +72,7 @@ So: copy the tree, then add, delete and replace.
| `doctor.py` | **Lift, extend** | Add checks for API key validity, proxy health, `.strm` orphan count. |
| `ytdlp.py` | **Lift, narrow** | Only the proxy calls yt-dlp now. |
| `download.py` (330 lines) | **Delete** | Replaced by `strm.py`, which writes a text file. |
| `reap.py` (150 lines) | **Delete, replace** | Reaping existed to reclaim disk. There is no disk to reclaim (§3). Replaced by a much smaller "retire a video whose channel is gone" path. |
| `reap.py` (150 lines) | **Lift, simplify** | Originally slated for deletion — the 30-day window (§5) brings it back. Same aging-out logic and tombstone semantics, minus the disk-cap machinery, and deleting a `.strm` instead of a 500 MB mp4. |
| `runner.py` | **Rewrite** | The run loop changes shape: sync subscriptions → poll → materialise → refresh. |
| — | **New: `strm.py`** | Writes `.strm` + `.nfo` + thumbnail for one video. |
| — | **New: `subsync.py`** | The brother-subscription puller (§4). |
@@ -98,24 +98,28 @@ That changes the arithmetic completely:
| Job | Method | Calls | Units |
|---|---|---|---|
| Sync one account's subscriptions, hourly | `subscriptions.list` | 24/day (1 page) | 24 |
| Backfill one channel to the §5 bound (≤300 videos) | `playlistItems.list` @50 | ≤6 | ≤6 |
| Durations for those videos | `videos.list` @50 ids | ≤6 | ≤6 |
| Sync subscriptions, hourly | `subscriptions.list` | 24/day (1 page) | 24 |
| Backfill one channel's 30-day window (~20 videos) | `playlistItems.list` @50 | 1 | 1 |
| Durations for those videos | `videos.list` @50 ids | 1 | 1 |
| Steady-state incremental discovery | **RSS feed** | — | **0** |
| Durations for the day's new videos | `videos.list` @50 ids | 1 | 1 |
A 20-channel initial build costs at most **240 units** — under 3% of one day's quota, once. Steady
state costs under 100 units/day. **We will never come close to the limit**, provided we never touch
`search.list` (100 calls/day, and we have no use for it).
A 20-channel initial build costs about **40 units**. Steady state is **~25 units/day**, nearly all of
it the hourly subscription poll. Against 10,000/day this is not a constraint worth thinking about,
provided we never touch `search.list` (100 calls/day, and we have no use for it).
### Why this matters more than the quota
**It takes yt-dlp out of the cataloguing path entirely.** Today, `youtube-automate` needs a yt-dlp
extraction per video to learn an upload date during backfill, at roughly 2.3s each. For a 20-channel
build at the §5 bound that is 6,000 videos ≈ **4 hours of continuous requests** to YouTube from a
residential IP — the "scan storm" failure mode the original handover warned about, and it recurs
every time a channel is added. The API is a documented, keyed, quota-metered endpoint that does not
care about PO tokens, SABR, or client selection, and cannot be rate-limited by YouTube's anti-bot
heuristics.
**It takes yt-dlp out of the cataloguing path entirely.** `youtube-automate` needs a yt-dlp
extraction per video to learn an upload date whenever it backfills past the RSS feed, at roughly 2.3s
each. That is the "scan storm" failure mode the original handover warned about: requests to YouTube
from a residential IP, in bulk, for videos nobody asked to watch.
Be honest about the size of the win now that the window is 30 days: at ~20 videos per channel, the
yt-dlp route would be ~400 extractions ≈ 15 minutes, not the 4 hours a 90-day window implied. The
API is still the right answer — it is documented, keyed, quota-metered, indifferent to PO tokens and
SABR, and cannot be rate-limited by YouTube's anti-bot heuristics — but it is now buying
*correctness* (exact dates, real durations) far more than it is buying *safety*.
So the split becomes:
@@ -282,8 +286,10 @@ exactly why §4.4 refuses to act on a single bad response.
Two failure modes here are genuinely destructive, and both are cheap to defend against.
**Runaway additions.** If he has 400 subscriptions, the first sync queues 400 channels — at the §5
bound that is 120,000 Jellyfin episodes, which will not end well. So:
**Runaway additions.** If he has 400 subscriptions, the first sync queues 400 channels — ~8,000
episodes at the measured rate in §5, and 400 backfills' worth of API calls and file writes in one
run. The 30-day window makes this survivable where the earlier 90-day draft did not, but it is still
not something to discover by accident. So:
- A sync that would add more than **`subsync_max_new`** channels adds none of them. It records them
as `pending_approval`, alerts, and waits for a click in the admin UI.
@@ -307,12 +313,17 @@ basis would be unrecoverable in any pleasant way. So:
indistinguishable from a broken one in consequence, and we prefer the harmless reading.
- A channel missing from an otherwise-healthy response increments `missing_syncs`. Only at
**`subsync_missing_threshold`** (default **3** consecutive syncs, so ~3 hours) is it unsubscribed.
- Unsubscribing **does not delete anything on disk**. It sets the channel `inactive`: polling stops,
the tree stays, and re-subscribing on YouTube restores it with no refetching. Actual deletion is a
separate, explicit, human-initiated admin action. `.strm` files cost ~50 bytes each; there is no
pressure to reclaim anything, so there is no reason to ever delete automatically.
- **Unsubscribing deletes the channel** — its directory, its `.strm` files, its NFOs and artwork, and
its rows. Re-subscribing rebuilds it from scratch, which under the 30-day window (§5) is one
`playlistItems.list` page and ~20 files: cheap enough that keeping a dormant copy around would be
the more complicated choice. The Jellyfin series disappears, taking its watch state with it.
- `source = 'manual'` channels are exempt from all of the above.
Note how the two halves interact: because the *action* is now destructive and irreversible-ish, the
*detection* rules above are what stand between a transient 403 and a wiped library. They are not
belt-and-braces. Do not weaken the three-sync threshold to make testing more convenient — make the
threshold configurable and set it to 1 in tests.
**Ordering:** the sync runs *before* the poll in the same run, so a channel added at 14:00 has its
catalogue built in the same pass.
@@ -323,82 +334,102 @@ Follow it. A sync that hits 403, trips the add cap, or accumulates failures must
the check goes red — a silently-broken subscription mirror is the worst possible outcome, because
nothing appears to be wrong until someone asks why a channel never showed up.
### 4.6 The brother gets admin access, so auth becomes multi-user
### 4.6 The brother gets admin access, on the existing shared password
This is the right call — the approval queue (§4.4) and the "channel went missing" alerts are about
*his* subscriptions, so they should be his to action rather than landing on Tom every time. But it
does mean a change: `youtube-automate` has exactly one credential, `admin_password_hash` in the
`setting` table. Sharing it would mean neither party can have their access revoked or their password
changed independently, and the approval log would not record who approved what.
He gets access, which is right — the approval queue (§4.4) and the "channel went missing" alerts are
about *his* subscriptions, so they should be his to action rather than landing on Tom every time.
So replace the single setting with a minimal `user` table (§7): `username`, `password_hash`
(scrypt, same parameters), `created_at`, `last_login_at`. Two rows. **No roles, no permissions
system** — both accounts can do everything, which is correct for two brothers and one media server,
and the alternative is a permission model nobody will maintain.
**Auth does not change.** One shared `admin_password_hash` in the `setting` table, exactly as
`youtube-automate` already does it, matching how every other service on susan is shared. An earlier
draft proposed a two-row `user` table; it is not worth the code. The two things given up are worth
naming once and then forgetting: neither party's access can be revoked without changing the other's
password, and the approval log records *that* something was approved rather than by whom.
Everything else in `web/auth.py` carries over unchanged, and it is adequate for an
internet-facing login: scrypt hashing and per-address failed-login throttling that reads
`X-Forwarded-For` (which nginx must set — it already does, see §12 step 4).
Record `approved_by` on `pending_approval` resolutions. It costs one column and it answers "why is
this channel here" six months later.
`web/auth.py` therefore carries over **unchanged**, and it is adequate for an internet-facing login:
scrypt hashing and per-address failed-login throttling that reads `X-Forwarded-For` (which nginx must
set — it already does, see §12 step 4).
---
## 5. Scale — bounded backfill, unbounded growth
## 5. Scale — a rolling 30-day window
Measured during the PoC: **Pitch Side has 1,249 videos.** The `youtube-automate` DB holds 15 of
them, because a 9-day retention window and 1.3 TB of free disk is what bounded it. `.strm` files
remove that bound entirely, so the bound is now a policy choice, and the choice is:
> **`retention_days = 30`.** A video older than 30 days has its `.strm`, `.nfo` and thumbnail
> deleted and its DB row tombstoned. Backfill on subscribe reaches back the same 30 days. One
> number governs both ends, so the library cannot grow.
> **Backfill 3 months or 300 videos per channel, whichever comes first.**
This supersedes the earlier "3 months or 300 videos" answer, which was about *backfill depth* while
this is about *retention*; keeping both would mean backfilling 90 days and then immediately deleting
two thirds of it. If the intent was really "go back 3 months, keep everything", set
`retention_days = 90` and this section still holds — only the numbers move.
Both settings exist (`backfill_max_days` = 90, `backfill_max_videos` = 300) and whichever is reached
first stops the walk. The two are well matched in practice: a channel uploading 2–3 times a day —
which is what Pitch Side's episode numbering implies — produces roughly 270 videos in 90 days, so
neither bound dominates. A weekly uploader gets ~13 videos and is bounded by time; a daily-podcast
firehose gets 300 and is bounded by count. That is the right behaviour in both directions.
### What this actually costs, measured
**This bounds the *initial* build, not the library.** Nothing ages out — `.strm` files cost ~50
bytes and there is no disk to reclaim, so aging out would destroy metadata for no gain. The
consequence is honest and worth stating: the library **grows forever from the subscribe date**. At
20 channels averaging 2 uploads/day that is ~15,000 new episodes a year, so the 20,000-episode
question in point 1 below is deferred by about eighteen months, not answered. The lever, if it ever
bites: a `hidden` video state that removes the `.strm` and `.nfo` from disk while keeping the DB row,
so the catalogue stays re-materialisable. **Do not build it now** — build the measurement that tells
us when we need it.
The upload rates matter more than the total catalogue size, so I measured them from the live feeds
on 2026-08-12 rather than guessing:
**Initial build:** ~20 channels × ≤300 = **≤6,000 episodes, ≤12,000 files.** Comfortable.
| Channel | UULF (long-form) | UC feed (everything) | UULF filters |
|---|---|---|---|
| Pitch Side | 0.60/day → **~18 per 30d** | 2.33/day → ~70 per 30d | 74% |
| The Pyramid Podcast | 0.77/day → **~23 per 30d** | 1.91/day → ~57 per 30d | 60% |
1. **Jellyfin scan cost is still unmeasured**, and is the number that decides everything above.
Every episode is a `.strm` + a `.nfo` + a thumbnail. Normal scans were measured to make **0 media
probes** (FINDINGS §6), which is the thing that makes this viable at all — but stat-ing and
NFO-parsing 12,000 files on a Westmere with the library DB on spinning disk is its own cost.
**Measure in Phase 3** by building one channel to the bound, timing a full scan, and recording
seconds-per-1,000-episodes here. That single figure sizes both the initial build and the growth
runway.
2. **`replaceAllMetadata` remains a catastrophe rather than an annoyance.** It is the one operation
verified to probe media, and at 6,000 items it means 6,000 cold starts. The cold-start rate
limiter (20/hour) contains the damage to YouTube's side, but the library-side result is 6,000
items whose metadata got wiped and not re-derived. Defences, all of them:
So a 30-day window is roughly **20 episodes per channel**, and 20 channels is **~400 episodes,
~1,200 files.** That is two orders of magnitude below the 20,000 the previous draft was braced for,
and it makes the entire scale section boring — which is the point. Pitch Side's 1,249-video back
catalogue simply never enters the library.
It also confirms the UULF feed is doing real work: it filters 60–74% of what the channel publishes,
and that Shorts-and-livestreams majority is exactly what nobody wants as Jellyfin episodes.
### A useful side effect: RSS nearly covers the whole window
The UULF feed returns 15 entries, and for Pitch Side those 15 span **23.3 days** — because the feed
holds 15 *long-form* items, not 15 uploads. At these rates a single free RSS fetch covers most of a
30-day window on its own.
That narrows what the API is for (§3): subscription reading, durations, and topping up the few days
RSS does not reach on a fresh subscribe. It does not eliminate it — RSS carries no duration at all,
and a busier channel would truncate sooner — but nobody should be surprised when the API turns out
to be handling a few dozen calls a day rather than thousands.
### Aging out has two consequences worth stating
1. **Tombstones are mandatory, not optional.** Delete a `.strm` and the next poll finds the video in
the feed again and re-materialises it, forever. `youtube-automate` already solves this: the DB row
survives deletion in a terminal state (`aged_out`) and discovery skips it. Lift that behaviour
exactly; it is the single most likely thing to get wrong here.
2. **Jellyfin loses watch state for aged-out episodes.** Removing the media file removes the item,
and with it play counts and resume positions. For a 30-day window on a subscription feed that is
acceptable — this is a "what's new" library, not an archive — but it should be a deliberate choice
rather than a surprise. Anything worth keeping permanently wants a different tool (§1).
1. **Jellyfin scan cost is now a non-issue, but still worth one measurement.** Every episode is a
`.strm` + a `.nfo` + a thumbnail. Normal scans were measured to make **0 media probes** (FINDINGS
§6), which is what makes this viable at all. At ~1,200 files the stat-and-parse cost will not be
noticeable. **Record it in Phase 3 anyway**, as seconds-per-1,000-episodes, because it is the
number that would let anyone judge a future change to `retention_days` without re-deriving it.
2. **`replaceAllMetadata` remains the one real hazard.** It is the one operation verified to probe
media, and at ~400 items it means ~400 cold starts. The cold-start rate limiter (20/hour) contains
the damage to YouTube's side, but the library-side result is items whose metadata got wiped and
not re-derived. Defences, all of them:
- the proxy 503s past the budget (built, tested — `test_limits.py`)
- `ytstream` never itself issues a refresh with `replaceAllMetadata=true`
- the admin UI documents "do not click Replace all metadata" in the place where someone would be
tempted to
- **NFOs are the source of truth**, so recovery is a re-materialise pass over the tree, not a
re-fetch from YouTube. Make sure that pass exists and is one CLI command.
3. **Episode numbering holds.** `MMDD*10 + ordinal` clamps at 10 uploads/channel/day, computed
against the DB rather than the batch, so it is stable across a 300-video backfill. Season =
upload year means a 3-month window usually spans one season and occasionally two (a January
subscribe reaches back into the previous year), which is correct and needs no special handling.
3. **Episode numbering holds, and tombstones are what make it hold.** `MMDD*10 + ordinal` clamps at
10 uploads/channel/day and is computed against the DB, not the batch. Since aged-out rows stay in
the DB, the ordinal for a given day never shifts as videos are deleted — which is the second
reason the tombstones are load-bearing. Season = upload year means a 30-day window usually spans
one season and occasionally two, which needs no special handling.
4. **The `.strm` URL is baked into every file.** Changing the proxy's host or port means rewriting
all of them. Cheap (it is a tree walk) but it must be a supported CLI command, not a `sed`
someone invents under pressure. Settle the URL now: `http://127.0.0.1:8099/watch/<id>`. Jellyfin
is the only client that ever reads it, it runs on the same host, and 127.0.0.1 means the proxy is
unreachable from the network by construction.
5. **Backfill must be resumable.** Quota is a non-issue but a crash halfway through 12,000 files
needs to resume, not restart. Track progress per channel (`channel.backfill_cursor`), commit per
page of 50.
5. **Backfill stays resumable** (`channel.backfill_cursor`, commit per page of 50) even though a
30-day backfill is now one or two pages. It costs a column and it is the difference between a
crashed first sync resuming and starting over across 20 channels.
---
@@ -440,9 +471,9 @@ noticeable.
### 6.2 `ytstream-admin` — systemd service
Lifted from `youtube_automate/web/`. scrypt password hashes, now per user in the `user` table (§4.6)
rather than one shared setting, per-address failed-login throttling reading `X-Forwarded-For`, behind
nginx on `127.0.0.1:8086` (8085 is `youtube-automate`'s during the overlap).
Lifted from `youtube_automate/web/` unchanged: shared-password scrypt auth (§4.6), per-address
failed-login throttling reading `X-Forwarded-For`, behind nginx on `127.0.0.1:8086` (8085 is
`youtube-automate`'s during the overlap).
New routes beyond what exists: subscription **sources** (add/remove a mirrored YouTube account, show
its last sync, its failure count, its `pending_approval` queue with an approve/reject action), the
@@ -471,17 +502,25 @@ subscribed, and when someone explicitly asks for a deeper catalogue.
`/var/lib/ytstream/ytstream.db`. Start from `youtube-automate`'s schema — `channel`, `video`,
`setting` are the right shape — with these changes:
**`channel`** — add `backfill_cursor TEXT` (resumable backfill), `backfill_max_videos INTEGER` and
`backfill_max_days INTEGER` (per-channel overrides of the §5 bounds; NULL = use the global setting),
`active INTEGER NOT NULL DEFAULT 1` (§4.4 soft-delete), `uploads_playlist TEXT` (whether UULF or UU
won, §3). Drop `retention_days`.
**`channel`** — add `backfill_cursor TEXT` (resumable backfill) and `uploads_playlist TEXT` (whether
UULF or UU won, §3). **Keep `retention_days`** as a per-channel override of the 30-day window (NULL =
use the global setting) — `youtube-automate` already has this column and the `rescan` semantics that
go with it. No `active` column: unsubscribing now deletes rather than deactivating (§4.4).
**`video`** — drop `size_bytes`, `downloaded_at`, `deleted_at`, `attempts`, `last_error`; none of
them mean anything when there is no download. Keep `duration` (now from `videos.list`, and it feeds
`<durationinseconds>`). `state` collapses to `{listed, materialised, skipped_short, skipped_live,
hidden}` — no `pending`/`downloading`/`failed`, because materialising a `.strm` cannot
meaningfully fail. Add `published_at TEXT` (exact RFC-3339 from the API) alongside the existing
date-only `upload_date` that naming uses.
**`video`** — drop `size_bytes`, `downloaded_at`, `attempts`, `last_error`; none of them mean anything
when there is no download. **Keep `deleted_at`** — it is the aging-out timestamp now (§5). Keep
`duration` (from `videos.list`; it feeds `<durationinseconds>`). Add `published_at TEXT` (exact
RFC-3339 from the API) alongside the date-only `upload_date` that naming uses.
`state` becomes `{listed, materialised, skipped_short, skipped_live, skipped_old, aged_out}`. No
`pending`/`downloading`/`failed`, because materialising a `.strm` cannot meaningfully fail. The two
terminal states matter and are not the same thing:
- **`skipped_old`** — discovered already outside the window; never materialised. `youtube-automate`'s
`rescan` can revive these if `retention_days` is raised.
- **`aged_out`** — was materialised, then deleted by the sweep. **Never revived**, or raising
`retention_days` would resurrect months of episodes into Jellyfin as "new". These are tombstones
and discovery must skip them unconditionally (§5).
**`channel`, continued** — two more columns instead of the join table an earlier draft had (§4.3):
`source TEXT NOT NULL` (`'youtube'` for synced, `'manual'` for the pinning escape hatch) and
@@ -492,19 +531,16 @@ date-only `upload_date` that naming uses.
`setting` keys, because these are fields of one thing and they change together.
**`pending_approval`** — new, the §4.4 add-cap and first-sync import queue: `source`, `channel_id`,
`title`, `seen_at`, `resolved_at`, `resolution` (`approved`/`rejected`), `approved_by`.
**`user`** — new (§4.6), replacing the single `admin_password_hash` setting: `username` (PK),
`password_hash` (scrypt, same parameters as `youtube-automate`), `created_at`, `last_login_at`. Two
rows, no roles.
`title`, `seen_at`, `resolved_at`, `resolution` (`approved`/`rejected`).
**`setting`** — carry over what still applies (`jellyfin_url`, `jellyfin_api_key`,
`pot_provider_url`, `max_height`, `min_duration_seconds`, `session_secret`, `last_run_at`). Drop
`retention_days`, `disk_cap_gb`, `write_subs`, `sub_langs`, `sponsorblock_mark`, `max_attempts`,
`backfill_days`, `admin_password_hash` — download-era concepts plus the credential that moved to
`user`. Add `youtube_api_key`, `subsync_max_new` (25, provisional — §4.4),
`subsync_missing_threshold` (3), `proxy_base_url` (`http://127.0.0.1:8099`), `backfill_max_videos`
(**300**), `backfill_max_days` (**90**).
`pot_provider_url`, `max_height`, `min_duration_seconds`, `admin_password_hash`, `session_secret`,
`last_run_at`), including **`retention_days`** — now **30**, and governing both the aging-out sweep
and the backfill reach (§5). Drop `disk_cap_gb`, `write_subs`, `sub_langs`, `sponsorblock_mark`,
`max_attempts`, `backfill_days` — download-era concepts. Add `youtube_api_key`, `subsync_max_new`
(25, provisional — §4.4), `subsync_missing_threshold` (3), `proxy_base_url`
(`http://127.0.0.1:8099`), and `backfill_max_videos` (**300**) purely as a runaway guard on a channel
that turns out to upload 50 times a day.
**No migration from `subs.db`.** It holds 2 channels and 18 videos. Re-subscribe by hand and let
the backfill do the rest; a migration script would be more code than the data is worth.
@@ -613,8 +649,8 @@ Secrets live in `/var/lib/ytstream/ytstream.db`, mode `0640` `susan:automation`,
through the admin UI. Never in the repo, never in a systemd unit, never in a cron line. This matches
`youtube-automate`, where `jellyfin_api_key` and `session_secret` already live that way.
Three kinds: `youtube_api_key` (new) and `jellyfin_api_key` in `setting`; `session_secret` in
`setting`; and two scrypt password hashes in `user` (§4.6).
Four keys, all in `setting`: `youtube_api_key` (new), `jellyfin_api_key`, `session_secret`, and
`admin_password_hash` — one shared admin password, as everywhere else on susan (§4.6).
The YouTube API key is restricted to the YouTube Data API v3 and has read-only reach over public
data. Worst case on leak is quota exhaustion; rotation is a two-minute job in the console.
@@ -654,19 +690,16 @@ is the thing that keeps playback working as YouTube changes.
Each phase ends in something checkable. Do not start the next one until it does.
**Phase 0 — verify the assumptions the plan rests on, and get the one number it is missing.**
~15 minutes, before any code. `git init --bare /disks/git-repos/ytstream.git`. Get the API key
(§4.1). Then three curls:
**Phase 0 — verify the assumptions the plan rests on.** Mostly **done, 2026-08-12** — see §16 for
what was verified and what is still blocked. Remaining: paste the API key in and run
1. `subscriptions.list` on `UCPcTWaLV8zwx4WP4QExHj4Q` returns 200 with items — **and record
`pageInfo.totalResults`**, which is what sets `subsync_max_new` (§4.4). This is a hard blocker:
there is no fallback that does not involve him doing something (§4.2).
2. `playlistItems.list` accepts a UULF playlist id — or does not, and we take the documented `UU`
fallback plus duration filtering (§3).
3. `videos.list` returns `contentDetails.duration` for a batch of 50 ids.
```sh
python3 /opt/ytstream/tools/verify_api.py --key AIza...
```
→ *Done when: all three answers are written back into this file as verified facts or corrections,
and `subsync_max_new` has a real value.*
which performs all three API checks, costs ~5 quota units, prints the `subsync_max_new` value, and
exits non-zero if anything fails.
→ *Done when: `verify_api.py` reports ALL PASS and its numbers are written into §4.1 and §16.*
**Phase 1 — skeleton and lift.** Fork the tree, new package name, new DB path, new schema (§7),
lifted modules and their tests passing. No new behaviour.
@@ -677,19 +710,21 @@ test, write `deploy/deploy.sh`, operator runs it.
→ *Done when: `systemctl status ytstream-proxy` is active after a reboot, `/healthz` answers, and
Jellyfin direct-plays a cold video end to end.*
**Phase 3 — catalogue.** `api.py`, `strm.py`, the 3-month/300-video bounded resumable backfill, the
hourly run. Build **Pitch Side alone** and **time a Jellyfin scan** — the §5 measurement. Worth
doing twice, once at the 300-video bound and once unbounded at 1,249, since the second gives the
seconds-per-1,000-episodes figure that sizes the growth runway for free.
→ *Done when: the bounded episode count is visible with correct titles, dates, durations and
thumbnails; scan time per 1,000 episodes is recorded in §5; and one of them plays.*
**Phase 3 — catalogue and retention.** `api.py`, `strm.py`, the 30-day bounded resumable backfill, the
aging-out sweep with its tombstones, the hourly run. Build **Pitch Side alone** (expect ~18 episodes)
and time a Jellyfin scan for the §5 record.
→ *Done when: the expected episode count is visible with correct titles, dates, durations and
thumbnails; one of them plays; scan time per 1,000 episodes is recorded in §5; and — the part most
likely to be wrong — a video forced past the window is deleted from disk, and the **next two polls do
not bring it back**.*
**Phase 4 — subscription sync.** `subsync.py`, the first-sync bulk import, the add cap, the
missing-threshold, the `user` table and two accounts (§4.6), the admin routes, the healthchecks UUID.
→ *Done when: the brother logs in with his own credentials and approves the first import; then he
subscribes to a new channel on YouTube and within an hour it is a series in Jellyfin with episodes
that play, nobody having touched the admin UI. Plus: a forced 403 and a forced empty response both
leave the DB untouched and turn the check red.*
missing-threshold, channel deletion on unsubscribe, the admin routes, the healthchecks UUID.
→ *Done when: he approves the first import; then he subscribes to a new channel on YouTube and within
an hour it is a series in Jellyfin with episodes that play, nobody having touched the admin UI. Then
the destructive half: he unsubscribes, and after three syncs the channel and its tree are gone. Plus
a forced 403 and a forced empty response, each of which must leave the DB untouched and turn the check
red — test these before trusting the deletion path, not after.*
**Phase 5 — cut over.** §12 steps 1–2, run for a week.
→ *Done when: nothing has broken and nobody has used the old library.*
@@ -703,15 +738,19 @@ leave the DB untouched and turn the check red.*
1. **Account to mirror:** `@cflux1030` → `UCPcTWaLV8zwx4WP4QExHj4Q` ("C Flux"), resolved and
recorded in §4. **Sole source of truth** — Tom's own subscriptions are not part of this at all,
which removed the multi-source claim model from §4.3 and put the weight on §4.4 instead.
2. **Catalogue depth:** 3 months or 300 videos, whichever comes first. §5 rewritten around it, and
both bounds are settings with per-channel overrides. Note the consequence recorded there: this
bounds the initial build, not the library, which grows forward indefinitely.
3. **`subsync_max_new`:** to be set from `pageInfo.totalResults` on the day-one
`subscriptions.list` call, since that is the only way to learn how many channels he follows.
Provisional 25; rule of thumb `max(10, ceil(totalResults × 0.2))` (§4.4). **The one open number
in this plan** — Phase 0 closes it.
4. **Brother gets admin access:** yes, which turns the single shared password into a two-row `user`
table (§4.6) and adds `approved_by` to the approval log.
2. **Retention:** a **rolling 30-day window** — anything older is deleted from disk and tombstoned
(§5). This supersedes the earlier "3 months or 300 videos" answer, which was about backfill depth;
holding both would mean backfilling 90 days and deleting two thirds of it immediately. If the
intent was "reach back 3 months and keep it", `retention_days = 90` is a one-line change and §5
still holds. Measured consequence: **~400 episodes total**, not 20,000.
3. **`subsync_max_new`:** still open — set from `pageInfo.totalResults` on the first
`subscriptions.list` call, which `tools/verify_api.py` prints. Provisional 25; rule of thumb
`max(10, ceil(totalResults × 0.2))` (§4.4).
4. **Brother gets admin access:** yes, on the **existing shared password** — no `user` table, no
per-account credentials (§4.6). `web/auth.py` carries over untouched.
5. **Unsubscribe deletes the channel** rather than deactivating it, since re-subscribing is one API
page and ~20 files under the 30-day window (§4.4). This makes the removal *detection* safeguards
load-bearing rather than precautionary.
---
@@ -739,3 +778,45 @@ Things already paid for once. All of these are verified.
spend an afternoon trying to scrape it; the API is the only route (§4).
- **`pkill -f 'ytstream.py'` kills the shell that runs it**, because the command string contains its
own pattern. Bracket it: `pkill -f 'ytstrea[m].py'`.
---
## 16. Phase 0 results — 2026-08-12
### Done
- **Bare repo initialised.** `/disks/git-repos/ytstream.git` existed as an empty directory; it is now
a real `--bare --shared=group` repo with `HEAD` → `refs/heads/main`, matching
`youtube-automate.git`. `/opt/ytstream` is a checkout with `origin` pointed at it.
- **The mirrored channel is resolved.** `@cflux1030` → `UCPcTWaLV8zwx4WP4QExHj4Q` ("C Flux", 55
subscribers). No `channels.list` call needed at build time.
- **There is no scraping fallback** — `/@cflux1030/channels` serves the Home tab with zero channel
ids (§15). The privacy checkbox is mandatory.
- **UULF feeds work for both existing channels**, and the filtering is substantial — 60–74% of what
these channels publish is Shorts or livestreams that UULF correctly excludes.
- **Upload rates measured**, which is what sized §5: ~18–23 long-form videos per channel per 30 days,
so ~400 episodes for a 20-channel library. Pitch Side's UULF feed spans **23.3 days in 15 entries**,
meaning RSS alone nearly covers the retention window.
- **`tools/verify_api.py` written and self-tested** (ISO-8601 duration parser unit-checked against six
cases; argparse and import verified). It runs all three API checks in one command.
### Blocked on one thing only
**There is no Google API key on this machine** — I searched for `AIza`-shaped strings across
`/opt/*`, `/home/susan`, the settings table and the config trees, and there is none. Creating one
needs a browser and Tom's Google account (§4.1, ~5 minutes, free, no billing). Until then these three
remain unverified:
| # | Assumption | Consequence if it fails |
|---|---|---|
| 1 | `subscriptions.list?channelId=` returns his subscriptions | **Feature is impossible as designed.** Falls back to §4.2, all of which need him to do something. |
| 2 | `playlistItems.list` accepts a `UULF…` playlist id | Take the documented `UU` fallback and filter Shorts/livestreams via `videos.list` duration + `liveStreamingDetails`. Costs one extra unit per 50. |
| 3 | `videos.list` returns parseable `contentDetails.duration` | No `<durationinseconds>` in NFOs without a yt-dlp extraction per video. Degrades, does not block. |
Only #1 is a genuine blocker, and it also carries the number that sets `subsync_max_new`. Two
prerequisites, both external: **Tom creates the API key**, and **his brother unchecks "Keep all my
subscriptions private"**. Then one command closes Phase 0:
```sh
python3 /opt/ytstream/tools/verify_api.py --key AIza...
```
tools/verify_api.py
Executable
+225
View File
@@ -0,0 +1,225 @@
#!/usr/bin/env python3
"""Phase 0 verification: everything the plan assumes about the YouTube Data API.
The plan (§13 Phase 0) rests on three answers that can only come from a live
key. This runs all three, costs about 5 quota units of the 10,000/day budget,
and prints the numbers to paste back into plan.md.
python3 tools/verify_api.py --key AIza...
Or, once the key is in the settings table:
python3 tools/verify_api.py --key "$(sqlite3 /var/lib/ytstream/ytstream.db \
"select value from setting where key='youtube_api_key'")"
Exit status is 0 only if all three checks pass, so this is safe to gate on.
"""
from __future__ import annotations
import argparse
import json
import re
import sys
import urllib.error
import urllib.parse
import urllib.request
API = "https://www.googleapis.com/youtube/v3"
# The account being mirrored — @cflux1030, resolved via yt-dlp on 2026-08-12.
BROTHER = "UCPcTWaLV8zwx4WP4QExHj4Q"
# A channel with a known-large back catalogue, used to exercise pagination.
SAMPLE_CHANNEL = "UCjCJ2LaOIsPzOoXUTMDI3wg" # Pitch Side
ISO8601 = re.compile(
r"^P(?:(\d+)D)?T?(?:(\d+)H)?(?:(\d+)M)?(?:(\d+)S)?$"
)
class ApiError(Exception):
def __init__(self, status, reason, body):
super().__init__(f"HTTP {status} {reason}")
self.status = status
self.reason = reason
self.body = body
def call(endpoint: str, key: str, **params) -> dict:
"""One API call. Raises ApiError with the parsed reason on 4xx/5xx."""
params["key"] = key
url = f"{API}/{endpoint}?" + urllib.parse.urlencode(params)
try:
with urllib.request.urlopen(url, timeout=30) as response:
return json.load(response)
except urllib.error.HTTPError as exc:
raw = exc.read().decode("utf8", "replace")
reason = ""
try:
errors = json.loads(raw).get("error", {}).get("errors", [])
reason = errors[0].get("reason", "") if errors else ""
except ValueError:
pass
raise ApiError(exc.code, reason, raw[:400]) from exc
def iso8601_seconds(text: str) -> int | None:
"""PT1H2M3S -> 3723. Returns None for anything unparseable."""
match = ISO8601.match(text or "")
if not match:
return None
days, hours, minutes, seconds = (int(g or 0) for g in match.groups())
return days * 86400 + hours * 3600 + minutes * 60 + seconds
# --------------------------------------------------------------------------
results: list[tuple[str, bool, str]] = []
def record(name: str, ok: bool, detail: str) -> None:
print(f" {'PASS' if ok else 'FAIL'} {name}")
for line in detail.splitlines():
print(f" {line}")
results.append((name, ok, detail))
def check_subscriptions(key: str, channel_id: str) -> None:
"""The one that decides whether the whole feature is possible."""
print("\n1. subscriptions.list on the mirrored account")
try:
page = call("subscriptions", key, part="snippet",
channelId=channel_id, maxResults=50)
except ApiError as exc:
if exc.status == 403 and exc.reason == "subscriptionForbidden":
record("subscriptions readable", False,
"403 subscriptionForbidden -- subscriptions are still PRIVATE.\n"
"Fix: youtube.com -> Settings -> Privacy -> uncheck\n"
'"Keep all my subscriptions private". There is no workaround;\n'
"the public HTML route no longer exists (plan.md §15).")
else:
record("subscriptions readable", False,
f"{exc} reason={exc.reason!r}\n{exc.body}")
return
total = (page.get("pageInfo") or {}).get("totalResults")
items = page.get("items") or []
titles = [i["snippet"]["title"] for i in items]
# Paginate so the recorded total is the real one, not just page 1.
seen = len(items)
token = page.get("nextPageToken")
pages = 1
while token and pages < 20:
page = call("subscriptions", key, part="snippet", channelId=channel_id,
maxResults=50, pageToken=token)
batch = page.get("items") or []
seen += len(batch)
titles.extend(i["snippet"]["title"] for i in batch)
token = page.get("nextPageToken")
pages += 1
suggested = max(10, -(-int(total or seen) // 5)) # ceil(total * 0.2)
record("subscriptions readable", True,
f"totalResults = {total}, fetched {seen} across {pages} page(s)\n"
f"-> set subsync_max_new = {suggested} [max(10, ceil(total*0.2))]\n"
f"first few: {', '.join(titles[:8])}"
+ (" ..." if len(titles) > 8 else ""))
def check_uploads_playlist(key: str, channel_id: str) -> None:
"""Does playlistItems.list accept the undocumented UULF playlist id?"""
print("\n2. playlistItems.list on UULF (long-form-only) vs UU (documented)")
outcome = {}
for kind, playlist_id in (("UULF", "UULF" + channel_id[2:]),
("UU", "UU" + channel_id[2:])):
try:
page = call("playlistItems", key, part="contentDetails",
playlistId=playlist_id, maxResults=5)
except ApiError as exc:
outcome[kind] = (False, f"{exc} reason={exc.reason!r}")
continue
items = page.get("items") or []
stamps = [i["contentDetails"].get("videoPublishedAt") for i in items]
exact = all(s and s.endswith("Z") and "T" in s for s in stamps)
outcome[kind] = (
bool(items) and exact,
f"{len(items)} items, videoPublishedAt exact={exact}, "
f"e.g. {stamps[0] if stamps else 'n/a'}, "
f"total={(page.get('pageInfo') or {}).get('totalResults')}",
)
uulf_ok, uulf_detail = outcome["UULF"]
uu_ok, uu_detail = outcome["UU"]
if uulf_ok:
record("uploads playlist usable", True,
f"UULF WORKS -> use it, existing Shorts/livestream filtering carries over\n"
f" UULF: {uulf_detail}\n UU: {uu_detail}")
elif uu_ok:
record("uploads playlist usable", True,
f"UULF REJECTED -> take the UU fallback and filter by duration +\n"
f"liveStreamingDetails (plan.md §3)\n"
f" UULF: {uulf_detail}\n UU: {uu_detail}")
else:
record("uploads playlist usable", False,
f"neither worked\n UULF: {uulf_detail}\n UU: {uu_detail}")
def check_durations(key: str, channel_id: str) -> None:
"""Durations for a batch of ids -- what feeds <durationinseconds>."""
print("\n3. videos.list durations for a batch of ids")
try:
listing = call("playlistItems", key, part="contentDetails",
playlistId="UU" + channel_id[2:], maxResults=50)
except ApiError as exc:
record("durations available", False, f"could not list ids: {exc}")
return
ids = [i["contentDetails"]["videoId"] for i in listing.get("items") or []]
if not ids:
record("durations available", False, "no video ids to test with")
return
try:
page = call("videos", key, part="contentDetails",
id=",".join(ids), maxResults=50)
except ApiError as exc:
record("durations available", False, f"{exc} reason={exc.reason!r}")
return
items = page.get("items") or []
parsed = [(i["id"], iso8601_seconds(i["contentDetails"].get("duration", "")))
for i in items]
bad = [vid for vid, secs in parsed if secs is None]
shorts = [vid for vid, secs in parsed if secs is not None and secs <= 120]
record("durations available", not bad and len(items) == len(ids),
f"asked for {len(ids)} ids in 1 call, got {len(items)} back, "
f"{len(bad)} unparseable\n"
f"{len(shorts)} of {len(items)} are <=120s (would be filtered as Shorts)\n"
f"sample: " + ", ".join(f"{v}={s}s" for v, s in parsed[:5]))
def main() -> int:
parser = argparse.ArgumentParser()
parser.add_argument("--key", required=True, help="YouTube Data API v3 key")
parser.add_argument("--brother", default=BROTHER,
help=f"channel id whose subscriptions to read (default {BROTHER})")
parser.add_argument("--sample", default=SAMPLE_CHANNEL,
help="channel id to exercise the playlist/duration calls against")
args = parser.parse_args()
print(f"Phase 0 verification against the live API (~5 quota units of 10,000/day)")
check_subscriptions(args.key, args.brother)
check_uploads_playlist(args.key, args.sample)
check_durations(args.key, args.sample)
failed = [name for name, ok, _ in results if not ok]
print()
if failed:
print(f"{len(failed)} of {len(results)} FAILED: {', '.join(failed)}")
return 1
print(f"ALL {len(results)} PASS -- paste the numbers above into plan.md §4.1 and §14")
return 0
if __name__ == "__main__":
sys.exit(main())