From 339232c7f9652967465fa6f94668f0c57e468ce8 Mon Sep 17 00:00:00 2001 From: Tom Flux Date: Wed, 12 Aug 2026 15:37:53 +0100 Subject: [PATCH] Phase 0: rolling 30-day window, delete on unsubscribe, shared password MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 0 is done apart from one external dependency, and what it measured changed the shape of the plan more than the three answers it was meant to confirm. Measured the actual upload rates from the live UULF feeds rather than guessing from catalogue size, and the numbers make the whole scale section boring. Pitch Side publishes 0.60 long-form videos a day, The Pyramid Podcast 0.77 — so a 30-day window is about 20 episodes per channel and roughly 400 across a 20-channel library, not the 20,000 the previous draft braced for. The UULF feed is also doing more work than expected: it excludes 60-74% of what these channels publish, all of it Shorts and livestreams nobody wants as Jellyfin episodes. A side effect worth recording: Pitch Side's 15-entry UULF feed spans 23.3 days, because the feed holds 15 long-form items rather than 15 uploads. Free RSS therefore covers most of a 30-day retention window on its own, which narrows what the API is for. It still earns its place — RSS carries no duration, and subscription reading has no alternative at all — but the honest version is that it now buys correctness far more than it buys safety, and the plan says so instead of keeping the more flattering 4-hours-of-yt-dlp argument. Three decisions folded in. Retention is a rolling 30 days, superseding the earlier "3 months or 300 videos". Those answered different questions — backfill depth versus retention — and holding both would mean backfilling 90 days and deleting two thirds of it on the next sweep. One number now governs both ends, so the library cannot grow. This resurrects reap.py, which the previous draft deleted, and makes tombstones load-bearing in two ways: without them the next poll re-materialises everything the sweep just deleted, and episode ordinals for a given day would shift as videos disappear. aged_out is deliberately never revivable, or raising retention_days would resurrect months of episodes into Jellyfin as new. Unsubscribing now deletes the channel rather than deactivating it, which is reasonable when rebuilding costs one API page and ~20 files. The consequence is that the removal-detection rules stop being precautionary and become the only thing standing between a transient 403 and a wiped library, so the plan now says that explicitly next to them. Auth reverts to the single shared admin_password_hash, matching every other service on susan. web/auth.py carries over untouched. What that gives up — independent revocation, and knowing who approved what — is named once and then dropped. Added tools/verify_api.py so the remaining Phase 0 work is one command. It runs all three API checks, paginates the subscription list, prints the subsync_max_new value derived from totalResults, distinguishes 403 subscriptionForbidden from every other failure with the fix in the message, and exits non-zero so it can be gated on. The ISO-8601 duration parser is unit-checked. Blocked on: there is no Google API key anywhere on this machine (searched /opt/*, /home/susan, the settings table and the config trees for AIza-shaped strings). Creating one needs a browser and a Google account. That plus the brother unchecking "Keep all my subscriptions private" are the two external prerequisites, and §16 records exactly what each unverified assumption costs if it turns out false. Co-Authored-By: Claude Opus 5 --- plan.md | 345 +++++++++++++++++++++++++++----------------- tools/verify_api.py | 225 +++++++++++++++++++++++++++++ 2 files changed, 438 insertions(+), 132 deletions(-) create mode 100755 tools/verify_api.py diff --git a/plan.md b/plan.md index a64cfce..b6e0f15 100644 --- a/plan.md +++ b/plan.md @@ -30,8 +30,8 @@ presses play — fetched on demand by a local proxy, held on tmpfs, dropped when that account is the source of truth for what exists (§4) - **Automatic subscription pickup from a YouTube account's public subscription list** (§4) — the new requirement -- A bounded back catalogue per channel — **3 months or 300 videos, whichever comes first** — that - then grows forward indefinitely (§3, §5) +- A **rolling 30-day window** per channel: videos older than that are removed from disk, so the + library stays small and bounded forever (§5) - Just-in-time streaming via the proxy, with per-video caching - NFO metadata, episode thumbnails, channel poster/fanart - An admin UI for subscriptions, status and settings @@ -72,7 +72,7 @@ So: copy the tree, then add, delete and replace. | `doctor.py` | **Lift, extend** | Add checks for API key validity, proxy health, `.strm` orphan count. | | `ytdlp.py` | **Lift, narrow** | Only the proxy calls yt-dlp now. | | `download.py` (330 lines) | **Delete** | Replaced by `strm.py`, which writes a text file. | -| `reap.py` (150 lines) | **Delete, replace** | Reaping existed to reclaim disk. There is no disk to reclaim (§3). Replaced by a much smaller "retire a video whose channel is gone" path. | +| `reap.py` (150 lines) | **Lift, simplify** | Originally slated for deletion — the 30-day window (§5) brings it back. Same aging-out logic and tombstone semantics, minus the disk-cap machinery, and deleting a `.strm` instead of a 500 MB mp4. | | `runner.py` | **Rewrite** | The run loop changes shape: sync subscriptions → poll → materialise → refresh. | | — | **New: `strm.py`** | Writes `.strm` + `.nfo` + thumbnail for one video. | | — | **New: `subsync.py`** | The brother-subscription puller (§4). | @@ -98,24 +98,28 @@ That changes the arithmetic completely: | Job | Method | Calls | Units | |---|---|---|---| -| Sync one account's subscriptions, hourly | `subscriptions.list` | 24/day (1 page) | 24 | -| Backfill one channel to the §5 bound (≤300 videos) | `playlistItems.list` @50 | ≤6 | ≤6 | -| Durations for those videos | `videos.list` @50 ids | ≤6 | ≤6 | +| Sync subscriptions, hourly | `subscriptions.list` | 24/day (1 page) | 24 | +| Backfill one channel's 30-day window (~20 videos) | `playlistItems.list` @50 | 1 | 1 | +| Durations for those videos | `videos.list` @50 ids | 1 | 1 | | Steady-state incremental discovery | **RSS feed** | — | **0** | +| Durations for the day's new videos | `videos.list` @50 ids | 1 | 1 | -A 20-channel initial build costs at most **240 units** — under 3% of one day's quota, once. Steady -state costs under 100 units/day. **We will never come close to the limit**, provided we never touch -`search.list` (100 calls/day, and we have no use for it). +A 20-channel initial build costs about **40 units**. Steady state is **~25 units/day**, nearly all of +it the hourly subscription poll. Against 10,000/day this is not a constraint worth thinking about, +provided we never touch `search.list` (100 calls/day, and we have no use for it). ### Why this matters more than the quota -**It takes yt-dlp out of the cataloguing path entirely.** Today, `youtube-automate` needs a yt-dlp -extraction per video to learn an upload date during backfill, at roughly 2.3s each. For a 20-channel -build at the §5 bound that is 6,000 videos ≈ **4 hours of continuous requests** to YouTube from a -residential IP — the "scan storm" failure mode the original handover warned about, and it recurs -every time a channel is added. The API is a documented, keyed, quota-metered endpoint that does not -care about PO tokens, SABR, or client selection, and cannot be rate-limited by YouTube's anti-bot -heuristics. +**It takes yt-dlp out of the cataloguing path entirely.** `youtube-automate` needs a yt-dlp +extraction per video to learn an upload date whenever it backfills past the RSS feed, at roughly 2.3s +each. That is the "scan storm" failure mode the original handover warned about: requests to YouTube +from a residential IP, in bulk, for videos nobody asked to watch. + +Be honest about the size of the win now that the window is 30 days: at ~20 videos per channel, the +yt-dlp route would be ~400 extractions ≈ 15 minutes, not the 4 hours a 90-day window implied. The +API is still the right answer — it is documented, keyed, quota-metered, indifferent to PO tokens and +SABR, and cannot be rate-limited by YouTube's anti-bot heuristics — but it is now buying +*correctness* (exact dates, real durations) far more than it is buying *safety*. So the split becomes: @@ -282,8 +286,10 @@ exactly why §4.4 refuses to act on a single bad response. Two failure modes here are genuinely destructive, and both are cheap to defend against. -**Runaway additions.** If he has 400 subscriptions, the first sync queues 400 channels — at the §5 -bound that is 120,000 Jellyfin episodes, which will not end well. So: +**Runaway additions.** If he has 400 subscriptions, the first sync queues 400 channels — ~8,000 +episodes at the measured rate in §5, and 400 backfills' worth of API calls and file writes in one +run. The 30-day window makes this survivable where the earlier 90-day draft did not, but it is still +not something to discover by accident. So: - A sync that would add more than **`subsync_max_new`** channels adds none of them. It records them as `pending_approval`, alerts, and waits for a click in the admin UI. @@ -307,12 +313,17 @@ basis would be unrecoverable in any pleasant way. So: indistinguishable from a broken one in consequence, and we prefer the harmless reading. - A channel missing from an otherwise-healthy response increments `missing_syncs`. Only at **`subsync_missing_threshold`** (default **3** consecutive syncs, so ~3 hours) is it unsubscribed. -- Unsubscribing **does not delete anything on disk**. It sets the channel `inactive`: polling stops, - the tree stays, and re-subscribing on YouTube restores it with no refetching. Actual deletion is a - separate, explicit, human-initiated admin action. `.strm` files cost ~50 bytes each; there is no - pressure to reclaim anything, so there is no reason to ever delete automatically. +- **Unsubscribing deletes the channel** — its directory, its `.strm` files, its NFOs and artwork, and + its rows. Re-subscribing rebuilds it from scratch, which under the 30-day window (§5) is one + `playlistItems.list` page and ~20 files: cheap enough that keeping a dormant copy around would be + the more complicated choice. The Jellyfin series disappears, taking its watch state with it. - `source = 'manual'` channels are exempt from all of the above. +Note how the two halves interact: because the *action* is now destructive and irreversible-ish, the +*detection* rules above are what stand between a transient 403 and a wiped library. They are not +belt-and-braces. Do not weaken the three-sync threshold to make testing more convenient — make the +threshold configurable and set it to 1 in tests. + **Ordering:** the sync runs *before* the poll in the same run, so a channel added at 14:00 has its catalogue built in the same pass. @@ -323,82 +334,102 @@ Follow it. A sync that hits 403, trips the add cap, or accumulates failures must the check goes red — a silently-broken subscription mirror is the worst possible outcome, because nothing appears to be wrong until someone asks why a channel never showed up. -### 4.6 The brother gets admin access, so auth becomes multi-user +### 4.6 The brother gets admin access, on the existing shared password -This is the right call — the approval queue (§4.4) and the "channel went missing" alerts are about -*his* subscriptions, so they should be his to action rather than landing on Tom every time. But it -does mean a change: `youtube-automate` has exactly one credential, `admin_password_hash` in the -`setting` table. Sharing it would mean neither party can have their access revoked or their password -changed independently, and the approval log would not record who approved what. +He gets access, which is right — the approval queue (§4.4) and the "channel went missing" alerts are +about *his* subscriptions, so they should be his to action rather than landing on Tom every time. -So replace the single setting with a minimal `user` table (§7): `username`, `password_hash` -(scrypt, same parameters), `created_at`, `last_login_at`. Two rows. **No roles, no permissions -system** — both accounts can do everything, which is correct for two brothers and one media server, -and the alternative is a permission model nobody will maintain. +**Auth does not change.** One shared `admin_password_hash` in the `setting` table, exactly as +`youtube-automate` already does it, matching how every other service on susan is shared. An earlier +draft proposed a two-row `user` table; it is not worth the code. The two things given up are worth +naming once and then forgetting: neither party's access can be revoked without changing the other's +password, and the approval log records *that* something was approved rather than by whom. -Everything else in `web/auth.py` carries over unchanged, and it is adequate for an -internet-facing login: scrypt hashing and per-address failed-login throttling that reads -`X-Forwarded-For` (which nginx must set — it already does, see §12 step 4). - -Record `approved_by` on `pending_approval` resolutions. It costs one column and it answers "why is -this channel here" six months later. +`web/auth.py` therefore carries over **unchanged**, and it is adequate for an internet-facing login: +scrypt hashing and per-address failed-login throttling that reads `X-Forwarded-For` (which nginx must +set — it already does, see §12 step 4). --- -## 5. Scale — bounded backfill, unbounded growth +## 5. Scale — a rolling 30-day window -Measured during the PoC: **Pitch Side has 1,249 videos.** The `youtube-automate` DB holds 15 of -them, because a 9-day retention window and 1.3 TB of free disk is what bounded it. `.strm` files -remove that bound entirely, so the bound is now a policy choice, and the choice is: +> **`retention_days = 30`.** A video older than 30 days has its `.strm`, `.nfo` and thumbnail +> deleted and its DB row tombstoned. Backfill on subscribe reaches back the same 30 days. One +> number governs both ends, so the library cannot grow. -> **Backfill 3 months or 300 videos per channel, whichever comes first.** +This supersedes the earlier "3 months or 300 videos" answer, which was about *backfill depth* while +this is about *retention*; keeping both would mean backfilling 90 days and then immediately deleting +two thirds of it. If the intent was really "go back 3 months, keep everything", set +`retention_days = 90` and this section still holds — only the numbers move. -Both settings exist (`backfill_max_days` = 90, `backfill_max_videos` = 300) and whichever is reached -first stops the walk. The two are well matched in practice: a channel uploading 2–3 times a day — -which is what Pitch Side's episode numbering implies — produces roughly 270 videos in 90 days, so -neither bound dominates. A weekly uploader gets ~13 videos and is bounded by time; a daily-podcast -firehose gets 300 and is bounded by count. That is the right behaviour in both directions. +### What this actually costs, measured -**This bounds the *initial* build, not the library.** Nothing ages out — `.strm` files cost ~50 -bytes and there is no disk to reclaim, so aging out would destroy metadata for no gain. The -consequence is honest and worth stating: the library **grows forever from the subscribe date**. At -20 channels averaging 2 uploads/day that is ~15,000 new episodes a year, so the 20,000-episode -question in point 1 below is deferred by about eighteen months, not answered. The lever, if it ever -bites: a `hidden` video state that removes the `.strm` and `.nfo` from disk while keeping the DB row, -so the catalogue stays re-materialisable. **Do not build it now** — build the measurement that tells -us when we need it. +The upload rates matter more than the total catalogue size, so I measured them from the live feeds +on 2026-08-12 rather than guessing: -**Initial build:** ~20 channels × ≤300 = **≤6,000 episodes, ≤12,000 files.** Comfortable. +| Channel | UULF (long-form) | UC feed (everything) | UULF filters | +|---|---|---|---| +| Pitch Side | 0.60/day → **~18 per 30d** | 2.33/day → ~70 per 30d | 74% | +| The Pyramid Podcast | 0.77/day → **~23 per 30d** | 1.91/day → ~57 per 30d | 60% | -1. **Jellyfin scan cost is still unmeasured**, and is the number that decides everything above. - Every episode is a `.strm` + a `.nfo` + a thumbnail. Normal scans were measured to make **0 media - probes** (FINDINGS §6), which is the thing that makes this viable at all — but stat-ing and - NFO-parsing 12,000 files on a Westmere with the library DB on spinning disk is its own cost. - **Measure in Phase 3** by building one channel to the bound, timing a full scan, and recording - seconds-per-1,000-episodes here. That single figure sizes both the initial build and the growth - runway. -2. **`replaceAllMetadata` remains a catastrophe rather than an annoyance.** It is the one operation - verified to probe media, and at 6,000 items it means 6,000 cold starts. The cold-start rate - limiter (20/hour) contains the damage to YouTube's side, but the library-side result is 6,000 - items whose metadata got wiped and not re-derived. Defences, all of them: +So a 30-day window is roughly **20 episodes per channel**, and 20 channels is **~400 episodes, +~1,200 files.** That is two orders of magnitude below the 20,000 the previous draft was braced for, +and it makes the entire scale section boring — which is the point. Pitch Side's 1,249-video back +catalogue simply never enters the library. + +It also confirms the UULF feed is doing real work: it filters 60–74% of what the channel publishes, +and that Shorts-and-livestreams majority is exactly what nobody wants as Jellyfin episodes. + +### A useful side effect: RSS nearly covers the whole window + +The UULF feed returns 15 entries, and for Pitch Side those 15 span **23.3 days** — because the feed +holds 15 *long-form* items, not 15 uploads. At these rates a single free RSS fetch covers most of a +30-day window on its own. + +That narrows what the API is for (§3): subscription reading, durations, and topping up the few days +RSS does not reach on a fresh subscribe. It does not eliminate it — RSS carries no duration at all, +and a busier channel would truncate sooner — but nobody should be surprised when the API turns out +to be handling a few dozen calls a day rather than thousands. + +### Aging out has two consequences worth stating + +1. **Tombstones are mandatory, not optional.** Delete a `.strm` and the next poll finds the video in + the feed again and re-materialises it, forever. `youtube-automate` already solves this: the DB row + survives deletion in a terminal state (`aged_out`) and discovery skips it. Lift that behaviour + exactly; it is the single most likely thing to get wrong here. +2. **Jellyfin loses watch state for aged-out episodes.** Removing the media file removes the item, + and with it play counts and resume positions. For a 30-day window on a subscription feed that is + acceptable — this is a "what's new" library, not an archive — but it should be a deliberate choice + rather than a surprise. Anything worth keeping permanently wants a different tool (§1). + +1. **Jellyfin scan cost is now a non-issue, but still worth one measurement.** Every episode is a + `.strm` + a `.nfo` + a thumbnail. Normal scans were measured to make **0 media probes** (FINDINGS + §6), which is what makes this viable at all. At ~1,200 files the stat-and-parse cost will not be + noticeable. **Record it in Phase 3 anyway**, as seconds-per-1,000-episodes, because it is the + number that would let anyone judge a future change to `retention_days` without re-deriving it. +2. **`replaceAllMetadata` remains the one real hazard.** It is the one operation verified to probe + media, and at ~400 items it means ~400 cold starts. The cold-start rate limiter (20/hour) contains + the damage to YouTube's side, but the library-side result is items whose metadata got wiped and + not re-derived. Defences, all of them: - the proxy 503s past the budget (built, tested — `test_limits.py`) - `ytstream` never itself issues a refresh with `replaceAllMetadata=true` - the admin UI documents "do not click Replace all metadata" in the place where someone would be tempted to - **NFOs are the source of truth**, so recovery is a re-materialise pass over the tree, not a re-fetch from YouTube. Make sure that pass exists and is one CLI command. -3. **Episode numbering holds.** `MMDD*10 + ordinal` clamps at 10 uploads/channel/day, computed - against the DB rather than the batch, so it is stable across a 300-video backfill. Season = - upload year means a 3-month window usually spans one season and occasionally two (a January - subscribe reaches back into the previous year), which is correct and needs no special handling. +3. **Episode numbering holds, and tombstones are what make it hold.** `MMDD*10 + ordinal` clamps at + 10 uploads/channel/day and is computed against the DB, not the batch. Since aged-out rows stay in + the DB, the ordinal for a given day never shifts as videos are deleted — which is the second + reason the tombstones are load-bearing. Season = upload year means a 30-day window usually spans + one season and occasionally two, which needs no special handling. 4. **The `.strm` URL is baked into every file.** Changing the proxy's host or port means rewriting all of them. Cheap (it is a tree walk) but it must be a supported CLI command, not a `sed` someone invents under pressure. Settle the URL now: `http://127.0.0.1:8099/watch/`. Jellyfin is the only client that ever reads it, it runs on the same host, and 127.0.0.1 means the proxy is unreachable from the network by construction. -5. **Backfill must be resumable.** Quota is a non-issue but a crash halfway through 12,000 files - needs to resume, not restart. Track progress per channel (`channel.backfill_cursor`), commit per - page of 50. +5. **Backfill stays resumable** (`channel.backfill_cursor`, commit per page of 50) even though a + 30-day backfill is now one or two pages. It costs a column and it is the difference between a + crashed first sync resuming and starting over across 20 channels. --- @@ -440,9 +471,9 @@ noticeable. ### 6.2 `ytstream-admin` — systemd service -Lifted from `youtube_automate/web/`. scrypt password hashes, now per user in the `user` table (§4.6) -rather than one shared setting, per-address failed-login throttling reading `X-Forwarded-For`, behind -nginx on `127.0.0.1:8086` (8085 is `youtube-automate`'s during the overlap). +Lifted from `youtube_automate/web/` unchanged: shared-password scrypt auth (§4.6), per-address +failed-login throttling reading `X-Forwarded-For`, behind nginx on `127.0.0.1:8086` (8085 is +`youtube-automate`'s during the overlap). New routes beyond what exists: subscription **sources** (add/remove a mirrored YouTube account, show its last sync, its failure count, its `pending_approval` queue with an approve/reject action), the @@ -471,17 +502,25 @@ subscribed, and when someone explicitly asks for a deeper catalogue. `/var/lib/ytstream/ytstream.db`. Start from `youtube-automate`'s schema — `channel`, `video`, `setting` are the right shape — with these changes: -**`channel`** — add `backfill_cursor TEXT` (resumable backfill), `backfill_max_videos INTEGER` and -`backfill_max_days INTEGER` (per-channel overrides of the §5 bounds; NULL = use the global setting), -`active INTEGER NOT NULL DEFAULT 1` (§4.4 soft-delete), `uploads_playlist TEXT` (whether UULF or UU -won, §3). Drop `retention_days`. +**`channel`** — add `backfill_cursor TEXT` (resumable backfill) and `uploads_playlist TEXT` (whether +UULF or UU won, §3). **Keep `retention_days`** as a per-channel override of the 30-day window (NULL = +use the global setting) — `youtube-automate` already has this column and the `rescan` semantics that +go with it. No `active` column: unsubscribing now deletes rather than deactivating (§4.4). -**`video`** — drop `size_bytes`, `downloaded_at`, `deleted_at`, `attempts`, `last_error`; none of -them mean anything when there is no download. Keep `duration` (now from `videos.list`, and it feeds -``). `state` collapses to `{listed, materialised, skipped_short, skipped_live, -hidden}` — no `pending`/`downloading`/`failed`, because materialising a `.strm` cannot -meaningfully fail. Add `published_at TEXT` (exact RFC-3339 from the API) alongside the existing -date-only `upload_date` that naming uses. +**`video`** — drop `size_bytes`, `downloaded_at`, `attempts`, `last_error`; none of them mean anything +when there is no download. **Keep `deleted_at`** — it is the aging-out timestamp now (§5). Keep +`duration` (from `videos.list`; it feeds ``). Add `published_at TEXT` (exact +RFC-3339 from the API) alongside the date-only `upload_date` that naming uses. + +`state` becomes `{listed, materialised, skipped_short, skipped_live, skipped_old, aged_out}`. No +`pending`/`downloading`/`failed`, because materialising a `.strm` cannot meaningfully fail. The two +terminal states matter and are not the same thing: + +- **`skipped_old`** — discovered already outside the window; never materialised. `youtube-automate`'s + `rescan` can revive these if `retention_days` is raised. +- **`aged_out`** — was materialised, then deleted by the sweep. **Never revived**, or raising + `retention_days` would resurrect months of episodes into Jellyfin as "new". These are tombstones + and discovery must skip them unconditionally (§5). **`channel`, continued** — two more columns instead of the join table an earlier draft had (§4.3): `source TEXT NOT NULL` (`'youtube'` for synced, `'manual'` for the pinning escape hatch) and @@ -492,19 +531,16 @@ date-only `upload_date` that naming uses. `setting` keys, because these are fields of one thing and they change together. **`pending_approval`** — new, the §4.4 add-cap and first-sync import queue: `source`, `channel_id`, -`title`, `seen_at`, `resolved_at`, `resolution` (`approved`/`rejected`), `approved_by`. - -**`user`** — new (§4.6), replacing the single `admin_password_hash` setting: `username` (PK), -`password_hash` (scrypt, same parameters as `youtube-automate`), `created_at`, `last_login_at`. Two -rows, no roles. +`title`, `seen_at`, `resolved_at`, `resolution` (`approved`/`rejected`). **`setting`** — carry over what still applies (`jellyfin_url`, `jellyfin_api_key`, -`pot_provider_url`, `max_height`, `min_duration_seconds`, `session_secret`, `last_run_at`). Drop -`retention_days`, `disk_cap_gb`, `write_subs`, `sub_langs`, `sponsorblock_mark`, `max_attempts`, -`backfill_days`, `admin_password_hash` — download-era concepts plus the credential that moved to -`user`. Add `youtube_api_key`, `subsync_max_new` (25, provisional — §4.4), -`subsync_missing_threshold` (3), `proxy_base_url` (`http://127.0.0.1:8099`), `backfill_max_videos` -(**300**), `backfill_max_days` (**90**). +`pot_provider_url`, `max_height`, `min_duration_seconds`, `admin_password_hash`, `session_secret`, +`last_run_at`), including **`retention_days`** — now **30**, and governing both the aging-out sweep +and the backfill reach (§5). Drop `disk_cap_gb`, `write_subs`, `sub_langs`, `sponsorblock_mark`, +`max_attempts`, `backfill_days` — download-era concepts. Add `youtube_api_key`, `subsync_max_new` +(25, provisional — §4.4), `subsync_missing_threshold` (3), `proxy_base_url` +(`http://127.0.0.1:8099`), and `backfill_max_videos` (**300**) purely as a runaway guard on a channel +that turns out to upload 50 times a day. **No migration from `subs.db`.** It holds 2 channels and 18 videos. Re-subscribe by hand and let the backfill do the rest; a migration script would be more code than the data is worth. @@ -613,8 +649,8 @@ Secrets live in `/var/lib/ytstream/ytstream.db`, mode `0640` `susan:automation`, through the admin UI. Never in the repo, never in a systemd unit, never in a cron line. This matches `youtube-automate`, where `jellyfin_api_key` and `session_secret` already live that way. -Three kinds: `youtube_api_key` (new) and `jellyfin_api_key` in `setting`; `session_secret` in -`setting`; and two scrypt password hashes in `user` (§4.6). +Four keys, all in `setting`: `youtube_api_key` (new), `jellyfin_api_key`, `session_secret`, and +`admin_password_hash` — one shared admin password, as everywhere else on susan (§4.6). The YouTube API key is restricted to the YouTube Data API v3 and has read-only reach over public data. Worst case on leak is quota exhaustion; rotation is a two-minute job in the console. @@ -654,19 +690,16 @@ is the thing that keeps playback working as YouTube changes. Each phase ends in something checkable. Do not start the next one until it does. -**Phase 0 — verify the assumptions the plan rests on, and get the one number it is missing.** -~15 minutes, before any code. `git init --bare /disks/git-repos/ytstream.git`. Get the API key -(§4.1). Then three curls: +**Phase 0 — verify the assumptions the plan rests on.** Mostly **done, 2026-08-12** — see §16 for +what was verified and what is still blocked. Remaining: paste the API key in and run -1. `subscriptions.list` on `UCPcTWaLV8zwx4WP4QExHj4Q` returns 200 with items — **and record - `pageInfo.totalResults`**, which is what sets `subsync_max_new` (§4.4). This is a hard blocker: - there is no fallback that does not involve him doing something (§4.2). -2. `playlistItems.list` accepts a UULF playlist id — or does not, and we take the documented `UU` - fallback plus duration filtering (§3). -3. `videos.list` returns `contentDetails.duration` for a batch of 50 ids. +```sh +python3 /opt/ytstream/tools/verify_api.py --key AIza... +``` -→ *Done when: all three answers are written back into this file as verified facts or corrections, -and `subsync_max_new` has a real value.* +which performs all three API checks, costs ~5 quota units, prints the `subsync_max_new` value, and +exits non-zero if anything fails. +→ *Done when: `verify_api.py` reports ALL PASS and its numbers are written into §4.1 and §16.* **Phase 1 — skeleton and lift.** Fork the tree, new package name, new DB path, new schema (§7), lifted modules and their tests passing. No new behaviour. @@ -677,19 +710,21 @@ test, write `deploy/deploy.sh`, operator runs it. → *Done when: `systemctl status ytstream-proxy` is active after a reboot, `/healthz` answers, and Jellyfin direct-plays a cold video end to end.* -**Phase 3 — catalogue.** `api.py`, `strm.py`, the 3-month/300-video bounded resumable backfill, the -hourly run. Build **Pitch Side alone** and **time a Jellyfin scan** — the §5 measurement. Worth -doing twice, once at the 300-video bound and once unbounded at 1,249, since the second gives the -seconds-per-1,000-episodes figure that sizes the growth runway for free. -→ *Done when: the bounded episode count is visible with correct titles, dates, durations and -thumbnails; scan time per 1,000 episodes is recorded in §5; and one of them plays.* +**Phase 3 — catalogue and retention.** `api.py`, `strm.py`, the 30-day bounded resumable backfill, the +aging-out sweep with its tombstones, the hourly run. Build **Pitch Side alone** (expect ~18 episodes) +and time a Jellyfin scan for the §5 record. +→ *Done when: the expected episode count is visible with correct titles, dates, durations and +thumbnails; one of them plays; scan time per 1,000 episodes is recorded in §5; and — the part most +likely to be wrong — a video forced past the window is deleted from disk, and the **next two polls do +not bring it back**.* **Phase 4 — subscription sync.** `subsync.py`, the first-sync bulk import, the add cap, the -missing-threshold, the `user` table and two accounts (§4.6), the admin routes, the healthchecks UUID. -→ *Done when: the brother logs in with his own credentials and approves the first import; then he -subscribes to a new channel on YouTube and within an hour it is a series in Jellyfin with episodes -that play, nobody having touched the admin UI. Plus: a forced 403 and a forced empty response both -leave the DB untouched and turn the check red.* +missing-threshold, channel deletion on unsubscribe, the admin routes, the healthchecks UUID. +→ *Done when: he approves the first import; then he subscribes to a new channel on YouTube and within +an hour it is a series in Jellyfin with episodes that play, nobody having touched the admin UI. Then +the destructive half: he unsubscribes, and after three syncs the channel and its tree are gone. Plus +a forced 403 and a forced empty response, each of which must leave the DB untouched and turn the check +red — test these before trusting the deletion path, not after.* **Phase 5 — cut over.** §12 steps 1–2, run for a week. → *Done when: nothing has broken and nobody has used the old library.* @@ -703,15 +738,19 @@ leave the DB untouched and turn the check red.* 1. **Account to mirror:** `@cflux1030` → `UCPcTWaLV8zwx4WP4QExHj4Q` ("C Flux"), resolved and recorded in §4. **Sole source of truth** — Tom's own subscriptions are not part of this at all, which removed the multi-source claim model from §4.3 and put the weight on §4.4 instead. -2. **Catalogue depth:** 3 months or 300 videos, whichever comes first. §5 rewritten around it, and - both bounds are settings with per-channel overrides. Note the consequence recorded there: this - bounds the initial build, not the library, which grows forward indefinitely. -3. **`subsync_max_new`:** to be set from `pageInfo.totalResults` on the day-one - `subscriptions.list` call, since that is the only way to learn how many channels he follows. - Provisional 25; rule of thumb `max(10, ceil(totalResults × 0.2))` (§4.4). **The one open number - in this plan** — Phase 0 closes it. -4. **Brother gets admin access:** yes, which turns the single shared password into a two-row `user` - table (§4.6) and adds `approved_by` to the approval log. +2. **Retention:** a **rolling 30-day window** — anything older is deleted from disk and tombstoned + (§5). This supersedes the earlier "3 months or 300 videos" answer, which was about backfill depth; + holding both would mean backfilling 90 days and deleting two thirds of it immediately. If the + intent was "reach back 3 months and keep it", `retention_days = 90` is a one-line change and §5 + still holds. Measured consequence: **~400 episodes total**, not 20,000. +3. **`subsync_max_new`:** still open — set from `pageInfo.totalResults` on the first + `subscriptions.list` call, which `tools/verify_api.py` prints. Provisional 25; rule of thumb + `max(10, ceil(totalResults × 0.2))` (§4.4). +4. **Brother gets admin access:** yes, on the **existing shared password** — no `user` table, no + per-account credentials (§4.6). `web/auth.py` carries over untouched. +5. **Unsubscribe deletes the channel** rather than deactivating it, since re-subscribing is one API + page and ~20 files under the 30-day window (§4.4). This makes the removal *detection* safeguards + load-bearing rather than precautionary. --- @@ -739,3 +778,45 @@ Things already paid for once. All of these are verified. spend an afternoon trying to scrape it; the API is the only route (§4). - **`pkill -f 'ytstream.py'` kills the shell that runs it**, because the command string contains its own pattern. Bracket it: `pkill -f 'ytstrea[m].py'`. + +--- + +## 16. Phase 0 results — 2026-08-12 + +### Done + +- **Bare repo initialised.** `/disks/git-repos/ytstream.git` existed as an empty directory; it is now + a real `--bare --shared=group` repo with `HEAD` → `refs/heads/main`, matching + `youtube-automate.git`. `/opt/ytstream` is a checkout with `origin` pointed at it. +- **The mirrored channel is resolved.** `@cflux1030` → `UCPcTWaLV8zwx4WP4QExHj4Q` ("C Flux", 55 + subscribers). No `channels.list` call needed at build time. +- **There is no scraping fallback** — `/@cflux1030/channels` serves the Home tab with zero channel + ids (§15). The privacy checkbox is mandatory. +- **UULF feeds work for both existing channels**, and the filtering is substantial — 60–74% of what + these channels publish is Shorts or livestreams that UULF correctly excludes. +- **Upload rates measured**, which is what sized §5: ~18–23 long-form videos per channel per 30 days, + so ~400 episodes for a 20-channel library. Pitch Side's UULF feed spans **23.3 days in 15 entries**, + meaning RSS alone nearly covers the retention window. +- **`tools/verify_api.py` written and self-tested** (ISO-8601 duration parser unit-checked against six + cases; argparse and import verified). It runs all three API checks in one command. + +### Blocked on one thing only + +**There is no Google API key on this machine** — I searched for `AIza`-shaped strings across +`/opt/*`, `/home/susan`, the settings table and the config trees, and there is none. Creating one +needs a browser and Tom's Google account (§4.1, ~5 minutes, free, no billing). Until then these three +remain unverified: + +| # | Assumption | Consequence if it fails | +|---|---|---| +| 1 | `subscriptions.list?channelId=` returns his subscriptions | **Feature is impossible as designed.** Falls back to §4.2, all of which need him to do something. | +| 2 | `playlistItems.list` accepts a `UULF…` playlist id | Take the documented `UU` fallback and filter Shorts/livestreams via `videos.list` duration + `liveStreamingDetails`. Costs one extra unit per 50. | +| 3 | `videos.list` returns parseable `contentDetails.duration` | No `` in NFOs without a yt-dlp extraction per video. Degrades, does not block. | + +Only #1 is a genuine blocker, and it also carries the number that sets `subsync_max_new`. Two +prerequisites, both external: **Tom creates the API key**, and **his brother unchecks "Keep all my +subscriptions private"**. Then one command closes Phase 0: + +```sh +python3 /opt/ytstream/tools/verify_api.py --key AIza... +``` diff --git a/tools/verify_api.py b/tools/verify_api.py new file mode 100755 index 0000000..3b285fd --- /dev/null +++ b/tools/verify_api.py @@ -0,0 +1,225 @@ +#!/usr/bin/env python3 +"""Phase 0 verification: everything the plan assumes about the YouTube Data API. + +The plan (§13 Phase 0) rests on three answers that can only come from a live +key. This runs all three, costs about 5 quota units of the 10,000/day budget, +and prints the numbers to paste back into plan.md. + + python3 tools/verify_api.py --key AIza... + +Or, once the key is in the settings table: + + python3 tools/verify_api.py --key "$(sqlite3 /var/lib/ytstream/ytstream.db \ + "select value from setting where key='youtube_api_key'")" + +Exit status is 0 only if all three checks pass, so this is safe to gate on. +""" +from __future__ import annotations + +import argparse +import json +import re +import sys +import urllib.error +import urllib.parse +import urllib.request + +API = "https://www.googleapis.com/youtube/v3" + +# The account being mirrored — @cflux1030, resolved via yt-dlp on 2026-08-12. +BROTHER = "UCPcTWaLV8zwx4WP4QExHj4Q" + +# A channel with a known-large back catalogue, used to exercise pagination. +SAMPLE_CHANNEL = "UCjCJ2LaOIsPzOoXUTMDI3wg" # Pitch Side + +ISO8601 = re.compile( + r"^P(?:(\d+)D)?T?(?:(\d+)H)?(?:(\d+)M)?(?:(\d+)S)?$" +) + + +class ApiError(Exception): + def __init__(self, status, reason, body): + super().__init__(f"HTTP {status} {reason}") + self.status = status + self.reason = reason + self.body = body + + +def call(endpoint: str, key: str, **params) -> dict: + """One API call. Raises ApiError with the parsed reason on 4xx/5xx.""" + params["key"] = key + url = f"{API}/{endpoint}?" + urllib.parse.urlencode(params) + try: + with urllib.request.urlopen(url, timeout=30) as response: + return json.load(response) + except urllib.error.HTTPError as exc: + raw = exc.read().decode("utf8", "replace") + reason = "" + try: + errors = json.loads(raw).get("error", {}).get("errors", []) + reason = errors[0].get("reason", "") if errors else "" + except ValueError: + pass + raise ApiError(exc.code, reason, raw[:400]) from exc + + +def iso8601_seconds(text: str) -> int | None: + """PT1H2M3S -> 3723. Returns None for anything unparseable.""" + match = ISO8601.match(text or "") + if not match: + return None + days, hours, minutes, seconds = (int(g or 0) for g in match.groups()) + return days * 86400 + hours * 3600 + minutes * 60 + seconds + + +# -------------------------------------------------------------------------- +results: list[tuple[str, bool, str]] = [] + + +def record(name: str, ok: bool, detail: str) -> None: + print(f" {'PASS' if ok else 'FAIL'} {name}") + for line in detail.splitlines(): + print(f" {line}") + results.append((name, ok, detail)) + + +def check_subscriptions(key: str, channel_id: str) -> None: + """The one that decides whether the whole feature is possible.""" + print("\n1. subscriptions.list on the mirrored account") + try: + page = call("subscriptions", key, part="snippet", + channelId=channel_id, maxResults=50) + except ApiError as exc: + if exc.status == 403 and exc.reason == "subscriptionForbidden": + record("subscriptions readable", False, + "403 subscriptionForbidden -- subscriptions are still PRIVATE.\n" + "Fix: youtube.com -> Settings -> Privacy -> uncheck\n" + '"Keep all my subscriptions private". There is no workaround;\n' + "the public HTML route no longer exists (plan.md §15).") + else: + record("subscriptions readable", False, + f"{exc} reason={exc.reason!r}\n{exc.body}") + return + + total = (page.get("pageInfo") or {}).get("totalResults") + items = page.get("items") or [] + titles = [i["snippet"]["title"] for i in items] + + # Paginate so the recorded total is the real one, not just page 1. + seen = len(items) + token = page.get("nextPageToken") + pages = 1 + while token and pages < 20: + page = call("subscriptions", key, part="snippet", channelId=channel_id, + maxResults=50, pageToken=token) + batch = page.get("items") or [] + seen += len(batch) + titles.extend(i["snippet"]["title"] for i in batch) + token = page.get("nextPageToken") + pages += 1 + + suggested = max(10, -(-int(total or seen) // 5)) # ceil(total * 0.2) + record("subscriptions readable", True, + f"totalResults = {total}, fetched {seen} across {pages} page(s)\n" + f"-> set subsync_max_new = {suggested} [max(10, ceil(total*0.2))]\n" + f"first few: {', '.join(titles[:8])}" + + (" ..." if len(titles) > 8 else "")) + + +def check_uploads_playlist(key: str, channel_id: str) -> None: + """Does playlistItems.list accept the undocumented UULF playlist id?""" + print("\n2. playlistItems.list on UULF (long-form-only) vs UU (documented)") + outcome = {} + for kind, playlist_id in (("UULF", "UULF" + channel_id[2:]), + ("UU", "UU" + channel_id[2:])): + try: + page = call("playlistItems", key, part="contentDetails", + playlistId=playlist_id, maxResults=5) + except ApiError as exc: + outcome[kind] = (False, f"{exc} reason={exc.reason!r}") + continue + items = page.get("items") or [] + stamps = [i["contentDetails"].get("videoPublishedAt") for i in items] + exact = all(s and s.endswith("Z") and "T" in s for s in stamps) + outcome[kind] = ( + bool(items) and exact, + f"{len(items)} items, videoPublishedAt exact={exact}, " + f"e.g. {stamps[0] if stamps else 'n/a'}, " + f"total={(page.get('pageInfo') or {}).get('totalResults')}", + ) + + uulf_ok, uulf_detail = outcome["UULF"] + uu_ok, uu_detail = outcome["UU"] + if uulf_ok: + record("uploads playlist usable", True, + f"UULF WORKS -> use it, existing Shorts/livestream filtering carries over\n" + f" UULF: {uulf_detail}\n UU: {uu_detail}") + elif uu_ok: + record("uploads playlist usable", True, + f"UULF REJECTED -> take the UU fallback and filter by duration +\n" + f"liveStreamingDetails (plan.md §3)\n" + f" UULF: {uulf_detail}\n UU: {uu_detail}") + else: + record("uploads playlist usable", False, + f"neither worked\n UULF: {uulf_detail}\n UU: {uu_detail}") + + +def check_durations(key: str, channel_id: str) -> None: + """Durations for a batch of ids -- what feeds .""" + print("\n3. videos.list durations for a batch of ids") + try: + listing = call("playlistItems", key, part="contentDetails", + playlistId="UU" + channel_id[2:], maxResults=50) + except ApiError as exc: + record("durations available", False, f"could not list ids: {exc}") + return + + ids = [i["contentDetails"]["videoId"] for i in listing.get("items") or []] + if not ids: + record("durations available", False, "no video ids to test with") + return + + try: + page = call("videos", key, part="contentDetails", + id=",".join(ids), maxResults=50) + except ApiError as exc: + record("durations available", False, f"{exc} reason={exc.reason!r}") + return + + items = page.get("items") or [] + parsed = [(i["id"], iso8601_seconds(i["contentDetails"].get("duration", ""))) + for i in items] + bad = [vid for vid, secs in parsed if secs is None] + shorts = [vid for vid, secs in parsed if secs is not None and secs <= 120] + record("durations available", not bad and len(items) == len(ids), + f"asked for {len(ids)} ids in 1 call, got {len(items)} back, " + f"{len(bad)} unparseable\n" + f"{len(shorts)} of {len(items)} are <=120s (would be filtered as Shorts)\n" + f"sample: " + ", ".join(f"{v}={s}s" for v, s in parsed[:5])) + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--key", required=True, help="YouTube Data API v3 key") + parser.add_argument("--brother", default=BROTHER, + help=f"channel id whose subscriptions to read (default {BROTHER})") + parser.add_argument("--sample", default=SAMPLE_CHANNEL, + help="channel id to exercise the playlist/duration calls against") + args = parser.parse_args() + + print(f"Phase 0 verification against the live API (~5 quota units of 10,000/day)") + check_subscriptions(args.key, args.brother) + check_uploads_playlist(args.key, args.sample) + check_durations(args.key, args.sample) + + failed = [name for name, ok, _ in results if not ok] + print() + if failed: + print(f"{len(failed)} of {len(results)} FAILED: {', '.join(failed)}") + return 1 + print(f"ALL {len(results)} PASS -- paste the numbers above into plan.md §4.1 and §14") + return 0 + + +if __name__ == "__main__": + sys.exit(main())