diff --git a/plan.md b/plan.md index a64cfce..b6e0f15 100644 --- a/plan.md +++ b/plan.md @@ -30,8 +30,8 @@ presses play — fetched on demand by a local proxy, held on tmpfs, dropped when that account is the source of truth for what exists (§4) - **Automatic subscription pickup from a YouTube account's public subscription list** (§4) — the new requirement -- A bounded back catalogue per channel — **3 months or 300 videos, whichever comes first** — that - then grows forward indefinitely (§3, §5) +- A **rolling 30-day window** per channel: videos older than that are removed from disk, so the + library stays small and bounded forever (§5) - Just-in-time streaming via the proxy, with per-video caching - NFO metadata, episode thumbnails, channel poster/fanart - An admin UI for subscriptions, status and settings @@ -72,7 +72,7 @@ So: copy the tree, then add, delete and replace. | `doctor.py` | **Lift, extend** | Add checks for API key validity, proxy health, `.strm` orphan count. | | `ytdlp.py` | **Lift, narrow** | Only the proxy calls yt-dlp now. | | `download.py` (330 lines) | **Delete** | Replaced by `strm.py`, which writes a text file. | -| `reap.py` (150 lines) | **Delete, replace** | Reaping existed to reclaim disk. There is no disk to reclaim (§3). Replaced by a much smaller "retire a video whose channel is gone" path. | +| `reap.py` (150 lines) | **Lift, simplify** | Originally slated for deletion — the 30-day window (§5) brings it back. Same aging-out logic and tombstone semantics, minus the disk-cap machinery, and deleting a `.strm` instead of a 500 MB mp4. | | `runner.py` | **Rewrite** | The run loop changes shape: sync subscriptions → poll → materialise → refresh. | | — | **New: `strm.py`** | Writes `.strm` + `.nfo` + thumbnail for one video. | | — | **New: `subsync.py`** | The brother-subscription puller (§4). | @@ -98,24 +98,28 @@ That changes the arithmetic completely: | Job | Method | Calls | Units | |---|---|---|---| -| Sync one account's subscriptions, hourly | `subscriptions.list` | 24/day (1 page) | 24 | -| Backfill one channel to the §5 bound (≤300 videos) | `playlistItems.list` @50 | ≤6 | ≤6 | -| Durations for those videos | `videos.list` @50 ids | ≤6 | ≤6 | +| Sync subscriptions, hourly | `subscriptions.list` | 24/day (1 page) | 24 | +| Backfill one channel's 30-day window (~20 videos) | `playlistItems.list` @50 | 1 | 1 | +| Durations for those videos | `videos.list` @50 ids | 1 | 1 | | Steady-state incremental discovery | **RSS feed** | — | **0** | +| Durations for the day's new videos | `videos.list` @50 ids | 1 | 1 | -A 20-channel initial build costs at most **240 units** — under 3% of one day's quota, once. Steady -state costs under 100 units/day. **We will never come close to the limit**, provided we never touch -`search.list` (100 calls/day, and we have no use for it). +A 20-channel initial build costs about **40 units**. Steady state is **~25 units/day**, nearly all of +it the hourly subscription poll. Against 10,000/day this is not a constraint worth thinking about, +provided we never touch `search.list` (100 calls/day, and we have no use for it). ### Why this matters more than the quota -**It takes yt-dlp out of the cataloguing path entirely.** Today, `youtube-automate` needs a yt-dlp -extraction per video to learn an upload date during backfill, at roughly 2.3s each. For a 20-channel -build at the §5 bound that is 6,000 videos ≈ **4 hours of continuous requests** to YouTube from a -residential IP — the "scan storm" failure mode the original handover warned about, and it recurs -every time a channel is added. The API is a documented, keyed, quota-metered endpoint that does not -care about PO tokens, SABR, or client selection, and cannot be rate-limited by YouTube's anti-bot -heuristics. +**It takes yt-dlp out of the cataloguing path entirely.** `youtube-automate` needs a yt-dlp +extraction per video to learn an upload date whenever it backfills past the RSS feed, at roughly 2.3s +each. That is the "scan storm" failure mode the original handover warned about: requests to YouTube +from a residential IP, in bulk, for videos nobody asked to watch. + +Be honest about the size of the win now that the window is 30 days: at ~20 videos per channel, the +yt-dlp route would be ~400 extractions ≈ 15 minutes, not the 4 hours a 90-day window implied. The +API is still the right answer — it is documented, keyed, quota-metered, indifferent to PO tokens and +SABR, and cannot be rate-limited by YouTube's anti-bot heuristics — but it is now buying +*correctness* (exact dates, real durations) far more than it is buying *safety*. So the split becomes: @@ -282,8 +286,10 @@ exactly why §4.4 refuses to act on a single bad response. Two failure modes here are genuinely destructive, and both are cheap to defend against. -**Runaway additions.** If he has 400 subscriptions, the first sync queues 400 channels — at the §5 -bound that is 120,000 Jellyfin episodes, which will not end well. So: +**Runaway additions.** If he has 400 subscriptions, the first sync queues 400 channels — ~8,000 +episodes at the measured rate in §5, and 400 backfills' worth of API calls and file writes in one +run. The 30-day window makes this survivable where the earlier 90-day draft did not, but it is still +not something to discover by accident. So: - A sync that would add more than **`subsync_max_new`** channels adds none of them. It records them as `pending_approval`, alerts, and waits for a click in the admin UI. @@ -307,12 +313,17 @@ basis would be unrecoverable in any pleasant way. So: indistinguishable from a broken one in consequence, and we prefer the harmless reading. - A channel missing from an otherwise-healthy response increments `missing_syncs`. Only at **`subsync_missing_threshold`** (default **3** consecutive syncs, so ~3 hours) is it unsubscribed. -- Unsubscribing **does not delete anything on disk**. It sets the channel `inactive`: polling stops, - the tree stays, and re-subscribing on YouTube restores it with no refetching. Actual deletion is a - separate, explicit, human-initiated admin action. `.strm` files cost ~50 bytes each; there is no - pressure to reclaim anything, so there is no reason to ever delete automatically. +- **Unsubscribing deletes the channel** — its directory, its `.strm` files, its NFOs and artwork, and + its rows. Re-subscribing rebuilds it from scratch, which under the 30-day window (§5) is one + `playlistItems.list` page and ~20 files: cheap enough that keeping a dormant copy around would be + the more complicated choice. The Jellyfin series disappears, taking its watch state with it. - `source = 'manual'` channels are exempt from all of the above. +Note how the two halves interact: because the *action* is now destructive and irreversible-ish, the +*detection* rules above are what stand between a transient 403 and a wiped library. They are not +belt-and-braces. Do not weaken the three-sync threshold to make testing more convenient — make the +threshold configurable and set it to 1 in tests. + **Ordering:** the sync runs *before* the poll in the same run, so a channel added at 14:00 has its catalogue built in the same pass. @@ -323,82 +334,102 @@ Follow it. A sync that hits 403, trips the add cap, or accumulates failures must the check goes red — a silently-broken subscription mirror is the worst possible outcome, because nothing appears to be wrong until someone asks why a channel never showed up. -### 4.6 The brother gets admin access, so auth becomes multi-user +### 4.6 The brother gets admin access, on the existing shared password -This is the right call — the approval queue (§4.4) and the "channel went missing" alerts are about -*his* subscriptions, so they should be his to action rather than landing on Tom every time. But it -does mean a change: `youtube-automate` has exactly one credential, `admin_password_hash` in the -`setting` table. Sharing it would mean neither party can have their access revoked or their password -changed independently, and the approval log would not record who approved what. +He gets access, which is right — the approval queue (§4.4) and the "channel went missing" alerts are +about *his* subscriptions, so they should be his to action rather than landing on Tom every time. -So replace the single setting with a minimal `user` table (§7): `username`, `password_hash` -(scrypt, same parameters), `created_at`, `last_login_at`. Two rows. **No roles, no permissions -system** — both accounts can do everything, which is correct for two brothers and one media server, -and the alternative is a permission model nobody will maintain. +**Auth does not change.** One shared `admin_password_hash` in the `setting` table, exactly as +`youtube-automate` already does it, matching how every other service on susan is shared. An earlier +draft proposed a two-row `user` table; it is not worth the code. The two things given up are worth +naming once and then forgetting: neither party's access can be revoked without changing the other's +password, and the approval log records *that* something was approved rather than by whom. -Everything else in `web/auth.py` carries over unchanged, and it is adequate for an -internet-facing login: scrypt hashing and per-address failed-login throttling that reads -`X-Forwarded-For` (which nginx must set — it already does, see §12 step 4). - -Record `approved_by` on `pending_approval` resolutions. It costs one column and it answers "why is -this channel here" six months later. +`web/auth.py` therefore carries over **unchanged**, and it is adequate for an internet-facing login: +scrypt hashing and per-address failed-login throttling that reads `X-Forwarded-For` (which nginx must +set — it already does, see §12 step 4). --- -## 5. Scale — bounded backfill, unbounded growth +## 5. Scale — a rolling 30-day window -Measured during the PoC: **Pitch Side has 1,249 videos.** The `youtube-automate` DB holds 15 of -them, because a 9-day retention window and 1.3 TB of free disk is what bounded it. `.strm` files -remove that bound entirely, so the bound is now a policy choice, and the choice is: +> **`retention_days = 30`.** A video older than 30 days has its `.strm`, `.nfo` and thumbnail +> deleted and its DB row tombstoned. Backfill on subscribe reaches back the same 30 days. One +> number governs both ends, so the library cannot grow. -> **Backfill 3 months or 300 videos per channel, whichever comes first.** +This supersedes the earlier "3 months or 300 videos" answer, which was about *backfill depth* while +this is about *retention*; keeping both would mean backfilling 90 days and then immediately deleting +two thirds of it. If the intent was really "go back 3 months, keep everything", set +`retention_days = 90` and this section still holds — only the numbers move. -Both settings exist (`backfill_max_days` = 90, `backfill_max_videos` = 300) and whichever is reached -first stops the walk. The two are well matched in practice: a channel uploading 2–3 times a day — -which is what Pitch Side's episode numbering implies — produces roughly 270 videos in 90 days, so -neither bound dominates. A weekly uploader gets ~13 videos and is bounded by time; a daily-podcast -firehose gets 300 and is bounded by count. That is the right behaviour in both directions. +### What this actually costs, measured -**This bounds the *initial* build, not the library.** Nothing ages out — `.strm` files cost ~50 -bytes and there is no disk to reclaim, so aging out would destroy metadata for no gain. The -consequence is honest and worth stating: the library **grows forever from the subscribe date**. At -20 channels averaging 2 uploads/day that is ~15,000 new episodes a year, so the 20,000-episode -question in point 1 below is deferred by about eighteen months, not answered. The lever, if it ever -bites: a `hidden` video state that removes the `.strm` and `.nfo` from disk while keeping the DB row, -so the catalogue stays re-materialisable. **Do not build it now** — build the measurement that tells -us when we need it. +The upload rates matter more than the total catalogue size, so I measured them from the live feeds +on 2026-08-12 rather than guessing: -**Initial build:** ~20 channels × ≤300 = **≤6,000 episodes, ≤12,000 files.** Comfortable. +| Channel | UULF (long-form) | UC feed (everything) | UULF filters | +|---|---|---|---| +| Pitch Side | 0.60/day → **~18 per 30d** | 2.33/day → ~70 per 30d | 74% | +| The Pyramid Podcast | 0.77/day → **~23 per 30d** | 1.91/day → ~57 per 30d | 60% | -1. **Jellyfin scan cost is still unmeasured**, and is the number that decides everything above. - Every episode is a `.strm` + a `.nfo` + a thumbnail. Normal scans were measured to make **0 media - probes** (FINDINGS §6), which is the thing that makes this viable at all — but stat-ing and - NFO-parsing 12,000 files on a Westmere with the library DB on spinning disk is its own cost. - **Measure in Phase 3** by building one channel to the bound, timing a full scan, and recording - seconds-per-1,000-episodes here. That single figure sizes both the initial build and the growth - runway. -2. **`replaceAllMetadata` remains a catastrophe rather than an annoyance.** It is the one operation - verified to probe media, and at 6,000 items it means 6,000 cold starts. The cold-start rate - limiter (20/hour) contains the damage to YouTube's side, but the library-side result is 6,000 - items whose metadata got wiped and not re-derived. Defences, all of them: +So a 30-day window is roughly **20 episodes per channel**, and 20 channels is **~400 episodes, +~1,200 files.** That is two orders of magnitude below the 20,000 the previous draft was braced for, +and it makes the entire scale section boring — which is the point. Pitch Side's 1,249-video back +catalogue simply never enters the library. + +It also confirms the UULF feed is doing real work: it filters 60–74% of what the channel publishes, +and that Shorts-and-livestreams majority is exactly what nobody wants as Jellyfin episodes. + +### A useful side effect: RSS nearly covers the whole window + +The UULF feed returns 15 entries, and for Pitch Side those 15 span **23.3 days** — because the feed +holds 15 *long-form* items, not 15 uploads. At these rates a single free RSS fetch covers most of a +30-day window on its own. + +That narrows what the API is for (§3): subscription reading, durations, and topping up the few days +RSS does not reach on a fresh subscribe. It does not eliminate it — RSS carries no duration at all, +and a busier channel would truncate sooner — but nobody should be surprised when the API turns out +to be handling a few dozen calls a day rather than thousands. + +### Aging out has two consequences worth stating + +1. **Tombstones are mandatory, not optional.** Delete a `.strm` and the next poll finds the video in + the feed again and re-materialises it, forever. `youtube-automate` already solves this: the DB row + survives deletion in a terminal state (`aged_out`) and discovery skips it. Lift that behaviour + exactly; it is the single most likely thing to get wrong here. +2. **Jellyfin loses watch state for aged-out episodes.** Removing the media file removes the item, + and with it play counts and resume positions. For a 30-day window on a subscription feed that is + acceptable — this is a "what's new" library, not an archive — but it should be a deliberate choice + rather than a surprise. Anything worth keeping permanently wants a different tool (§1). + +1. **Jellyfin scan cost is now a non-issue, but still worth one measurement.** Every episode is a + `.strm` + a `.nfo` + a thumbnail. Normal scans were measured to make **0 media probes** (FINDINGS + §6), which is what makes this viable at all. At ~1,200 files the stat-and-parse cost will not be + noticeable. **Record it in Phase 3 anyway**, as seconds-per-1,000-episodes, because it is the + number that would let anyone judge a future change to `retention_days` without re-deriving it. +2. **`replaceAllMetadata` remains the one real hazard.** It is the one operation verified to probe + media, and at ~400 items it means ~400 cold starts. The cold-start rate limiter (20/hour) contains + the damage to YouTube's side, but the library-side result is items whose metadata got wiped and + not re-derived. Defences, all of them: - the proxy 503s past the budget (built, tested — `test_limits.py`) - `ytstream` never itself issues a refresh with `replaceAllMetadata=true` - the admin UI documents "do not click Replace all metadata" in the place where someone would be tempted to - **NFOs are the source of truth**, so recovery is a re-materialise pass over the tree, not a re-fetch from YouTube. Make sure that pass exists and is one CLI command. -3. **Episode numbering holds.** `MMDD*10 + ordinal` clamps at 10 uploads/channel/day, computed - against the DB rather than the batch, so it is stable across a 300-video backfill. Season = - upload year means a 3-month window usually spans one season and occasionally two (a January - subscribe reaches back into the previous year), which is correct and needs no special handling. +3. **Episode numbering holds, and tombstones are what make it hold.** `MMDD*10 + ordinal` clamps at + 10 uploads/channel/day and is computed against the DB, not the batch. Since aged-out rows stay in + the DB, the ordinal for a given day never shifts as videos are deleted — which is the second + reason the tombstones are load-bearing. Season = upload year means a 30-day window usually spans + one season and occasionally two, which needs no special handling. 4. **The `.strm` URL is baked into every file.** Changing the proxy's host or port means rewriting all of them. Cheap (it is a tree walk) but it must be a supported CLI command, not a `sed` someone invents under pressure. Settle the URL now: `http://127.0.0.1:8099/watch/`. Jellyfin is the only client that ever reads it, it runs on the same host, and 127.0.0.1 means the proxy is unreachable from the network by construction. -5. **Backfill must be resumable.** Quota is a non-issue but a crash halfway through 12,000 files - needs to resume, not restart. Track progress per channel (`channel.backfill_cursor`), commit per - page of 50. +5. **Backfill stays resumable** (`channel.backfill_cursor`, commit per page of 50) even though a + 30-day backfill is now one or two pages. It costs a column and it is the difference between a + crashed first sync resuming and starting over across 20 channels. --- @@ -440,9 +471,9 @@ noticeable. ### 6.2 `ytstream-admin` — systemd service -Lifted from `youtube_automate/web/`. scrypt password hashes, now per user in the `user` table (§4.6) -rather than one shared setting, per-address failed-login throttling reading `X-Forwarded-For`, behind -nginx on `127.0.0.1:8086` (8085 is `youtube-automate`'s during the overlap). +Lifted from `youtube_automate/web/` unchanged: shared-password scrypt auth (§4.6), per-address +failed-login throttling reading `X-Forwarded-For`, behind nginx on `127.0.0.1:8086` (8085 is +`youtube-automate`'s during the overlap). New routes beyond what exists: subscription **sources** (add/remove a mirrored YouTube account, show its last sync, its failure count, its `pending_approval` queue with an approve/reject action), the @@ -471,17 +502,25 @@ subscribed, and when someone explicitly asks for a deeper catalogue. `/var/lib/ytstream/ytstream.db`. Start from `youtube-automate`'s schema — `channel`, `video`, `setting` are the right shape — with these changes: -**`channel`** — add `backfill_cursor TEXT` (resumable backfill), `backfill_max_videos INTEGER` and -`backfill_max_days INTEGER` (per-channel overrides of the §5 bounds; NULL = use the global setting), -`active INTEGER NOT NULL DEFAULT 1` (§4.4 soft-delete), `uploads_playlist TEXT` (whether UULF or UU -won, §3). Drop `retention_days`. +**`channel`** — add `backfill_cursor TEXT` (resumable backfill) and `uploads_playlist TEXT` (whether +UULF or UU won, §3). **Keep `retention_days`** as a per-channel override of the 30-day window (NULL = +use the global setting) — `youtube-automate` already has this column and the `rescan` semantics that +go with it. No `active` column: unsubscribing now deletes rather than deactivating (§4.4). -**`video`** — drop `size_bytes`, `downloaded_at`, `deleted_at`, `attempts`, `last_error`; none of -them mean anything when there is no download. Keep `duration` (now from `videos.list`, and it feeds -``). `state` collapses to `{listed, materialised, skipped_short, skipped_live, -hidden}` — no `pending`/`downloading`/`failed`, because materialising a `.strm` cannot -meaningfully fail. Add `published_at TEXT` (exact RFC-3339 from the API) alongside the existing -date-only `upload_date` that naming uses. +**`video`** — drop `size_bytes`, `downloaded_at`, `attempts`, `last_error`; none of them mean anything +when there is no download. **Keep `deleted_at`** — it is the aging-out timestamp now (§5). Keep +`duration` (from `videos.list`; it feeds ``). Add `published_at TEXT` (exact +RFC-3339 from the API) alongside the date-only `upload_date` that naming uses. + +`state` becomes `{listed, materialised, skipped_short, skipped_live, skipped_old, aged_out}`. No +`pending`/`downloading`/`failed`, because materialising a `.strm` cannot meaningfully fail. The two +terminal states matter and are not the same thing: + +- **`skipped_old`** — discovered already outside the window; never materialised. `youtube-automate`'s + `rescan` can revive these if `retention_days` is raised. +- **`aged_out`** — was materialised, then deleted by the sweep. **Never revived**, or raising + `retention_days` would resurrect months of episodes into Jellyfin as "new". These are tombstones + and discovery must skip them unconditionally (§5). **`channel`, continued** — two more columns instead of the join table an earlier draft had (§4.3): `source TEXT NOT NULL` (`'youtube'` for synced, `'manual'` for the pinning escape hatch) and @@ -492,19 +531,16 @@ date-only `upload_date` that naming uses. `setting` keys, because these are fields of one thing and they change together. **`pending_approval`** — new, the §4.4 add-cap and first-sync import queue: `source`, `channel_id`, -`title`, `seen_at`, `resolved_at`, `resolution` (`approved`/`rejected`), `approved_by`. - -**`user`** — new (§4.6), replacing the single `admin_password_hash` setting: `username` (PK), -`password_hash` (scrypt, same parameters as `youtube-automate`), `created_at`, `last_login_at`. Two -rows, no roles. +`title`, `seen_at`, `resolved_at`, `resolution` (`approved`/`rejected`). **`setting`** — carry over what still applies (`jellyfin_url`, `jellyfin_api_key`, -`pot_provider_url`, `max_height`, `min_duration_seconds`, `session_secret`, `last_run_at`). Drop -`retention_days`, `disk_cap_gb`, `write_subs`, `sub_langs`, `sponsorblock_mark`, `max_attempts`, -`backfill_days`, `admin_password_hash` — download-era concepts plus the credential that moved to -`user`. Add `youtube_api_key`, `subsync_max_new` (25, provisional — §4.4), -`subsync_missing_threshold` (3), `proxy_base_url` (`http://127.0.0.1:8099`), `backfill_max_videos` -(**300**), `backfill_max_days` (**90**). +`pot_provider_url`, `max_height`, `min_duration_seconds`, `admin_password_hash`, `session_secret`, +`last_run_at`), including **`retention_days`** — now **30**, and governing both the aging-out sweep +and the backfill reach (§5). Drop `disk_cap_gb`, `write_subs`, `sub_langs`, `sponsorblock_mark`, +`max_attempts`, `backfill_days` — download-era concepts. Add `youtube_api_key`, `subsync_max_new` +(25, provisional — §4.4), `subsync_missing_threshold` (3), `proxy_base_url` +(`http://127.0.0.1:8099`), and `backfill_max_videos` (**300**) purely as a runaway guard on a channel +that turns out to upload 50 times a day. **No migration from `subs.db`.** It holds 2 channels and 18 videos. Re-subscribe by hand and let the backfill do the rest; a migration script would be more code than the data is worth. @@ -613,8 +649,8 @@ Secrets live in `/var/lib/ytstream/ytstream.db`, mode `0640` `susan:automation`, through the admin UI. Never in the repo, never in a systemd unit, never in a cron line. This matches `youtube-automate`, where `jellyfin_api_key` and `session_secret` already live that way. -Three kinds: `youtube_api_key` (new) and `jellyfin_api_key` in `setting`; `session_secret` in -`setting`; and two scrypt password hashes in `user` (§4.6). +Four keys, all in `setting`: `youtube_api_key` (new), `jellyfin_api_key`, `session_secret`, and +`admin_password_hash` — one shared admin password, as everywhere else on susan (§4.6). The YouTube API key is restricted to the YouTube Data API v3 and has read-only reach over public data. Worst case on leak is quota exhaustion; rotation is a two-minute job in the console. @@ -654,19 +690,16 @@ is the thing that keeps playback working as YouTube changes. Each phase ends in something checkable. Do not start the next one until it does. -**Phase 0 — verify the assumptions the plan rests on, and get the one number it is missing.** -~15 minutes, before any code. `git init --bare /disks/git-repos/ytstream.git`. Get the API key -(§4.1). Then three curls: +**Phase 0 — verify the assumptions the plan rests on.** Mostly **done, 2026-08-12** — see §16 for +what was verified and what is still blocked. Remaining: paste the API key in and run -1. `subscriptions.list` on `UCPcTWaLV8zwx4WP4QExHj4Q` returns 200 with items — **and record - `pageInfo.totalResults`**, which is what sets `subsync_max_new` (§4.4). This is a hard blocker: - there is no fallback that does not involve him doing something (§4.2). -2. `playlistItems.list` accepts a UULF playlist id — or does not, and we take the documented `UU` - fallback plus duration filtering (§3). -3. `videos.list` returns `contentDetails.duration` for a batch of 50 ids. +```sh +python3 /opt/ytstream/tools/verify_api.py --key AIza... +``` -→ *Done when: all three answers are written back into this file as verified facts or corrections, -and `subsync_max_new` has a real value.* +which performs all three API checks, costs ~5 quota units, prints the `subsync_max_new` value, and +exits non-zero if anything fails. +→ *Done when: `verify_api.py` reports ALL PASS and its numbers are written into §4.1 and §16.* **Phase 1 — skeleton and lift.** Fork the tree, new package name, new DB path, new schema (§7), lifted modules and their tests passing. No new behaviour. @@ -677,19 +710,21 @@ test, write `deploy/deploy.sh`, operator runs it. → *Done when: `systemctl status ytstream-proxy` is active after a reboot, `/healthz` answers, and Jellyfin direct-plays a cold video end to end.* -**Phase 3 — catalogue.** `api.py`, `strm.py`, the 3-month/300-video bounded resumable backfill, the -hourly run. Build **Pitch Side alone** and **time a Jellyfin scan** — the §5 measurement. Worth -doing twice, once at the 300-video bound and once unbounded at 1,249, since the second gives the -seconds-per-1,000-episodes figure that sizes the growth runway for free. -→ *Done when: the bounded episode count is visible with correct titles, dates, durations and -thumbnails; scan time per 1,000 episodes is recorded in §5; and one of them plays.* +**Phase 3 — catalogue and retention.** `api.py`, `strm.py`, the 30-day bounded resumable backfill, the +aging-out sweep with its tombstones, the hourly run. Build **Pitch Side alone** (expect ~18 episodes) +and time a Jellyfin scan for the §5 record. +→ *Done when: the expected episode count is visible with correct titles, dates, durations and +thumbnails; one of them plays; scan time per 1,000 episodes is recorded in §5; and — the part most +likely to be wrong — a video forced past the window is deleted from disk, and the **next two polls do +not bring it back**.* **Phase 4 — subscription sync.** `subsync.py`, the first-sync bulk import, the add cap, the -missing-threshold, the `user` table and two accounts (§4.6), the admin routes, the healthchecks UUID. -→ *Done when: the brother logs in with his own credentials and approves the first import; then he -subscribes to a new channel on YouTube and within an hour it is a series in Jellyfin with episodes -that play, nobody having touched the admin UI. Plus: a forced 403 and a forced empty response both -leave the DB untouched and turn the check red.* +missing-threshold, channel deletion on unsubscribe, the admin routes, the healthchecks UUID. +→ *Done when: he approves the first import; then he subscribes to a new channel on YouTube and within +an hour it is a series in Jellyfin with episodes that play, nobody having touched the admin UI. Then +the destructive half: he unsubscribes, and after three syncs the channel and its tree are gone. Plus +a forced 403 and a forced empty response, each of which must leave the DB untouched and turn the check +red — test these before trusting the deletion path, not after.* **Phase 5 — cut over.** §12 steps 1–2, run for a week. → *Done when: nothing has broken and nobody has used the old library.* @@ -703,15 +738,19 @@ leave the DB untouched and turn the check red.* 1. **Account to mirror:** `@cflux1030` → `UCPcTWaLV8zwx4WP4QExHj4Q` ("C Flux"), resolved and recorded in §4. **Sole source of truth** — Tom's own subscriptions are not part of this at all, which removed the multi-source claim model from §4.3 and put the weight on §4.4 instead. -2. **Catalogue depth:** 3 months or 300 videos, whichever comes first. §5 rewritten around it, and - both bounds are settings with per-channel overrides. Note the consequence recorded there: this - bounds the initial build, not the library, which grows forward indefinitely. -3. **`subsync_max_new`:** to be set from `pageInfo.totalResults` on the day-one - `subscriptions.list` call, since that is the only way to learn how many channels he follows. - Provisional 25; rule of thumb `max(10, ceil(totalResults × 0.2))` (§4.4). **The one open number - in this plan** — Phase 0 closes it. -4. **Brother gets admin access:** yes, which turns the single shared password into a two-row `user` - table (§4.6) and adds `approved_by` to the approval log. +2. **Retention:** a **rolling 30-day window** — anything older is deleted from disk and tombstoned + (§5). This supersedes the earlier "3 months or 300 videos" answer, which was about backfill depth; + holding both would mean backfilling 90 days and deleting two thirds of it immediately. If the + intent was "reach back 3 months and keep it", `retention_days = 90` is a one-line change and §5 + still holds. Measured consequence: **~400 episodes total**, not 20,000. +3. **`subsync_max_new`:** still open — set from `pageInfo.totalResults` on the first + `subscriptions.list` call, which `tools/verify_api.py` prints. Provisional 25; rule of thumb + `max(10, ceil(totalResults × 0.2))` (§4.4). +4. **Brother gets admin access:** yes, on the **existing shared password** — no `user` table, no + per-account credentials (§4.6). `web/auth.py` carries over untouched. +5. **Unsubscribe deletes the channel** rather than deactivating it, since re-subscribing is one API + page and ~20 files under the 30-day window (§4.4). This makes the removal *detection* safeguards + load-bearing rather than precautionary. --- @@ -739,3 +778,45 @@ Things already paid for once. All of these are verified. spend an afternoon trying to scrape it; the API is the only route (§4). - **`pkill -f 'ytstream.py'` kills the shell that runs it**, because the command string contains its own pattern. Bracket it: `pkill -f 'ytstrea[m].py'`. + +--- + +## 16. Phase 0 results — 2026-08-12 + +### Done + +- **Bare repo initialised.** `/disks/git-repos/ytstream.git` existed as an empty directory; it is now + a real `--bare --shared=group` repo with `HEAD` → `refs/heads/main`, matching + `youtube-automate.git`. `/opt/ytstream` is a checkout with `origin` pointed at it. +- **The mirrored channel is resolved.** `@cflux1030` → `UCPcTWaLV8zwx4WP4QExHj4Q` ("C Flux", 55 + subscribers). No `channels.list` call needed at build time. +- **There is no scraping fallback** — `/@cflux1030/channels` serves the Home tab with zero channel + ids (§15). The privacy checkbox is mandatory. +- **UULF feeds work for both existing channels**, and the filtering is substantial — 60–74% of what + these channels publish is Shorts or livestreams that UULF correctly excludes. +- **Upload rates measured**, which is what sized §5: ~18–23 long-form videos per channel per 30 days, + so ~400 episodes for a 20-channel library. Pitch Side's UULF feed spans **23.3 days in 15 entries**, + meaning RSS alone nearly covers the retention window. +- **`tools/verify_api.py` written and self-tested** (ISO-8601 duration parser unit-checked against six + cases; argparse and import verified). It runs all three API checks in one command. + +### Blocked on one thing only + +**There is no Google API key on this machine** — I searched for `AIza`-shaped strings across +`/opt/*`, `/home/susan`, the settings table and the config trees, and there is none. Creating one +needs a browser and Tom's Google account (§4.1, ~5 minutes, free, no billing). Until then these three +remain unverified: + +| # | Assumption | Consequence if it fails | +|---|---|---| +| 1 | `subscriptions.list?channelId=` returns his subscriptions | **Feature is impossible as designed.** Falls back to §4.2, all of which need him to do something. | +| 2 | `playlistItems.list` accepts a `UULF…` playlist id | Take the documented `UU` fallback and filter Shorts/livestreams via `videos.list` duration + `liveStreamingDetails`. Costs one extra unit per 50. | +| 3 | `videos.list` returns parseable `contentDetails.duration` | No `` in NFOs without a yt-dlp extraction per video. Degrades, does not block. | + +Only #1 is a genuine blocker, and it also carries the number that sets `subsync_max_new`. Two +prerequisites, both external: **Tom creates the API key**, and **his brother unchecks "Keep all my +subscriptions private"**. Then one command closes Phase 0: + +```sh +python3 /opt/ytstream/tools/verify_api.py --key AIza... +``` diff --git a/tools/verify_api.py b/tools/verify_api.py new file mode 100755 index 0000000..3b285fd --- /dev/null +++ b/tools/verify_api.py @@ -0,0 +1,225 @@ +#!/usr/bin/env python3 +"""Phase 0 verification: everything the plan assumes about the YouTube Data API. + +The plan (§13 Phase 0) rests on three answers that can only come from a live +key. This runs all three, costs about 5 quota units of the 10,000/day budget, +and prints the numbers to paste back into plan.md. + + python3 tools/verify_api.py --key AIza... + +Or, once the key is in the settings table: + + python3 tools/verify_api.py --key "$(sqlite3 /var/lib/ytstream/ytstream.db \ + "select value from setting where key='youtube_api_key'")" + +Exit status is 0 only if all three checks pass, so this is safe to gate on. +""" +from __future__ import annotations + +import argparse +import json +import re +import sys +import urllib.error +import urllib.parse +import urllib.request + +API = "https://www.googleapis.com/youtube/v3" + +# The account being mirrored — @cflux1030, resolved via yt-dlp on 2026-08-12. +BROTHER = "UCPcTWaLV8zwx4WP4QExHj4Q" + +# A channel with a known-large back catalogue, used to exercise pagination. +SAMPLE_CHANNEL = "UCjCJ2LaOIsPzOoXUTMDI3wg" # Pitch Side + +ISO8601 = re.compile( + r"^P(?:(\d+)D)?T?(?:(\d+)H)?(?:(\d+)M)?(?:(\d+)S)?$" +) + + +class ApiError(Exception): + def __init__(self, status, reason, body): + super().__init__(f"HTTP {status} {reason}") + self.status = status + self.reason = reason + self.body = body + + +def call(endpoint: str, key: str, **params) -> dict: + """One API call. Raises ApiError with the parsed reason on 4xx/5xx.""" + params["key"] = key + url = f"{API}/{endpoint}?" + urllib.parse.urlencode(params) + try: + with urllib.request.urlopen(url, timeout=30) as response: + return json.load(response) + except urllib.error.HTTPError as exc: + raw = exc.read().decode("utf8", "replace") + reason = "" + try: + errors = json.loads(raw).get("error", {}).get("errors", []) + reason = errors[0].get("reason", "") if errors else "" + except ValueError: + pass + raise ApiError(exc.code, reason, raw[:400]) from exc + + +def iso8601_seconds(text: str) -> int | None: + """PT1H2M3S -> 3723. Returns None for anything unparseable.""" + match = ISO8601.match(text or "") + if not match: + return None + days, hours, minutes, seconds = (int(g or 0) for g in match.groups()) + return days * 86400 + hours * 3600 + minutes * 60 + seconds + + +# -------------------------------------------------------------------------- +results: list[tuple[str, bool, str]] = [] + + +def record(name: str, ok: bool, detail: str) -> None: + print(f" {'PASS' if ok else 'FAIL'} {name}") + for line in detail.splitlines(): + print(f" {line}") + results.append((name, ok, detail)) + + +def check_subscriptions(key: str, channel_id: str) -> None: + """The one that decides whether the whole feature is possible.""" + print("\n1. subscriptions.list on the mirrored account") + try: + page = call("subscriptions", key, part="snippet", + channelId=channel_id, maxResults=50) + except ApiError as exc: + if exc.status == 403 and exc.reason == "subscriptionForbidden": + record("subscriptions readable", False, + "403 subscriptionForbidden -- subscriptions are still PRIVATE.\n" + "Fix: youtube.com -> Settings -> Privacy -> uncheck\n" + '"Keep all my subscriptions private". There is no workaround;\n' + "the public HTML route no longer exists (plan.md §15).") + else: + record("subscriptions readable", False, + f"{exc} reason={exc.reason!r}\n{exc.body}") + return + + total = (page.get("pageInfo") or {}).get("totalResults") + items = page.get("items") or [] + titles = [i["snippet"]["title"] for i in items] + + # Paginate so the recorded total is the real one, not just page 1. + seen = len(items) + token = page.get("nextPageToken") + pages = 1 + while token and pages < 20: + page = call("subscriptions", key, part="snippet", channelId=channel_id, + maxResults=50, pageToken=token) + batch = page.get("items") or [] + seen += len(batch) + titles.extend(i["snippet"]["title"] for i in batch) + token = page.get("nextPageToken") + pages += 1 + + suggested = max(10, -(-int(total or seen) // 5)) # ceil(total * 0.2) + record("subscriptions readable", True, + f"totalResults = {total}, fetched {seen} across {pages} page(s)\n" + f"-> set subsync_max_new = {suggested} [max(10, ceil(total*0.2))]\n" + f"first few: {', '.join(titles[:8])}" + + (" ..." if len(titles) > 8 else "")) + + +def check_uploads_playlist(key: str, channel_id: str) -> None: + """Does playlistItems.list accept the undocumented UULF playlist id?""" + print("\n2. playlistItems.list on UULF (long-form-only) vs UU (documented)") + outcome = {} + for kind, playlist_id in (("UULF", "UULF" + channel_id[2:]), + ("UU", "UU" + channel_id[2:])): + try: + page = call("playlistItems", key, part="contentDetails", + playlistId=playlist_id, maxResults=5) + except ApiError as exc: + outcome[kind] = (False, f"{exc} reason={exc.reason!r}") + continue + items = page.get("items") or [] + stamps = [i["contentDetails"].get("videoPublishedAt") for i in items] + exact = all(s and s.endswith("Z") and "T" in s for s in stamps) + outcome[kind] = ( + bool(items) and exact, + f"{len(items)} items, videoPublishedAt exact={exact}, " + f"e.g. {stamps[0] if stamps else 'n/a'}, " + f"total={(page.get('pageInfo') or {}).get('totalResults')}", + ) + + uulf_ok, uulf_detail = outcome["UULF"] + uu_ok, uu_detail = outcome["UU"] + if uulf_ok: + record("uploads playlist usable", True, + f"UULF WORKS -> use it, existing Shorts/livestream filtering carries over\n" + f" UULF: {uulf_detail}\n UU: {uu_detail}") + elif uu_ok: + record("uploads playlist usable", True, + f"UULF REJECTED -> take the UU fallback and filter by duration +\n" + f"liveStreamingDetails (plan.md §3)\n" + f" UULF: {uulf_detail}\n UU: {uu_detail}") + else: + record("uploads playlist usable", False, + f"neither worked\n UULF: {uulf_detail}\n UU: {uu_detail}") + + +def check_durations(key: str, channel_id: str) -> None: + """Durations for a batch of ids -- what feeds .""" + print("\n3. videos.list durations for a batch of ids") + try: + listing = call("playlistItems", key, part="contentDetails", + playlistId="UU" + channel_id[2:], maxResults=50) + except ApiError as exc: + record("durations available", False, f"could not list ids: {exc}") + return + + ids = [i["contentDetails"]["videoId"] for i in listing.get("items") or []] + if not ids: + record("durations available", False, "no video ids to test with") + return + + try: + page = call("videos", key, part="contentDetails", + id=",".join(ids), maxResults=50) + except ApiError as exc: + record("durations available", False, f"{exc} reason={exc.reason!r}") + return + + items = page.get("items") or [] + parsed = [(i["id"], iso8601_seconds(i["contentDetails"].get("duration", ""))) + for i in items] + bad = [vid for vid, secs in parsed if secs is None] + shorts = [vid for vid, secs in parsed if secs is not None and secs <= 120] + record("durations available", not bad and len(items) == len(ids), + f"asked for {len(ids)} ids in 1 call, got {len(items)} back, " + f"{len(bad)} unparseable\n" + f"{len(shorts)} of {len(items)} are <=120s (would be filtered as Shorts)\n" + f"sample: " + ", ".join(f"{v}={s}s" for v, s in parsed[:5])) + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--key", required=True, help="YouTube Data API v3 key") + parser.add_argument("--brother", default=BROTHER, + help=f"channel id whose subscriptions to read (default {BROTHER})") + parser.add_argument("--sample", default=SAMPLE_CHANNEL, + help="channel id to exercise the playlist/duration calls against") + args = parser.parse_args() + + print(f"Phase 0 verification against the live API (~5 quota units of 10,000/day)") + check_subscriptions(args.key, args.brother) + check_uploads_playlist(args.key, args.sample) + check_durations(args.key, args.sample) + + failed = [name for name, ok, _ in results if not ok] + print() + if failed: + print(f"{len(failed)} of {len(results)} FAILED: {', '.join(failed)}") + return 1 + print(f"ALL {len(results)} PASS -- paste the numbers above into plan.md §4.1 and §14") + return 0 + + +if __name__ == "__main__": + sys.exit(main())