diff --git a/plan.md b/plan.md index 74599ac..8122ce4 100644 --- a/plan.md +++ b/plan.md @@ -62,7 +62,7 @@ So: copy the tree, then add, delete and replace. | `youtube_automate` module | Fate in `ytstream` | Note | |---|---|---| -| `discovery.py` | **Lift, then extend** | UULF-feed polling is the cheapest correct incremental source and stays. Gains an API-backed full-catalogue backfill (§3). | +| `discovery.py` | **Lift, then extend** | UULF-feed polling is the cheapest correct incremental source and stays — and at 119 channels it still completes in 8 s (§16). Gains an API-backed 30-day backfill (§3). | | `naming.py` | **Lift unchanged** | Season = upload year, episode = `MMDD*10 + ordinal`. Keep byte-identical so both trees sort the same during the overlap. | | `nfo.py` | **Lift, small change** | Drop `` — measured to accomplish nothing (FINDINGS §6). Keep ``. | | `channels.py` | **Lift, minus artwork fetch** | Artwork moves to the API/`i.ytimg.com` path already built in `add_thumbnails.py`. | @@ -94,19 +94,24 @@ gets **10,000 units/day** shared across everything except `search.list` and `vid have their own 100-call/day buckets. `subscriptions.list`, `playlistItems.list`, `videos.list` and `channels.list` are **1 unit each**, and each returns up to 50 items. -That changes the arithmetic completely: +That changes the arithmetic completely. All figures below use the measured shape of the real account: +**119 subscriptions, 441 long-form videos in a 30-day window** (§16). | Job | Method | Calls | Units | |---|---|---|---| -| Sync subscriptions, hourly | `subscriptions.list` | 24/day (1 page) | 24 | -| Backfill one channel's 30-day window (~20 videos) | `playlistItems.list` @50 | 1 | 1 | -| Durations for those videos | `videos.list` @50 ids | 1 | 1 | -| Steady-state incremental discovery | **RSS feed** | — | **0** | -| Durations for the day's new videos | `videos.list` @50 ids | 1 | 1 | +| Sync subscriptions, hourly | `subscriptions.list` | 3 pages × 24/day | **72/day** | +| Initial backfill, all 119 channels | `playlistItems.list` @50 | 119 | 119 once | +| Durations for those 441 videos | `videos.list` @50 ids | 9 | 9 once | +| Steady-state incremental discovery | **RSS feed** | 119/hour | **0** | +| Durations for the day's ~15 new videos | `videos.list` @50 ids | 1 | 1/day | -A 20-channel initial build costs about **40 units**. Steady state is **~25 units/day**, nearly all of -it the hourly subscription poll. Against 10,000/day this is not a constraint worth thinking about, -provided we never touch `search.list` (100 calls/day, and we have no use for it). +**Initial build: ~130 units. Steady state: ~75 units/day**, nearly all of it the hourly subscription +poll — which is 3 pages rather than 1 precisely because there are 119 subscriptions. Against +10,000/day that is under 1%, so quota is not a constraint worth designing around, provided we never +touch `search.list` (100 calls/day, and we have no use for it). + +If it ever mattered, the lever is obvious: the subscription list changes daily at most, so polling it +hourly is already generous and dropping to every 4 hours would cut the bill by 75%. ### Why this matters more than the quota @@ -115,8 +120,8 @@ extraction per video to learn an upload date whenever it backfills past the RSS each. That is the "scan storm" failure mode the original handover warned about: requests to YouTube from a residential IP, in bulk, for videos nobody asked to watch. -Be honest about the size of the win now that the window is 30 days: at ~20 videos per channel, the -yt-dlp route would be ~400 extractions ≈ 15 minutes, not the 4 hours a 90-day window implied. The +Be honest about the size of the win now that the window is 30 days: at 441 videos, the yt-dlp route +would be ~441 extractions ≈ 17 minutes, not the hours a 90-day window implied. The API is still the right answer — it is documented, keyed, quota-metered, indifferent to PO tokens and SABR, and cannot be rate-limited by YouTube's anti-bot heuristics — but it is now buying *correctness* (exact dates, real durations) far more than it is buying *safety*. @@ -139,23 +144,26 @@ the upload date, an approximate date means episodes numbered into the wrong day caches episode numbers, so fixing it later is a metadata-wipe operation. `playlistItems.list` returns `contentDetails.videoPublishedAt` as an exact RFC-3339 timestamp. Use it. -### One thing to verify before relying on it +### UULF works through the API — verified -`youtube-automate` polls the **UULF** playlist (`UU` with `LF` spliced in), which is undocumented -but excludes Shorts and livestreams at the cheapest possible point — verified in specs.md §4. -Whether `playlistItems.list` accepts a UULF id is **unverified**; only `UU` is documented. +`youtube-automate` polls the **UULF** playlist (`UU` with `LF` spliced in), which is undocumented but +excludes Shorts and livestreams at the cheapest possible point (specs.md §4). Whether +`playlistItems.list` would accept an undocumented playlist id was the plan's main open technical +question. **It does** — verified 2026-08-12 against Pitch Side, returning exact +`contentDetails.videoPublishedAt` timestamps: -- If UULF works: use it, and the existing filtering carries over unchanged. -- If it 404s: fall back to `UU` (definitely works, includes Shorts and livestreams) and filter with - the `videos.list` call we are making anyway — `contentDetails.duration < min_duration_seconds` - drops Shorts, and the presence of `liveStreamingDetails` drops streams. +| Playlist | `totalResults` | | +|---|---|---| +| `UULFjCJ2LaOIsPzOoXUTMDI3wg` | **1,249** | long-form only | +| `UUjCJ2LaOIsPzOoXUTMDI3wg` | **2,724** | everything | -Either way it is one extra unit per 50 videos. Verify with a single curl on day one: +So UULF excludes 54% of that catalogue, and the existing Shorts/livestream filtering carries over +unchanged with no fallback needed. The `UU`-plus-duration-filter path in earlier drafts is not +required — though it remains the correct contingency if YouTube ever retires UULF, since `UU` is the +documented one. -```sh -curl -s "https://www.googleapis.com/youtube/v3/playlistItems?part=contentDetails\ -&playlistId=UULF2EvK7nHUOEw1IvWFpTourQ&maxResults=5&key=$KEY" | head -40 -``` +Corroborating, from the `videos.list` check: of 50 consecutive `UU` uploads, **38 were ≤120 s**. The +`min_duration_seconds = 120` default is filtering a real majority, not an edge case. --- @@ -238,11 +246,11 @@ curl -s "https://www.googleapis.com/youtube/v3/subscriptions?part=snippet\ A 200 with `items[]` confirms the design on the real account. A 403 means the privacy checkbox has not taken effect — and per the finding above, there is no way around it. -**This call also produces the number that sets `subsync_max_new`** (§4.4): `pageInfo.totalResults` -is how many channels he is subscribed to today, which is the one input the cap needs and which -cannot be discovered any other way. Record it here when known: +**Verified 2026-08-12: 200 with items.** The feature works as designed. -> `totalResults` = **TBD — Phase 0** +> **119 subscriptions** actually returned, across 3 pages. +> `pageInfo.totalResults` claims **127** — see §15; trust the fetched list, not the count. +> → `subsync_max_new` = **25**. ### 4.2 Fallback if `channelId` turns out not to work @@ -298,17 +306,19 @@ not something to discover by accident. So: as `pending_approval`, alerts, and waits for a click in the admin UI. - The cap applies per sync run, so ordinary drip-feed additions never trip it. -**Separate the first sync from steady state.** Otherwise the cap always trips on day one, whatever -it is set to, and the guard trains everyone to ignore it. So a source's **first** sync is an -explicit bulk import: the admin UI shows the whole list with per-channel checkboxes and a count of -the episodes it implies, and nothing is subscribed until someone confirms. From the second sync -onwards the cap is a runaway guard, and `subsync_max_new` is set from the day-one `totalResults` -(§4.1) — a sensible rule is **max(10, ceil(totalResults × 0.2))**, so a genuine burst of activity -gets through but an order-of-magnitude jump does not. Provisional default **25** until Phase 0 -produces the real number. +**Separate the first sync from steady state.** Otherwise the cap always trips on day one, whatever it +is set to, and the guard trains everyone to ignore it. This is now certain rather than hypothetical: +the measured list is **119 channels**, so the first sync trips any sane cap. So a source's **first** +sync is an explicit bulk import — the admin UI shows all 119 with per-channel checkboxes and the +episode count each implies, and nothing is subscribed until someone confirms. It is also the natural +place to *not* take all 119: a personal subscription list accumulated over years is not automatically +a list of things worth putting in a media library. + +From the second sync onwards the cap is a runaway guard. **`subsync_max_new` = 25**, from +`max(10, ceil(119 × 0.2))` — a genuine burst gets through, an order-of-magnitude jump does not. **Runaway removals.** A transient 403, a network blip, or him re-ticking the privacy box all look -like "he unsubscribed from everything". Deleting 20 channels' worth of Jellyfin metadata on that +like "he unsubscribed from everything". Deleting 119 channels' worth of Jellyfin metadata on that basis would be unrecoverable in any pleasant way. So: - **A 403, a 5xx, a timeout, or a zero-item 200 is never treated as a removal.** It increments the @@ -365,23 +375,48 @@ this is about *retention*; keeping both would mean backfilling 90 days and then two thirds of it. If the intent was really "go back 3 months, keep everything", set `retention_days = 90` and this section still holds — only the numbers move. -### What this actually costs, measured +### What this actually costs — measured across the real subscription list -The upload rates matter more than the total catalogue size, so I measured them from the live feeds -on 2026-08-12 rather than guessing: +Not extrapolated. On 2026-08-12 I pulled all 119 subscriptions and probed every one of their UULF +feeds: -| Channel | UULF (long-form) | UC feed (everything) | UULF filters | -|---|---|---|---| -| Pitch Side | 0.60/day → **~18 per 30d** | 2.33/day → ~70 per 30d | 74% | -| The Pyramid Podcast | 0.77/day → **~23 per 30d** | 1.91/day → ~57 per 30d | 60% | +| | | +|---|---| +| Channels subscribed | **119** (117 with working feeds; 2 return HTTP errors) | +| **Long-form videos in the last 30 days, all channels** | **441** | +| Median per channel | **1** | +| Channels with **zero** long-form uploads in 30 days | **52 of 117** | +| Busiest | penguinz0 57, Daggerwin 30, GothamChess 30, FORMULA 1 28 | +| Poll cycle for all 119 feeds | **7.9 s** wall, 8 threads (15 feeds/s) | -So a 30-day window is roughly **20 episodes per channel**, and 20 channels is **~400 episodes, -~1,200 files.** That is two orders of magnitude below the 20,000 the previous draft was braced for, -and it makes the entire scale section boring — which is the point. Pitch Side's 1,249-video back -catalogue simply never enters the library. +So the whole library is **~441 episodes, ~1,300 files** — smaller than the 20-channel guess this +section previously carried, despite six times as many channels, because the distribution is wildly +skewed. Jellyfin will not notice this. Neither will the network: an hourly poll of all 119 feeds costs +eight seconds. -It also confirms the UULF feed is doing real work: it filters 60–74% of what the channel publishes, -and that Shorts-and-livestreams majority is exactly what nobody wants as Jellyfin episodes. +UULF is doing substantial work. For Pitch Side, `playlistItems.list` reports **1,249** items in UULF +against **2,724** in UU — it excludes 54% of the catalogue — and of 50 consecutive UU uploads, **38 +were ≤120 s**. The Shorts majority is real, and it is exactly what nobody wants as Jellyfin episodes. + +### The skew creates a problem the window alone does not solve + +**52 of 117 channels would produce an empty series.** A channel that uploads every six weeks has +nothing inside a 30-day window, so it appears in Jellyfin as a show with no episodes — and worse, it +would flicker in and out of existence as its one video ages past the line. With a median of 1 video +per channel per month, this is the common case, not an edge case. + +Recommended fix, and it is a small one: make retention **`max(30 days, the N most recent videos)`** +via a `min_keep_videos` setting, default **5**. + +- A channel uploading twice a year keeps its last 5 videos permanently visible instead of showing an + empty shelf. +- A channel uploading daily is unaffected — 30 days already exceeds 5 videos. +- Cost: `441 + (52 × 5) ≈ 700` episodes. Still nothing. +- It removes the flicker, because a video only leaves once 5 newer ones exist. + +The alternative — per-channel `retention_days` overrides — is already in the schema but means hand- +tuning 119 channels, which nobody will do. **Also materialise a channel's directory only once it has +at least one in-window video**, so a genuinely dead channel does not create an empty shelf at all. ### A useful side effect: RSS nearly covers the whole window @@ -432,7 +467,7 @@ to be handling a few dozen calls a day rather than thousands. unreachable from the network by construction. 5. **Backfill stays resumable** (`channel.backfill_cursor`, commit per page of 50) even though a 30-day backfill is now one or two pages. It costs a column and it is the difference between a - crashed first sync resuming and starting over across 20 channels. + crashed first sync resuming and starting over across 119 channels. --- @@ -541,9 +576,10 @@ terminal states matter and are not the same thing: `last_run_at`), including **`retention_days`** — now **30**, and governing both the aging-out sweep and the backfill reach (§5). Drop `disk_cap_gb`, `write_subs`, `sub_langs`, `sponsorblock_mark`, `max_attempts`, `backfill_days` — download-era concepts. Add `youtube_api_key`, `subsync_max_new` -(25, provisional — §4.4), `subsync_missing_threshold` (3), `proxy_base_url` -(`http://127.0.0.1:8099`), and `backfill_max_videos` (**300**) purely as a runaway guard on a channel -that turns out to upload 50 times a day. +(**25**, derived in §4.4), `subsync_missing_threshold` (3), `proxy_base_url` +(`http://127.0.0.1:8099`), `backfill_max_videos` (**300**) purely as a runaway guard on a channel that +turns out to upload 50 times a day, and **`min_keep_videos` (5)** — the fix for the 52 empty series +measured in §5. **No migration from `subs.db`.** It holds 2 channels and 18 videos. Re-subscribe by hand and let the backfill do the rest; a migration script would be more code than the data is worth. @@ -658,6 +694,12 @@ Four keys, all in `setting`: `youtube_api_key` (new), `jellyfin_api_key`, `sessi The YouTube API key is restricted to the YouTube Data API v3 and has read-only reach over public data. Worst case on leak is quota exhaustion; rotation is a two-minute job in the console. +**The Phase 0 key is treated as disposable and rotated once the service is built** (Tom's call, and +the right one — it has been pasted into a terminal and a chat transcript while debugging). That makes +one thing a requirement rather than a nicety: **rotating the key must not need a restart or a code +change.** Read it from the `setting` table at the point of use, not once at process start, so the +admin UI's "save" is the whole rotation procedure. + --- ## 12. Decommissioning `youtube-automate` @@ -693,16 +735,9 @@ is the thing that keeps playback working as YouTube changes. Each phase ends in something checkable. Do not start the next one until it does. -**Phase 0 — verify the assumptions the plan rests on.** Mostly **done, 2026-08-12** — see §16 for -what was verified and what is still blocked. Remaining: paste the API key in and run - -```sh -python3 /opt/ytstream/tools/verify_api.py --key AIza... -``` - -which performs all three API checks, costs ~5 quota units, prints the `subsync_max_new` value, and -exits non-zero if anything fails. -→ *Done when: `verify_api.py` reports ALL PASS and its numbers are written into §4.1 and §16.* +**Phase 0 — verify the assumptions the plan rests on. ✅ COMPLETE 2026-08-12.** All three API checks +pass; `tools/verify_api.py` reproduces them in one command. Results and the numbers they changed are +in §16. **Phase 1 — skeleton and lift.** Fork the tree, new package name, new DB path, new schema (§7), lifted modules and their tests passing. No new behaviour. @@ -745,10 +780,9 @@ red — test these before trusting the deletion path, not after.* (§5). This supersedes the earlier "3 months or 300 videos" answer, which was about backfill depth; holding both would mean backfilling 90 days and deleting two thirds of it immediately. If the intent was "reach back 3 months and keep it", `retention_days = 90` is a one-line change and §5 - still holds. Measured consequence: **~400 episodes total**, not 20,000. -3. **`subsync_max_new`:** still open — set from `pageInfo.totalResults` on the first - `subscriptions.list` call, which `tools/verify_api.py` prints. Provisional 25; rule of thumb - `max(10, ceil(totalResults × 0.2))` (§4.4). + still holds. Measured consequence: **441 episodes total** across 119 channels, not 20,000. +3. **`subsync_max_new` = 25** — closed by Phase 0. Measured 119 subscriptions, and + `max(10, ceil(119 × 0.2)) = 25` (§4.4). 4. **Brother gets admin access:** yes, on the **existing shared password** — no `user` table, no per-account credentials (§4.6). `web/auth.py` carries over untouched. 5. **Unsubscribe deletes the channel** rather than deactivating it, since re-subscribing is one API @@ -786,58 +820,70 @@ Things already paid for once. All of these are verified. enabled on the project (and carries an `activationUrl` naming the project number); `API_KEY_SERVICE_BLOCKED` means the API is enabled but *this key's* restrictions exclude it. They are fixed on different console screens. `tools/verify_api.py` distinguishes them and prints the fix. +- **Console changes take up to ~2 minutes to propagate.** Measured: a key restriction change was still + returning `API_KEY_SERVICE_BLOCKED` at 0 s, 30 s and 60 s, and succeeded at **90 s**. Do not + conclude a setting is wrong until a couple of minutes have passed — retry before re-editing. +- **`subscriptions.list` `pageInfo.totalResults` overcounts.** It reported **127** while full + pagination returned **119** distinct channels, and 2 of those 119 have feeds that HTTP-error — + terminated or private channels still counted as subscriptions. Never treat `totalResults` as the + list length; page to exhaustion and count what arrives. Anything sized off it (a progress bar, a + cap, an "is the list complete" check) will be wrong. --- -## 16. Phase 0 results — 2026-08-12 +## 16. Phase 0 results — COMPLETE, 2026-08-12 -### Done +All three assumptions the plan rested on are verified against the live API. Nothing in the design had +to change; several numbers did. -- **Bare repo initialised.** `/disks/git-repos/ytstream.git` existed as an empty directory; it is now - a real `--bare --shared=group` repo with `HEAD` → `refs/heads/main`, matching - `youtube-automate.git`. `/opt/ytstream` is a checkout with `origin` pointed at it. -- **The mirrored channel is resolved.** `@cflux1030` → `UCPcTWaLV8zwx4WP4QExHj4Q` ("C Flux", 55 - subscribers). No `channels.list` call needed at build time. -- **There is no scraping fallback** — `/@cflux1030/channels` serves the Home tab with zero channel - ids (§15). The privacy checkbox is mandatory. -- **UULF feeds work for both existing channels**, and the filtering is substantial — 60–74% of what - these channels publish is Shorts or livestreams that UULF correctly excludes. -- **Upload rates measured**, which is what sized §5: ~18–23 long-form videos per channel per 30 days, - so ~400 episodes for a 20-channel library. Pitch Side's UULF feed spans **23.3 days in 15 entries**, - meaning RSS alone nearly covers the retention window. -- **`tools/verify_api.py` written and self-tested** (ISO-8601 duration parser unit-checked against six - cases; argparse and import verified). It runs all three API checks in one command. +### The three API checks -### Key created, project 510818173753 — two setup steps deep, one to go - -Progress on the key itself, all of it diagnosed from the error bodies: - -1. **Key created** and reaching Google — it authenticates, so the key string is good. -2. **`SERVICE_DISABLED`** — YouTube Data API v3 was not enabled on project `510818173753`. Fixed by - enabling it. -3. **`API_KEY_SERVICE_BLOCKED`** ← *current state.* The API is now enabled, but the key's own API - restrictions exclude it, so every method returns "Requests to this API youtube method … are - blocked". Fix at - `https://console.cloud.google.com/apis/credentials?project=510818173753` → the key → API - restrictions → *Don't restrict key*, or tick YouTube Data API v3. - -The ordering trap is worth remembering rather than rediscovering: the API must be enabled **before** -the key can be restricted to it, because it is absent from the picker until then. §4.1 step 4 now -says so. - -Until the key answers, these three assumptions remain unverified: - -| # | Assumption | Consequence if it fails | +| # | Assumption | Result | |---|---|---| -| 1 | `subscriptions.list?channelId=` returns his subscriptions | **Feature is impossible as designed.** Falls back to §4.2, all of which need him to do something. | -| 2 | `playlistItems.list` accepts a `UULF…` playlist id | Take the documented `UU` fallback and filter Shorts/livestreams via `videos.list` duration + `liveStreamingDetails`. Costs one extra unit per 50. | -| 3 | `videos.list` returns parseable `contentDetails.duration` | No `` in NFOs without a yt-dlp extraction per video. Degrades, does not block. | +| 1 | `subscriptions.list?channelId=` returns his subscriptions with only an API key | **PASS.** 200, **119** channels across 3 pages. The feature is possible exactly as designed. | +| 2 | `playlistItems.list` accepts an undocumented `UULF…` playlist id | **PASS.** 1,249 items with exact `videoPublishedAt`. No `UU` fallback needed (§3). | +| 3 | `videos.list` returns parseable `contentDetails.duration` | **PASS.** 50 ids in one call, 50 returned, 0 unparseable. | -Only #1 is a genuine blocker, and it also carries the number that sets `subsync_max_new`. Two -external prerequisites remain: **the key's API restriction** (above), and **his brother unchecking -"Keep all my subscriptions private"** — note that nothing so far has tested the second, because every -call has failed at the key before reaching YouTube's privacy check. Then one command closes Phase 0: +Reproduce with `python3 tools/verify_api.py --key ` (~5 quota units). -```sh -python3 /opt/ytstream/tools/verify_api.py --key AIza... -``` +### The numbers that came out of it + +- **119 subscriptions**, not the ~20 the plan assumed — and `pageInfo.totalResults` claims 127 (§15). +- **441 long-form videos across all 119 channels in the last 30 days.** That is the library size. +- **Median 1 video per channel per 30 days**, and **52 of 117 channels uploaded nothing** — the + distribution is severely skewed, which is what produced the `min_keep_videos` recommendation in §5. +- **UULF excludes 54%** of Pitch Side's catalogue (1,249 of 2,724), and **38 of 50** consecutive `UU` + uploads were ≤120 s. The Shorts filter earns its place. +- **All 119 feeds poll in 7.9 s** (8 threads). Hourly polling is free. +- `subsync_max_new` = **25**. + +### Also done + +- **Bare repo initialised.** `/disks/git-repos/ytstream.git` was an empty directory; it is now a real + `--bare --shared=group` repo with `HEAD` → `refs/heads/main`, matching `youtube-automate.git`. + `/opt/ytstream` is a checkout with `origin` pointed at it. +- **The mirrored channel is resolved:** `@cflux1030` → `UCPcTWaLV8zwx4WP4QExHj4Q` ("C Flux"). No + `channels.list` call needed at build time. +- **There is no scraping fallback** — `/@cflux1030/channels` serves the Home tab with zero channel ids + (§15), so the privacy setting is mandatory. It is now confirmed to be off, since the API returns his + list. +- **`tools/verify_api.py`** written, self-tested, and hardened against the two setup failures below. + +### Getting the key working took three attempts, and the errors were misleading + +Recorded because the next person to set up a Google Cloud project will hit the same wall. All three +states return HTTP 403 `forbidden`, and the distinguishing signal is `error.details[].reason`: + +1. **`SERVICE_DISABLED`** — YouTube Data API v3 not enabled on project `510818173753`. Carries an + `activationUrl` naming the project. +2. **`API_KEY_SERVICE_BLOCKED`** — API enabled, but the *key's own* API restrictions exclude it. A + different console screen entirely. The ordering is the trap: YouTube Data API v3 is absent from the + key's restriction picker until the API is enabled on the project, so restricting first yields a key + that blocks the only API it exists for. §4.1 step 4 now says enable-then-restrict. +3. **Propagation delay** — after the restriction was fixed, calls still failed at 0 s, 30 s and 60 s, + and succeeded at **90 s**. Nothing was wrong at that point except impatience. + +### What remains before Phase 1 + +Nothing. Phase 0 is closed. The one open decision is §14 item 2 — whether `min_keep_videos` (§5) is +wanted, which is a five-minute change either way and does not block starting Phase 1.