Phase 0 complete: all three API assumptions verified, and 119 subscriptions
The key works. All three assumptions the plan rested on hold, so nothing in the
design had to change — but the measurements moved several numbers by a lot, and
one of them exposed a design problem the plan did not have an answer for.
Verified against the live API:
subscriptions.list?channelId= 200, 119 channels over 3 pages. An API key is
sufficient; no OAuth, no consent screen.
playlistItems.list on UULF Accepts the undocumented long-form playlist id
and returns exact videoPublishedAt. 1,249 items
against 2,724 in UU, so it excludes 54% of the
catalogue. The UU-plus-duration-filter fallback
is not needed.
videos.list durations 50 ids in one call, 50 back, 0 unparseable. Of
those 50 consecutive UU uploads, 38 were <=120s,
which is the Shorts filter earning its keep.
The account has 119 subscriptions, not the ~20 this plan assumed, so rather than
extrapolate I probed all 119 UULF feeds: 441 long-form videos in the last 30
days, median 1 per channel, and all 119 feeds polled in 7.9s on 8 threads. The
library is therefore smaller than the previous 20-channel estimate despite six
times the channels, because the distribution is severely skewed. Quota, scan cost
and poll cost are all now measured rather than guessed, and none of them is a
constraint: ~130 units to build, ~75/day steady state, under 1% of the budget.
The problem that fell out: 52 of 117 channels uploaded nothing long-form in 30
days, so they would appear in Jellyfin as empty series, and a channel uploading
every six weeks would flicker in and out as its one video crossed the retention
line. With a median of 1 upload a month that is the common case, not an edge
case. Recommended fix in §5 — retention becomes max(30 days, N most recent
videos) via min_keep_videos, default 5, which costs ~260 extra episodes and
removes the flicker entirely because a video only leaves once 5 newer ones exist.
Flagged as a decision rather than applied unilaterally.
Also recorded, because it cost three attempts to get the key working and every
failure was an indistinguishable 403 forbidden: SERVICE_DISABLED means the API is
not enabled on the project, API_KEY_SERVICE_BLOCKED means the key's own
restrictions exclude it, they are fixed on different console screens, and the
signal is in error.details[].reason rather than error.errors[].reason. The
ordering matters — the API must be enabled before the key can be restricted to
it, because it is absent from the picker until then. And after the fix, calls
still failed at 0s, 30s and 60s and succeeded at 90s; nothing was wrong except
impatience.
One more trap worth having in writing: pageInfo.totalResults reported 127 while
pagination returned 119 distinct channels, and 2 of those 119 have feeds that
HTTP-error. Terminated and private channels are still counted as subscriptions,
so anything sized off totalResults will be wrong.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5
parent
f3d70f1c87
commit
f640c064c6
@@ -62,7 +62,7 @@ So: copy the tree, then add, delete and replace.
|
|||||||
|
|
||||||
| `youtube_automate` module | Fate in `ytstream` | Note |
|
| `youtube_automate` module | Fate in `ytstream` | Note |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `discovery.py` | **Lift, then extend** | UULF-feed polling is the cheapest correct incremental source and stays. Gains an API-backed full-catalogue backfill (§3). |
|
| `discovery.py` | **Lift, then extend** | UULF-feed polling is the cheapest correct incremental source and stays — and at 119 channels it still completes in 8 s (§16). Gains an API-backed 30-day backfill (§3). |
|
||||||
| `naming.py` | **Lift unchanged** | Season = upload year, episode = `MMDD*10 + ordinal`. Keep byte-identical so both trees sort the same during the overlap. |
|
| `naming.py` | **Lift unchanged** | Season = upload year, episode = `MMDD*10 + ordinal`. Keep byte-identical so both trees sort the same during the overlap. |
|
||||||
| `nfo.py` | **Lift, small change** | Drop `<fileinfo><streamdetails>` — measured to accomplish nothing (FINDINGS §6). Keep `<durationinseconds>`. |
|
| `nfo.py` | **Lift, small change** | Drop `<fileinfo><streamdetails>` — measured to accomplish nothing (FINDINGS §6). Keep `<durationinseconds>`. |
|
||||||
| `channels.py` | **Lift, minus artwork fetch** | Artwork moves to the API/`i.ytimg.com` path already built in `add_thumbnails.py`. |
|
| `channels.py` | **Lift, minus artwork fetch** | Artwork moves to the API/`i.ytimg.com` path already built in `add_thumbnails.py`. |
|
||||||
@@ -94,19 +94,24 @@ gets **10,000 units/day** shared across everything except `search.list` and `vid
|
|||||||
have their own 100-call/day buckets. `subscriptions.list`, `playlistItems.list`, `videos.list` and
|
have their own 100-call/day buckets. `subscriptions.list`, `playlistItems.list`, `videos.list` and
|
||||||
`channels.list` are **1 unit each**, and each returns up to 50 items.
|
`channels.list` are **1 unit each**, and each returns up to 50 items.
|
||||||
|
|
||||||
That changes the arithmetic completely:
|
That changes the arithmetic completely. All figures below use the measured shape of the real account:
|
||||||
|
**119 subscriptions, 441 long-form videos in a 30-day window** (§16).
|
||||||
|
|
||||||
| Job | Method | Calls | Units |
|
| Job | Method | Calls | Units |
|
||||||
|---|---|---|---|
|
|---|---|---|---|
|
||||||
| Sync subscriptions, hourly | `subscriptions.list` | 24/day (1 page) | 24 |
|
| Sync subscriptions, hourly | `subscriptions.list` | 3 pages × 24/day | **72/day** |
|
||||||
| Backfill one channel's 30-day window (~20 videos) | `playlistItems.list` @50 | 1 | 1 |
|
| Initial backfill, all 119 channels | `playlistItems.list` @50 | 119 | 119 once |
|
||||||
| Durations for those videos | `videos.list` @50 ids | 1 | 1 |
|
| Durations for those 441 videos | `videos.list` @50 ids | 9 | 9 once |
|
||||||
| Steady-state incremental discovery | **RSS feed** | — | **0** |
|
| Steady-state incremental discovery | **RSS feed** | 119/hour | **0** |
|
||||||
| Durations for the day's new videos | `videos.list` @50 ids | 1 | 1 |
|
| Durations for the day's ~15 new videos | `videos.list` @50 ids | 1 | 1/day |
|
||||||
|
|
||||||
A 20-channel initial build costs about **40 units**. Steady state is **~25 units/day**, nearly all of
|
**Initial build: ~130 units. Steady state: ~75 units/day**, nearly all of it the hourly subscription
|
||||||
it the hourly subscription poll. Against 10,000/day this is not a constraint worth thinking about,
|
poll — which is 3 pages rather than 1 precisely because there are 119 subscriptions. Against
|
||||||
provided we never touch `search.list` (100 calls/day, and we have no use for it).
|
10,000/day that is under 1%, so quota is not a constraint worth designing around, provided we never
|
||||||
|
touch `search.list` (100 calls/day, and we have no use for it).
|
||||||
|
|
||||||
|
If it ever mattered, the lever is obvious: the subscription list changes daily at most, so polling it
|
||||||
|
hourly is already generous and dropping to every 4 hours would cut the bill by 75%.
|
||||||
|
|
||||||
### Why this matters more than the quota
|
### Why this matters more than the quota
|
||||||
|
|
||||||
@@ -115,8 +120,8 @@ extraction per video to learn an upload date whenever it backfills past the RSS
|
|||||||
each. That is the "scan storm" failure mode the original handover warned about: requests to YouTube
|
each. That is the "scan storm" failure mode the original handover warned about: requests to YouTube
|
||||||
from a residential IP, in bulk, for videos nobody asked to watch.
|
from a residential IP, in bulk, for videos nobody asked to watch.
|
||||||
|
|
||||||
Be honest about the size of the win now that the window is 30 days: at ~20 videos per channel, the
|
Be honest about the size of the win now that the window is 30 days: at 441 videos, the yt-dlp route
|
||||||
yt-dlp route would be ~400 extractions ≈ 15 minutes, not the 4 hours a 90-day window implied. The
|
would be ~441 extractions ≈ 17 minutes, not the hours a 90-day window implied. The
|
||||||
API is still the right answer — it is documented, keyed, quota-metered, indifferent to PO tokens and
|
API is still the right answer — it is documented, keyed, quota-metered, indifferent to PO tokens and
|
||||||
SABR, and cannot be rate-limited by YouTube's anti-bot heuristics — but it is now buying
|
SABR, and cannot be rate-limited by YouTube's anti-bot heuristics — but it is now buying
|
||||||
*correctness* (exact dates, real durations) far more than it is buying *safety*.
|
*correctness* (exact dates, real durations) far more than it is buying *safety*.
|
||||||
@@ -139,23 +144,26 @@ the upload date, an approximate date means episodes numbered into the wrong day
|
|||||||
caches episode numbers, so fixing it later is a metadata-wipe operation. `playlistItems.list`
|
caches episode numbers, so fixing it later is a metadata-wipe operation. `playlistItems.list`
|
||||||
returns `contentDetails.videoPublishedAt` as an exact RFC-3339 timestamp. Use it.
|
returns `contentDetails.videoPublishedAt` as an exact RFC-3339 timestamp. Use it.
|
||||||
|
|
||||||
### One thing to verify before relying on it
|
### UULF works through the API — verified
|
||||||
|
|
||||||
`youtube-automate` polls the **UULF** playlist (`UU` with `LF` spliced in), which is undocumented
|
`youtube-automate` polls the **UULF** playlist (`UU` with `LF` spliced in), which is undocumented but
|
||||||
but excludes Shorts and livestreams at the cheapest possible point — verified in specs.md §4.
|
excludes Shorts and livestreams at the cheapest possible point (specs.md §4). Whether
|
||||||
Whether `playlistItems.list` accepts a UULF id is **unverified**; only `UU` is documented.
|
`playlistItems.list` would accept an undocumented playlist id was the plan's main open technical
|
||||||
|
question. **It does** — verified 2026-08-12 against Pitch Side, returning exact
|
||||||
|
`contentDetails.videoPublishedAt` timestamps:
|
||||||
|
|
||||||
- If UULF works: use it, and the existing filtering carries over unchanged.
|
| Playlist | `totalResults` | |
|
||||||
- If it 404s: fall back to `UU` (definitely works, includes Shorts and livestreams) and filter with
|
|---|---|---|
|
||||||
the `videos.list` call we are making anyway — `contentDetails.duration < min_duration_seconds`
|
| `UULFjCJ2LaOIsPzOoXUTMDI3wg` | **1,249** | long-form only |
|
||||||
drops Shorts, and the presence of `liveStreamingDetails` drops streams.
|
| `UUjCJ2LaOIsPzOoXUTMDI3wg` | **2,724** | everything |
|
||||||
|
|
||||||
Either way it is one extra unit per 50 videos. Verify with a single curl on day one:
|
So UULF excludes 54% of that catalogue, and the existing Shorts/livestream filtering carries over
|
||||||
|
unchanged with no fallback needed. The `UU`-plus-duration-filter path in earlier drafts is not
|
||||||
|
required — though it remains the correct contingency if YouTube ever retires UULF, since `UU` is the
|
||||||
|
documented one.
|
||||||
|
|
||||||
```sh
|
Corroborating, from the `videos.list` check: of 50 consecutive `UU` uploads, **38 were ≤120 s**. The
|
||||||
curl -s "https://www.googleapis.com/youtube/v3/playlistItems?part=contentDetails\
|
`min_duration_seconds = 120` default is filtering a real majority, not an edge case.
|
||||||
&playlistId=UULF2EvK7nHUOEw1IvWFpTourQ&maxResults=5&key=$KEY" | head -40
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -238,11 +246,11 @@ curl -s "https://www.googleapis.com/youtube/v3/subscriptions?part=snippet\
|
|||||||
A 200 with `items[]` confirms the design on the real account. A 403 means the privacy checkbox has
|
A 200 with `items[]` confirms the design on the real account. A 403 means the privacy checkbox has
|
||||||
not taken effect — and per the finding above, there is no way around it.
|
not taken effect — and per the finding above, there is no way around it.
|
||||||
|
|
||||||
**This call also produces the number that sets `subsync_max_new`** (§4.4): `pageInfo.totalResults`
|
**Verified 2026-08-12: 200 with items.** The feature works as designed.
|
||||||
is how many channels he is subscribed to today, which is the one input the cap needs and which
|
|
||||||
cannot be discovered any other way. Record it here when known:
|
|
||||||
|
|
||||||
> `totalResults` = **TBD — Phase 0**
|
> **119 subscriptions** actually returned, across 3 pages.
|
||||||
|
> `pageInfo.totalResults` claims **127** — see §15; trust the fetched list, not the count.
|
||||||
|
> → `subsync_max_new` = **25**.
|
||||||
|
|
||||||
### 4.2 Fallback if `channelId` turns out not to work
|
### 4.2 Fallback if `channelId` turns out not to work
|
||||||
|
|
||||||
@@ -298,17 +306,19 @@ not something to discover by accident. So:
|
|||||||
as `pending_approval`, alerts, and waits for a click in the admin UI.
|
as `pending_approval`, alerts, and waits for a click in the admin UI.
|
||||||
- The cap applies per sync run, so ordinary drip-feed additions never trip it.
|
- The cap applies per sync run, so ordinary drip-feed additions never trip it.
|
||||||
|
|
||||||
**Separate the first sync from steady state.** Otherwise the cap always trips on day one, whatever
|
**Separate the first sync from steady state.** Otherwise the cap always trips on day one, whatever it
|
||||||
it is set to, and the guard trains everyone to ignore it. So a source's **first** sync is an
|
is set to, and the guard trains everyone to ignore it. This is now certain rather than hypothetical:
|
||||||
explicit bulk import: the admin UI shows the whole list with per-channel checkboxes and a count of
|
the measured list is **119 channels**, so the first sync trips any sane cap. So a source's **first**
|
||||||
the episodes it implies, and nothing is subscribed until someone confirms. From the second sync
|
sync is an explicit bulk import — the admin UI shows all 119 with per-channel checkboxes and the
|
||||||
onwards the cap is a runaway guard, and `subsync_max_new` is set from the day-one `totalResults`
|
episode count each implies, and nothing is subscribed until someone confirms. It is also the natural
|
||||||
(§4.1) — a sensible rule is **max(10, ceil(totalResults × 0.2))**, so a genuine burst of activity
|
place to *not* take all 119: a personal subscription list accumulated over years is not automatically
|
||||||
gets through but an order-of-magnitude jump does not. Provisional default **25** until Phase 0
|
a list of things worth putting in a media library.
|
||||||
produces the real number.
|
|
||||||
|
From the second sync onwards the cap is a runaway guard. **`subsync_max_new` = 25**, from
|
||||||
|
`max(10, ceil(119 × 0.2))` — a genuine burst gets through, an order-of-magnitude jump does not.
|
||||||
|
|
||||||
**Runaway removals.** A transient 403, a network blip, or him re-ticking the privacy box all look
|
**Runaway removals.** A transient 403, a network blip, or him re-ticking the privacy box all look
|
||||||
like "he unsubscribed from everything". Deleting 20 channels' worth of Jellyfin metadata on that
|
like "he unsubscribed from everything". Deleting 119 channels' worth of Jellyfin metadata on that
|
||||||
basis would be unrecoverable in any pleasant way. So:
|
basis would be unrecoverable in any pleasant way. So:
|
||||||
|
|
||||||
- **A 403, a 5xx, a timeout, or a zero-item 200 is never treated as a removal.** It increments the
|
- **A 403, a 5xx, a timeout, or a zero-item 200 is never treated as a removal.** It increments the
|
||||||
@@ -365,23 +375,48 @@ this is about *retention*; keeping both would mean backfilling 90 days and then
|
|||||||
two thirds of it. If the intent was really "go back 3 months, keep everything", set
|
two thirds of it. If the intent was really "go back 3 months, keep everything", set
|
||||||
`retention_days = 90` and this section still holds — only the numbers move.
|
`retention_days = 90` and this section still holds — only the numbers move.
|
||||||
|
|
||||||
### What this actually costs, measured
|
### What this actually costs — measured across the real subscription list
|
||||||
|
|
||||||
The upload rates matter more than the total catalogue size, so I measured them from the live feeds
|
Not extrapolated. On 2026-08-12 I pulled all 119 subscriptions and probed every one of their UULF
|
||||||
on 2026-08-12 rather than guessing:
|
feeds:
|
||||||
|
|
||||||
| Channel | UULF (long-form) | UC feed (everything) | UULF filters |
|
| | |
|
||||||
|---|---|---|---|
|
|---|---|
|
||||||
| Pitch Side | 0.60/day → **~18 per 30d** | 2.33/day → ~70 per 30d | 74% |
|
| Channels subscribed | **119** (117 with working feeds; 2 return HTTP errors) |
|
||||||
| The Pyramid Podcast | 0.77/day → **~23 per 30d** | 1.91/day → ~57 per 30d | 60% |
|
| **Long-form videos in the last 30 days, all channels** | **441** |
|
||||||
|
| Median per channel | **1** |
|
||||||
|
| Channels with **zero** long-form uploads in 30 days | **52 of 117** |
|
||||||
|
| Busiest | penguinz0 57, Daggerwin 30, GothamChess 30, FORMULA 1 28 |
|
||||||
|
| Poll cycle for all 119 feeds | **7.9 s** wall, 8 threads (15 feeds/s) |
|
||||||
|
|
||||||
So a 30-day window is roughly **20 episodes per channel**, and 20 channels is **~400 episodes,
|
So the whole library is **~441 episodes, ~1,300 files** — smaller than the 20-channel guess this
|
||||||
~1,200 files.** That is two orders of magnitude below the 20,000 the previous draft was braced for,
|
section previously carried, despite six times as many channels, because the distribution is wildly
|
||||||
and it makes the entire scale section boring — which is the point. Pitch Side's 1,249-video back
|
skewed. Jellyfin will not notice this. Neither will the network: an hourly poll of all 119 feeds costs
|
||||||
catalogue simply never enters the library.
|
eight seconds.
|
||||||
|
|
||||||
It also confirms the UULF feed is doing real work: it filters 60–74% of what the channel publishes,
|
UULF is doing substantial work. For Pitch Side, `playlistItems.list` reports **1,249** items in UULF
|
||||||
and that Shorts-and-livestreams majority is exactly what nobody wants as Jellyfin episodes.
|
against **2,724** in UU — it excludes 54% of the catalogue — and of 50 consecutive UU uploads, **38
|
||||||
|
were ≤120 s**. The Shorts majority is real, and it is exactly what nobody wants as Jellyfin episodes.
|
||||||
|
|
||||||
|
### The skew creates a problem the window alone does not solve
|
||||||
|
|
||||||
|
**52 of 117 channels would produce an empty series.** A channel that uploads every six weeks has
|
||||||
|
nothing inside a 30-day window, so it appears in Jellyfin as a show with no episodes — and worse, it
|
||||||
|
would flicker in and out of existence as its one video ages past the line. With a median of 1 video
|
||||||
|
per channel per month, this is the common case, not an edge case.
|
||||||
|
|
||||||
|
Recommended fix, and it is a small one: make retention **`max(30 days, the N most recent videos)`**
|
||||||
|
via a `min_keep_videos` setting, default **5**.
|
||||||
|
|
||||||
|
- A channel uploading twice a year keeps its last 5 videos permanently visible instead of showing an
|
||||||
|
empty shelf.
|
||||||
|
- A channel uploading daily is unaffected — 30 days already exceeds 5 videos.
|
||||||
|
- Cost: `441 + (52 × 5) ≈ 700` episodes. Still nothing.
|
||||||
|
- It removes the flicker, because a video only leaves once 5 newer ones exist.
|
||||||
|
|
||||||
|
The alternative — per-channel `retention_days` overrides — is already in the schema but means hand-
|
||||||
|
tuning 119 channels, which nobody will do. **Also materialise a channel's directory only once it has
|
||||||
|
at least one in-window video**, so a genuinely dead channel does not create an empty shelf at all.
|
||||||
|
|
||||||
### A useful side effect: RSS nearly covers the whole window
|
### A useful side effect: RSS nearly covers the whole window
|
||||||
|
|
||||||
@@ -432,7 +467,7 @@ to be handling a few dozen calls a day rather than thousands.
|
|||||||
unreachable from the network by construction.
|
unreachable from the network by construction.
|
||||||
5. **Backfill stays resumable** (`channel.backfill_cursor`, commit per page of 50) even though a
|
5. **Backfill stays resumable** (`channel.backfill_cursor`, commit per page of 50) even though a
|
||||||
30-day backfill is now one or two pages. It costs a column and it is the difference between a
|
30-day backfill is now one or two pages. It costs a column and it is the difference between a
|
||||||
crashed first sync resuming and starting over across 20 channels.
|
crashed first sync resuming and starting over across 119 channels.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -541,9 +576,10 @@ terminal states matter and are not the same thing:
|
|||||||
`last_run_at`), including **`retention_days`** — now **30**, and governing both the aging-out sweep
|
`last_run_at`), including **`retention_days`** — now **30**, and governing both the aging-out sweep
|
||||||
and the backfill reach (§5). Drop `disk_cap_gb`, `write_subs`, `sub_langs`, `sponsorblock_mark`,
|
and the backfill reach (§5). Drop `disk_cap_gb`, `write_subs`, `sub_langs`, `sponsorblock_mark`,
|
||||||
`max_attempts`, `backfill_days` — download-era concepts. Add `youtube_api_key`, `subsync_max_new`
|
`max_attempts`, `backfill_days` — download-era concepts. Add `youtube_api_key`, `subsync_max_new`
|
||||||
(25, provisional — §4.4), `subsync_missing_threshold` (3), `proxy_base_url`
|
(**25**, derived in §4.4), `subsync_missing_threshold` (3), `proxy_base_url`
|
||||||
(`http://127.0.0.1:8099`), and `backfill_max_videos` (**300**) purely as a runaway guard on a channel
|
(`http://127.0.0.1:8099`), `backfill_max_videos` (**300**) purely as a runaway guard on a channel that
|
||||||
that turns out to upload 50 times a day.
|
turns out to upload 50 times a day, and **`min_keep_videos` (5)** — the fix for the 52 empty series
|
||||||
|
measured in §5.
|
||||||
|
|
||||||
**No migration from `subs.db`.** It holds 2 channels and 18 videos. Re-subscribe by hand and let
|
**No migration from `subs.db`.** It holds 2 channels and 18 videos. Re-subscribe by hand and let
|
||||||
the backfill do the rest; a migration script would be more code than the data is worth.
|
the backfill do the rest; a migration script would be more code than the data is worth.
|
||||||
@@ -658,6 +694,12 @@ Four keys, all in `setting`: `youtube_api_key` (new), `jellyfin_api_key`, `sessi
|
|||||||
The YouTube API key is restricted to the YouTube Data API v3 and has read-only reach over public
|
The YouTube API key is restricted to the YouTube Data API v3 and has read-only reach over public
|
||||||
data. Worst case on leak is quota exhaustion; rotation is a two-minute job in the console.
|
data. Worst case on leak is quota exhaustion; rotation is a two-minute job in the console.
|
||||||
|
|
||||||
|
**The Phase 0 key is treated as disposable and rotated once the service is built** (Tom's call, and
|
||||||
|
the right one — it has been pasted into a terminal and a chat transcript while debugging). That makes
|
||||||
|
one thing a requirement rather than a nicety: **rotating the key must not need a restart or a code
|
||||||
|
change.** Read it from the `setting` table at the point of use, not once at process start, so the
|
||||||
|
admin UI's "save" is the whole rotation procedure.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 12. Decommissioning `youtube-automate`
|
## 12. Decommissioning `youtube-automate`
|
||||||
@@ -693,16 +735,9 @@ is the thing that keeps playback working as YouTube changes.
|
|||||||
|
|
||||||
Each phase ends in something checkable. Do not start the next one until it does.
|
Each phase ends in something checkable. Do not start the next one until it does.
|
||||||
|
|
||||||
**Phase 0 — verify the assumptions the plan rests on.** Mostly **done, 2026-08-12** — see §16 for
|
**Phase 0 — verify the assumptions the plan rests on. ✅ COMPLETE 2026-08-12.** All three API checks
|
||||||
what was verified and what is still blocked. Remaining: paste the API key in and run
|
pass; `tools/verify_api.py` reproduces them in one command. Results and the numbers they changed are
|
||||||
|
in §16.
|
||||||
```sh
|
|
||||||
python3 /opt/ytstream/tools/verify_api.py --key AIza...
|
|
||||||
```
|
|
||||||
|
|
||||||
which performs all three API checks, costs ~5 quota units, prints the `subsync_max_new` value, and
|
|
||||||
exits non-zero if anything fails.
|
|
||||||
→ *Done when: `verify_api.py` reports ALL PASS and its numbers are written into §4.1 and §16.*
|
|
||||||
|
|
||||||
**Phase 1 — skeleton and lift.** Fork the tree, new package name, new DB path, new schema (§7),
|
**Phase 1 — skeleton and lift.** Fork the tree, new package name, new DB path, new schema (§7),
|
||||||
lifted modules and their tests passing. No new behaviour.
|
lifted modules and their tests passing. No new behaviour.
|
||||||
@@ -745,10 +780,9 @@ red — test these before trusting the deletion path, not after.*
|
|||||||
(§5). This supersedes the earlier "3 months or 300 videos" answer, which was about backfill depth;
|
(§5). This supersedes the earlier "3 months or 300 videos" answer, which was about backfill depth;
|
||||||
holding both would mean backfilling 90 days and deleting two thirds of it immediately. If the
|
holding both would mean backfilling 90 days and deleting two thirds of it immediately. If the
|
||||||
intent was "reach back 3 months and keep it", `retention_days = 90` is a one-line change and §5
|
intent was "reach back 3 months and keep it", `retention_days = 90` is a one-line change and §5
|
||||||
still holds. Measured consequence: **~400 episodes total**, not 20,000.
|
still holds. Measured consequence: **441 episodes total** across 119 channels, not 20,000.
|
||||||
3. **`subsync_max_new`:** still open — set from `pageInfo.totalResults` on the first
|
3. **`subsync_max_new` = 25** — closed by Phase 0. Measured 119 subscriptions, and
|
||||||
`subscriptions.list` call, which `tools/verify_api.py` prints. Provisional 25; rule of thumb
|
`max(10, ceil(119 × 0.2)) = 25` (§4.4).
|
||||||
`max(10, ceil(totalResults × 0.2))` (§4.4).
|
|
||||||
4. **Brother gets admin access:** yes, on the **existing shared password** — no `user` table, no
|
4. **Brother gets admin access:** yes, on the **existing shared password** — no `user` table, no
|
||||||
per-account credentials (§4.6). `web/auth.py` carries over untouched.
|
per-account credentials (§4.6). `web/auth.py` carries over untouched.
|
||||||
5. **Unsubscribe deletes the channel** rather than deactivating it, since re-subscribing is one API
|
5. **Unsubscribe deletes the channel** rather than deactivating it, since re-subscribing is one API
|
||||||
@@ -786,58 +820,70 @@ Things already paid for once. All of these are verified.
|
|||||||
enabled on the project (and carries an `activationUrl` naming the project number);
|
enabled on the project (and carries an `activationUrl` naming the project number);
|
||||||
`API_KEY_SERVICE_BLOCKED` means the API is enabled but *this key's* restrictions exclude it. They
|
`API_KEY_SERVICE_BLOCKED` means the API is enabled but *this key's* restrictions exclude it. They
|
||||||
are fixed on different console screens. `tools/verify_api.py` distinguishes them and prints the fix.
|
are fixed on different console screens. `tools/verify_api.py` distinguishes them and prints the fix.
|
||||||
|
- **Console changes take up to ~2 minutes to propagate.** Measured: a key restriction change was still
|
||||||
|
returning `API_KEY_SERVICE_BLOCKED` at 0 s, 30 s and 60 s, and succeeded at **90 s**. Do not
|
||||||
|
conclude a setting is wrong until a couple of minutes have passed — retry before re-editing.
|
||||||
|
- **`subscriptions.list` `pageInfo.totalResults` overcounts.** It reported **127** while full
|
||||||
|
pagination returned **119** distinct channels, and 2 of those 119 have feeds that HTTP-error —
|
||||||
|
terminated or private channels still counted as subscriptions. Never treat `totalResults` as the
|
||||||
|
list length; page to exhaustion and count what arrives. Anything sized off it (a progress bar, a
|
||||||
|
cap, an "is the list complete" check) will be wrong.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 16. Phase 0 results — 2026-08-12
|
## 16. Phase 0 results — COMPLETE, 2026-08-12
|
||||||
|
|
||||||
### Done
|
All three assumptions the plan rested on are verified against the live API. Nothing in the design had
|
||||||
|
to change; several numbers did.
|
||||||
|
|
||||||
- **Bare repo initialised.** `/disks/git-repos/ytstream.git` existed as an empty directory; it is now
|
### The three API checks
|
||||||
a real `--bare --shared=group` repo with `HEAD` → `refs/heads/main`, matching
|
|
||||||
`youtube-automate.git`. `/opt/ytstream` is a checkout with `origin` pointed at it.
|
|
||||||
- **The mirrored channel is resolved.** `@cflux1030` → `UCPcTWaLV8zwx4WP4QExHj4Q` ("C Flux", 55
|
|
||||||
subscribers). No `channels.list` call needed at build time.
|
|
||||||
- **There is no scraping fallback** — `/@cflux1030/channels` serves the Home tab with zero channel
|
|
||||||
ids (§15). The privacy checkbox is mandatory.
|
|
||||||
- **UULF feeds work for both existing channels**, and the filtering is substantial — 60–74% of what
|
|
||||||
these channels publish is Shorts or livestreams that UULF correctly excludes.
|
|
||||||
- **Upload rates measured**, which is what sized §5: ~18–23 long-form videos per channel per 30 days,
|
|
||||||
so ~400 episodes for a 20-channel library. Pitch Side's UULF feed spans **23.3 days in 15 entries**,
|
|
||||||
meaning RSS alone nearly covers the retention window.
|
|
||||||
- **`tools/verify_api.py` written and self-tested** (ISO-8601 duration parser unit-checked against six
|
|
||||||
cases; argparse and import verified). It runs all three API checks in one command.
|
|
||||||
|
|
||||||
### Key created, project 510818173753 — two setup steps deep, one to go
|
| # | Assumption | Result |
|
||||||
|
|
||||||
Progress on the key itself, all of it diagnosed from the error bodies:
|
|
||||||
|
|
||||||
1. **Key created** and reaching Google — it authenticates, so the key string is good.
|
|
||||||
2. **`SERVICE_DISABLED`** — YouTube Data API v3 was not enabled on project `510818173753`. Fixed by
|
|
||||||
enabling it.
|
|
||||||
3. **`API_KEY_SERVICE_BLOCKED`** ← *current state.* The API is now enabled, but the key's own API
|
|
||||||
restrictions exclude it, so every method returns "Requests to this API youtube method … are
|
|
||||||
blocked". Fix at
|
|
||||||
`https://console.cloud.google.com/apis/credentials?project=510818173753` → the key → API
|
|
||||||
restrictions → *Don't restrict key*, or tick YouTube Data API v3.
|
|
||||||
|
|
||||||
The ordering trap is worth remembering rather than rediscovering: the API must be enabled **before**
|
|
||||||
the key can be restricted to it, because it is absent from the picker until then. §4.1 step 4 now
|
|
||||||
says so.
|
|
||||||
|
|
||||||
Until the key answers, these three assumptions remain unverified:
|
|
||||||
|
|
||||||
| # | Assumption | Consequence if it fails |
|
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| 1 | `subscriptions.list?channelId=` returns his subscriptions | **Feature is impossible as designed.** Falls back to §4.2, all of which need him to do something. |
|
| 1 | `subscriptions.list?channelId=` returns his subscriptions with only an API key | **PASS.** 200, **119** channels across 3 pages. The feature is possible exactly as designed. |
|
||||||
| 2 | `playlistItems.list` accepts a `UULF…` playlist id | Take the documented `UU` fallback and filter Shorts/livestreams via `videos.list` duration + `liveStreamingDetails`. Costs one extra unit per 50. |
|
| 2 | `playlistItems.list` accepts an undocumented `UULF…` playlist id | **PASS.** 1,249 items with exact `videoPublishedAt`. No `UU` fallback needed (§3). |
|
||||||
| 3 | `videos.list` returns parseable `contentDetails.duration` | No `<durationinseconds>` in NFOs without a yt-dlp extraction per video. Degrades, does not block. |
|
| 3 | `videos.list` returns parseable `contentDetails.duration` | **PASS.** 50 ids in one call, 50 returned, 0 unparseable. |
|
||||||
|
|
||||||
Only #1 is a genuine blocker, and it also carries the number that sets `subsync_max_new`. Two
|
Reproduce with `python3 tools/verify_api.py --key <key>` (~5 quota units).
|
||||||
external prerequisites remain: **the key's API restriction** (above), and **his brother unchecking
|
|
||||||
"Keep all my subscriptions private"** — note that nothing so far has tested the second, because every
|
|
||||||
call has failed at the key before reaching YouTube's privacy check. Then one command closes Phase 0:
|
|
||||||
|
|
||||||
```sh
|
### The numbers that came out of it
|
||||||
python3 /opt/ytstream/tools/verify_api.py --key AIza...
|
|
||||||
```
|
- **119 subscriptions**, not the ~20 the plan assumed — and `pageInfo.totalResults` claims 127 (§15).
|
||||||
|
- **441 long-form videos across all 119 channels in the last 30 days.** That is the library size.
|
||||||
|
- **Median 1 video per channel per 30 days**, and **52 of 117 channels uploaded nothing** — the
|
||||||
|
distribution is severely skewed, which is what produced the `min_keep_videos` recommendation in §5.
|
||||||
|
- **UULF excludes 54%** of Pitch Side's catalogue (1,249 of 2,724), and **38 of 50** consecutive `UU`
|
||||||
|
uploads were ≤120 s. The Shorts filter earns its place.
|
||||||
|
- **All 119 feeds poll in 7.9 s** (8 threads). Hourly polling is free.
|
||||||
|
- `subsync_max_new` = **25**.
|
||||||
|
|
||||||
|
### Also done
|
||||||
|
|
||||||
|
- **Bare repo initialised.** `/disks/git-repos/ytstream.git` was an empty directory; it is now a real
|
||||||
|
`--bare --shared=group` repo with `HEAD` → `refs/heads/main`, matching `youtube-automate.git`.
|
||||||
|
`/opt/ytstream` is a checkout with `origin` pointed at it.
|
||||||
|
- **The mirrored channel is resolved:** `@cflux1030` → `UCPcTWaLV8zwx4WP4QExHj4Q` ("C Flux"). No
|
||||||
|
`channels.list` call needed at build time.
|
||||||
|
- **There is no scraping fallback** — `/@cflux1030/channels` serves the Home tab with zero channel ids
|
||||||
|
(§15), so the privacy setting is mandatory. It is now confirmed to be off, since the API returns his
|
||||||
|
list.
|
||||||
|
- **`tools/verify_api.py`** written, self-tested, and hardened against the two setup failures below.
|
||||||
|
|
||||||
|
### Getting the key working took three attempts, and the errors were misleading
|
||||||
|
|
||||||
|
Recorded because the next person to set up a Google Cloud project will hit the same wall. All three
|
||||||
|
states return HTTP 403 `forbidden`, and the distinguishing signal is `error.details[].reason`:
|
||||||
|
|
||||||
|
1. **`SERVICE_DISABLED`** — YouTube Data API v3 not enabled on project `510818173753`. Carries an
|
||||||
|
`activationUrl` naming the project.
|
||||||
|
2. **`API_KEY_SERVICE_BLOCKED`** — API enabled, but the *key's own* API restrictions exclude it. A
|
||||||
|
different console screen entirely. The ordering is the trap: YouTube Data API v3 is absent from the
|
||||||
|
key's restriction picker until the API is enabled on the project, so restricting first yields a key
|
||||||
|
that blocks the only API it exists for. §4.1 step 4 now says enable-then-restrict.
|
||||||
|
3. **Propagation delay** — after the restriction was fixed, calls still failed at 0 s, 30 s and 60 s,
|
||||||
|
and succeeded at **90 s**. Nothing was wrong at that point except impatience.
|
||||||
|
|
||||||
|
### What remains before Phase 1
|
||||||
|
|
||||||
|
Nothing. Phase 0 is closed. The one open decision is §14 item 2 — whether `min_keep_videos` (§5) is
|
||||||
|
wanted, which is a five-minute change either way and does not block starting Phase 1.
|
||||||
|
|||||||
Reference in New Issue
Block a user