61cc1672ec34bea6eeabfe724413e3597ddd0801
5
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Tom FluxandClaude Opus 5
|
61cc1672ec |
Admin routes for the subscription queue, and record the build in plan.md
Phase 4's acceptance criterion is that my brother approves the first import himself, which needs a UI, so /pending now carries source management, sync-now, and multi-select approve/reject. Driven over real HTTP rather than only through the templates: unauthenticated requests redirect to login, all four new routes reject a missing or forged CSRF token, and the live account rendered 117 checkboxes. Approving three at once added three channels, which is the point of the fix underneath. _form() collapses repeated fields to the last value, which is correct for every single-value field but silently wrong for a form of checkboxes all named `id` — it would have approved only the last box ticked. Added _form_list(), with the parsed body cached because rfile can only be read once and the approval path needs both views of it. The approval page is deliberately its own page rather than a section on the index: the first sync of the real account queued 119 channels, and that does not belong inline under the channel table. Source errors are shown in full rather than truncated, because the useful ones say exactly what to do — "subscriptions are private, uncheck Keep all my subscriptions private" — and hiding that behind a log file defeats the purpose of surfacing it. plan.md §13 now reflects what is actually built rather than what was intended, and a new §17 records the three bugs the build turned up, including which of them a test caught and which two needed real data. 337 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tom FluxandClaude Opus 5
|
f640c064c6 |
Phase 0 complete: all three API assumptions verified, and 119 subscriptions
The key works. All three assumptions the plan rested on hold, so nothing in the
design had to change — but the measurements moved several numbers by a lot, and
one of them exposed a design problem the plan did not have an answer for.
Verified against the live API:
subscriptions.list?channelId= 200, 119 channels over 3 pages. An API key is
sufficient; no OAuth, no consent screen.
playlistItems.list on UULF Accepts the undocumented long-form playlist id
and returns exact videoPublishedAt. 1,249 items
against 2,724 in UU, so it excludes 54% of the
catalogue. The UU-plus-duration-filter fallback
is not needed.
videos.list durations 50 ids in one call, 50 back, 0 unparseable. Of
those 50 consecutive UU uploads, 38 were <=120s,
which is the Shorts filter earning its keep.
The account has 119 subscriptions, not the ~20 this plan assumed, so rather than
extrapolate I probed all 119 UULF feeds: 441 long-form videos in the last 30
days, median 1 per channel, and all 119 feeds polled in 7.9s on 8 threads. The
library is therefore smaller than the previous 20-channel estimate despite six
times the channels, because the distribution is severely skewed. Quota, scan cost
and poll cost are all now measured rather than guessed, and none of them is a
constraint: ~130 units to build, ~75/day steady state, under 1% of the budget.
The problem that fell out: 52 of 117 channels uploaded nothing long-form in 30
days, so they would appear in Jellyfin as empty series, and a channel uploading
every six weeks would flicker in and out as its one video crossed the retention
line. With a median of 1 upload a month that is the common case, not an edge
case. Recommended fix in §5 — retention becomes max(30 days, N most recent
videos) via min_keep_videos, default 5, which costs ~260 extra episodes and
removes the flicker entirely because a video only leaves once 5 newer ones exist.
Flagged as a decision rather than applied unilaterally.
Also recorded, because it cost three attempts to get the key working and every
failure was an indistinguishable 403 forbidden: SERVICE_DISABLED means the API is
not enabled on the project, API_KEY_SERVICE_BLOCKED means the key's own
restrictions exclude it, they are fixed on different console screens, and the
signal is in error.details[].reason rather than error.errors[].reason. The
ordering matters — the API must be enabled before the key can be restricted to
it, because it is absent from the picker until then. And after the fix, calls
still failed at 0s, 30s and 60s and succeeded at 90s; nothing was wrong except
impatience.
One more trap worth having in writing: pageInfo.totalResults reported 127 while
pagination returned 119 distinct channels, and 2 of those 119 have feeds that
HTTP-error. Terminated and private channels are still counted as subscriptions,
so anything sized off totalResults will be wrong.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Tom FluxandClaude Opus 5
|
f3d70f1c87 |
Tell the two API-key setup failures apart in verify_api.py
Ran Phase 0 against a live key and hit both of the ways a fresh Google Cloud
project can be wrong, in sequence. Google reports them as the same 403
`forbidden`, so the first version of this script printed reason='forbidden'
three times and buried the one sentence that said what to do.
The distinguishing signal is in error.details[].reason, not
error.errors[].reason:
SERVICE_DISABLED YouTube Data API v3 is not enabled on the project.
Carries an activationUrl naming the project number.
API_KEY_SERVICE_BLOCKED The API is enabled, but this key's API restrictions
exclude it.
They are fixed on different console screens, so they are now separate exception
types with separate advice, and a one-call preflight reports either before the
three real checks run and fail identically.
The ordering between them is a trap worth writing down: YouTube Data API v3 does
not appear in a key's API-restriction picker until the API is enabled on the
project, so creating the key and restricting it first yields a key that blocks
the only API it exists for. That is precisely what happened here. §4.1 step 4
now says to enable before restricting.
Nothing has yet reached YouTube's own privacy check, so whether the brother's
subscriptions are readable is still untested — every call so far failed at the
key.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Tom FluxandClaude Opus 5
|
339232c7f9 |
Phase 0: rolling 30-day window, delete on unsubscribe, shared password
Phase 0 is done apart from one external dependency, and what it measured changed the shape of the plan more than the three answers it was meant to confirm. Measured the actual upload rates from the live UULF feeds rather than guessing from catalogue size, and the numbers make the whole scale section boring. Pitch Side publishes 0.60 long-form videos a day, The Pyramid Podcast 0.77 — so a 30-day window is about 20 episodes per channel and roughly 400 across a 20-channel library, not the 20,000 the previous draft braced for. The UULF feed is also doing more work than expected: it excludes 60-74% of what these channels publish, all of it Shorts and livestreams nobody wants as Jellyfin episodes. A side effect worth recording: Pitch Side's 15-entry UULF feed spans 23.3 days, because the feed holds 15 long-form items rather than 15 uploads. Free RSS therefore covers most of a 30-day retention window on its own, which narrows what the API is for. It still earns its place — RSS carries no duration, and subscription reading has no alternative at all — but the honest version is that it now buys correctness far more than it buys safety, and the plan says so instead of keeping the more flattering 4-hours-of-yt-dlp argument. Three decisions folded in. Retention is a rolling 30 days, superseding the earlier "3 months or 300 videos". Those answered different questions — backfill depth versus retention — and holding both would mean backfilling 90 days and deleting two thirds of it on the next sweep. One number now governs both ends, so the library cannot grow. This resurrects reap.py, which the previous draft deleted, and makes tombstones load-bearing in two ways: without them the next poll re-materialises everything the sweep just deleted, and episode ordinals for a given day would shift as videos disappear. aged_out is deliberately never revivable, or raising retention_days would resurrect months of episodes into Jellyfin as new. Unsubscribing now deletes the channel rather than deactivating it, which is reasonable when rebuilding costs one API page and ~20 files. The consequence is that the removal-detection rules stop being precautionary and become the only thing standing between a transient 403 and a wiped library, so the plan now says that explicitly next to them. Auth reverts to the single shared admin_password_hash, matching every other service on susan. web/auth.py carries over untouched. What that gives up — independent revocation, and knowing who approved what — is named once and then dropped. Added tools/verify_api.py so the remaining Phase 0 work is one command. It runs all three API checks, paginates the subscription list, prints the subsync_max_new value derived from totalResults, distinguishes 403 subscriptionForbidden from every other failure with the fix in the message, and exits non-zero so it can be gated on. The ISO-8601 duration parser is unit-checked. Blocked on: there is no Google API key anywhere on this machine (searched /opt/*, /home/susan, the settings table and the config trees for AIza-shaped strings). Creating one needs a browser and a Google account. That plus the brother unchecking "Keep all my subscriptions private" are the two external prerequisites, and §16 records exactly what each unverified assumption costs if it turns out false. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tom FluxandClaude Opus 5
|
8cca4fe1e2 |
Plan ytstream as a service, and mirror a YouTube account's subscriptions
The streaming proof of concept works — FINDINGS.md records the measurements —
but it runs by hand out of a home directory. This is the plan for making it a
real service, plus the design for the new requirement: my brother subscribes to
a channel on YouTube and it appears in Jellyfin without anyone touching an
admin page.
Three things drove the shape of the plan.
It is a fork of youtube-automate, not a green-field build. That project is 3,543
lines and most of it — RSS discovery, episode numbering, NFO schema, scrypt auth,
the admin UI — has nothing to do with downloading and was verified on this
machine on 2026-08-11. Only download.py and reap.py actually die. Re-deriving the
rest would mean re-earning knowledge we already paid for.
Getting a Google API key for the subscription feature turns out to pay for
itself twice over, so the plan leans on it much harder than the feature needs.
subscriptions.list, playlistItems.list and videos.list are 1 unit each against
10,000/day, which means the whole metadata path can move off yt-dlp: exact upload
dates from contentDetails.videoPublishedAt instead of approximate_date (measured
wrong by up to 2 days, and episode numbers are derived from it), durations
without an extraction per video, and no residential-IP request storm when a
channel is added. yt-dlp is then only ever invoked by the proxy, for one video,
because a human pressed play. That is a much better boundary than the one we
have.
The brother's channel is the sole source of truth. An earlier draft modelled
subscriptions as multi-source claims so a manually-added channel could not be
deleted by someone else's list; with one source that is dead weight, and
speculative generality in the code path that deletes things is the wrong place to
spend it. What survives instead is paranoia about removals, which now matter more
rather than less: a 403, a timeout or an empty response is never read as an
unsubscribe, a channel must be absent from three consecutive healthy syncs, and
even then the tree stays on disk and the channel merely goes inactive.
Verified while writing this, rather than assumed:
- subscriptions.list accepts a channelId filter and is not documented as
needing an authorized request, so an API key is enough. It returns 403 —
not an empty list — when subscriptions are private, which is what makes the
removal safeguards able to tell "he made them private again" apart from "he
unsubscribed from everything".
- There is no scraping fallback. /@cflux1030/channels returns 200 but silently
serves the Home tab: four tabs, zero channel ids in ytInitialData. YouTube
retired the public subscriptions tab, so the privacy checkbox is mandatory
rather than merely convenient.
- @cflux1030 resolves to UCPcTWaLV8zwx4WP4QExHj4Q, so no channels.list call is
needed at build time.
- OAuth as a fallback carries a trap worth writing down: a consent screen in
"Testing" status issues refresh tokens that expire in 7 days, and
youtube.readonly is not one of the exempt basic scopes.
Catalogue depth is bounded at 3 months or 300 videos per channel, whichever
comes first. The plan says plainly that this bounds the initial build and not the
library, which grows forward indefinitely — roughly 15,000 episodes a year at
20 channels — so the Jellyfin scan-cost measurement in Phase 3 is what tells us
when that becomes a problem. Deliberately not solved now.
One number is still open: subsync_max_new depends on how many channels he
actually follows, which only the day-one API call can tell us. Phase 0 closes it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|