339232c7f9652967465fa6f94668f0c57e468ce8
2
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Tom FluxandClaude Opus 5
|
339232c7f9 |
Phase 0: rolling 30-day window, delete on unsubscribe, shared password
Phase 0 is done apart from one external dependency, and what it measured changed the shape of the plan more than the three answers it was meant to confirm. Measured the actual upload rates from the live UULF feeds rather than guessing from catalogue size, and the numbers make the whole scale section boring. Pitch Side publishes 0.60 long-form videos a day, The Pyramid Podcast 0.77 — so a 30-day window is about 20 episodes per channel and roughly 400 across a 20-channel library, not the 20,000 the previous draft braced for. The UULF feed is also doing more work than expected: it excludes 60-74% of what these channels publish, all of it Shorts and livestreams nobody wants as Jellyfin episodes. A side effect worth recording: Pitch Side's 15-entry UULF feed spans 23.3 days, because the feed holds 15 long-form items rather than 15 uploads. Free RSS therefore covers most of a 30-day retention window on its own, which narrows what the API is for. It still earns its place — RSS carries no duration, and subscription reading has no alternative at all — but the honest version is that it now buys correctness far more than it buys safety, and the plan says so instead of keeping the more flattering 4-hours-of-yt-dlp argument. Three decisions folded in. Retention is a rolling 30 days, superseding the earlier "3 months or 300 videos". Those answered different questions — backfill depth versus retention — and holding both would mean backfilling 90 days and deleting two thirds of it on the next sweep. One number now governs both ends, so the library cannot grow. This resurrects reap.py, which the previous draft deleted, and makes tombstones load-bearing in two ways: without them the next poll re-materialises everything the sweep just deleted, and episode ordinals for a given day would shift as videos disappear. aged_out is deliberately never revivable, or raising retention_days would resurrect months of episodes into Jellyfin as new. Unsubscribing now deletes the channel rather than deactivating it, which is reasonable when rebuilding costs one API page and ~20 files. The consequence is that the removal-detection rules stop being precautionary and become the only thing standing between a transient 403 and a wiped library, so the plan now says that explicitly next to them. Auth reverts to the single shared admin_password_hash, matching every other service on susan. web/auth.py carries over untouched. What that gives up — independent revocation, and knowing who approved what — is named once and then dropped. Added tools/verify_api.py so the remaining Phase 0 work is one command. It runs all three API checks, paginates the subscription list, prints the subsync_max_new value derived from totalResults, distinguishes 403 subscriptionForbidden from every other failure with the fix in the message, and exits non-zero so it can be gated on. The ISO-8601 duration parser is unit-checked. Blocked on: there is no Google API key anywhere on this machine (searched /opt/*, /home/susan, the settings table and the config trees for AIza-shaped strings). Creating one needs a browser and a Google account. That plus the brother unchecking "Keep all my subscriptions private" are the two external prerequisites, and §16 records exactly what each unverified assumption costs if it turns out false. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tom FluxandClaude Opus 5
|
8cca4fe1e2 |
Plan ytstream as a service, and mirror a YouTube account's subscriptions
The streaming proof of concept works — FINDINGS.md records the measurements —
but it runs by hand out of a home directory. This is the plan for making it a
real service, plus the design for the new requirement: my brother subscribes to
a channel on YouTube and it appears in Jellyfin without anyone touching an
admin page.
Three things drove the shape of the plan.
It is a fork of youtube-automate, not a green-field build. That project is 3,543
lines and most of it — RSS discovery, episode numbering, NFO schema, scrypt auth,
the admin UI — has nothing to do with downloading and was verified on this
machine on 2026-08-11. Only download.py and reap.py actually die. Re-deriving the
rest would mean re-earning knowledge we already paid for.
Getting a Google API key for the subscription feature turns out to pay for
itself twice over, so the plan leans on it much harder than the feature needs.
subscriptions.list, playlistItems.list and videos.list are 1 unit each against
10,000/day, which means the whole metadata path can move off yt-dlp: exact upload
dates from contentDetails.videoPublishedAt instead of approximate_date (measured
wrong by up to 2 days, and episode numbers are derived from it), durations
without an extraction per video, and no residential-IP request storm when a
channel is added. yt-dlp is then only ever invoked by the proxy, for one video,
because a human pressed play. That is a much better boundary than the one we
have.
The brother's channel is the sole source of truth. An earlier draft modelled
subscriptions as multi-source claims so a manually-added channel could not be
deleted by someone else's list; with one source that is dead weight, and
speculative generality in the code path that deletes things is the wrong place to
spend it. What survives instead is paranoia about removals, which now matter more
rather than less: a 403, a timeout or an empty response is never read as an
unsubscribe, a channel must be absent from three consecutive healthy syncs, and
even then the tree stays on disk and the channel merely goes inactive.
Verified while writing this, rather than assumed:
- subscriptions.list accepts a channelId filter and is not documented as
needing an authorized request, so an API key is enough. It returns 403 —
not an empty list — when subscriptions are private, which is what makes the
removal safeguards able to tell "he made them private again" apart from "he
unsubscribed from everything".
- There is no scraping fallback. /@cflux1030/channels returns 200 but silently
serves the Home tab: four tabs, zero channel ids in ytInitialData. YouTube
retired the public subscriptions tab, so the privacy checkbox is mandatory
rather than merely convenient.
- @cflux1030 resolves to UCPcTWaLV8zwx4WP4QExHj4Q, so no channels.list call is
needed at build time.
- OAuth as a fallback carries a trap worth writing down: a consent screen in
"Testing" status issues refresh tokens that expire in 7 days, and
youtube.readonly is not one of the exempt basic scopes.
Catalogue depth is bounded at 3 months or 300 videos per channel, whichever
comes first. The plan says plainly that this bounds the initial build and not the
library, which grows forward indefinitely — roughly 15,000 episodes a year at
20 channels — so the Jellyfin scan-cost measurement in Phase 3 is what tells us
when that becomes a problem. Deliberately not solved now.
One number is still open: subsync_max_new depends on how many channels he
actually follows, which only the day-one API call can tell us. Phase 0 closes it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|