04721a78eac9a10efa4fdd30f2006b4b1e459766
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Tom FluxandClaude Opus 5
|
d3bf8d6f19 |
Deploy it, and fix the six things installation found
Both units are installed and running, 10 of 119 channels approved, 251 episodes live in Jellyfin with verified DirectPlay. 345 tests. Six problems surfaced that no test could have, and two of them were mine in the deploy scripts. deploy.sh had a circular dependency with bootstrap.sh: deploy started the units and told the operator to run bootstrap, but bootstrap refused to run until the state directory existed, which only deploy creates. The units started against a non-existent venv, failed 203/EXEC and restart-looped 17 and 21 times. deploy.sh now creates the directory, calls bootstrap itself through runuser so the venv is not left root-owned, and refuses to start units when the venv is still missing. Deno was absent, and `doctor` is the only reason we know. It is mandatory rather than nice-to-have — without a JS runtime yt-dlp cannot solve the n challenge, which youtube-automate measured on this machine as 22 formats instead of 29 plus throttling. Nothing else would have complained; playback would just have quietly degraded. bootstrap.sh now installs it and asserts yt-dlp reports it. Episodes had no synopsis at all, because materialise passed plot=None while both sources hand us descriptions for free. Now plumbed through from RSS (media:group/media:description) and from videos.list, which carries snippet.description in the call already being made for durations — so the ~40% of episodes older than RSS reaches get one too. That needed a schema v2 migration; v1 was left exactly as shipped so a fresh install and a migrated one are identical, and a test asserts it. `materialise --all` — the documented recovery from a Jellyfin metadata wipe — was itself creating duplicates. Episode numbers were re-derived each run, and next_episode() excludes the row being numbered, so re-materialising a day's videos in a different order renumbered them and orphaned the old files. One run left 102 orphaned NFOs against 251 episodes. An episode number is now permanent once assigned, and a video whose rel_path changes has its old files removed first. Running it twice is now a no-op. Two Jellyfin behaviours worth having in writing. It ignores <runtime> and <durationinseconds> for episodes while reading the rest of the NFO happily, so a .strm shows no duration until first played — not fixable without probing, which is the one thing this design exists to avoid. And a plain /Library/Refresh does not reliably re-read a rewritten NFO: after rewriting all 251, fifty kept their old empty metadata. The fix is metadataRefreshMode=Default with replaceAllMetadata=false, which took plots from 201 to 251 while the proxy served zero requests. §5's prohibition on replaceAllMetadata=true still stands — that one probes. Exposed as `ytstream refresh-metadata` and run automatically after `materialise --all`. The measurement §5 has been waiting for: a full Jellyfin scan of 251 .strm files took ~119 s, about 8 minutes per 1,000 episodes, and made zero media probes. That last number is the fact the whole design rests on, now confirmed at scale on the real library rather than on seven PoC files. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tom FluxandClaude Opus 5
|
61cc1672ec |
Admin routes for the subscription queue, and record the build in plan.md
Phase 4's acceptance criterion is that my brother approves the first import himself, which needs a UI, so /pending now carries source management, sync-now, and multi-select approve/reject. Driven over real HTTP rather than only through the templates: unauthenticated requests redirect to login, all four new routes reject a missing or forged CSRF token, and the live account rendered 117 checkboxes. Approving three at once added three channels, which is the point of the fix underneath. _form() collapses repeated fields to the last value, which is correct for every single-value field but silently wrong for a form of checkboxes all named `id` — it would have approved only the last box ticked. Added _form_list(), with the parsed body cached because rfile can only be read once and the approval path needs both views of it. The approval page is deliberately its own page rather than a section on the index: the first sync of the real account queued 119 channels, and that does not belong inline under the channel table. Source errors are shown in full rather than truncated, because the useful ones say exactly what to do — "subscriptions are private, uncheck Keep all my subscriptions private" — and hiding that behind a log file defeats the purpose of surfacing it. plan.md §13 now reflects what is actually built rather than what was intended, and a new §17 records the three bugs the build turned up, including which of them a test caught and which two needed real data. 337 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tom FluxandClaude Opus 5
|
155f05773d |
Build ytstream: catalogue, retention, subscription mirror, proxy
Phases 1-4 of plan.md §13. Forked from youtube-automate as planned rather than
written from scratch: naming, NFO, auth, the admin UI, settings and the DB layer
came across largely unchanged, download.py is gone, and the pieces that only make
sense for a streaming library are new.
api.py YouTube Data API v3 client. The whole metadata path.
strm.py Materialising: a .strm, an .nfo and a thumbnail. Replaces the
330-line download.py, because the job is writing a URL to a file.
subsync.py The subscription mirror, most of which is refusals.
reap.py Retention, rewritten around the 30-day window and min_keep_videos.
discovery.py RSS polling plus an API-backed, resumable, bounded backfill.
proxy/ The verified PoC, moved in with a systemd unit.
330 tests, all passing, no network and no yt-dlp in any of them. The suite leans
towards the failure paths, because that is where this design can actually hurt
someone: a 403 that looks like an unsubscribe, a video that ages out and comes
back, a title that never arrives. tests/test_proxy.py replaces the two standalone
scripts under proxy/ and now drives the real make_handler(mgr, ...) rather than the
PoC's make_handler(path, done), so routing and video-id validation are covered too.
Ran it end to end against the live API and it found three real bugs.
The first was mine and the tests caught it: strm.remove pruned empty directories
up to the media root, so a channel directory whose tvshow.nfo happened to be
missing would be deleted along with the season. It only looked safe because
tvshow.nfo normally stops the walk. The prune boundary is now the channel
directory explicitly.
The other two only showed up against real data, and they compounded. The backfill
inserted rows with no title and left the RSS poll to fill them in — but RSS returns
15 entries, which for Pitch Side spans 23 days against a 30-day window, so five of
twenty episodes were named after their video ids. Worse, strm.materialise wrote
that fallback back to the database as the title, which made the row look titled and
permanently disabled the repair path. Both fixed: playlistItems.list now requests
snippet as well as contentDetails, which costs the same single quota unit and
carries the title alongside the exact publish date, and the fallback is used for the
filename without being persisted. A title that does arrive late now also removes the
badly-named files and re-queues, so the episode is rewritten rather than keeping its
video-id name forever. Verified against the live API: all twenty Pitch Side episodes
now carry real titles.
Measured on the real account: 119 subscriptions queued for approval and none added
on the first sync, then a two-channel run backfilled and materialised 26 episodes in
under seven seconds.
Two deliberate departures from plan.md, both recorded there:
min_keep_videos defaults to 5 rather than being left as an open question. Without
it 52 of 117 measured channels are empty Jellyfin series that flicker in and out as
their single video crosses the retention line, and the plan already recommended it.
The Jellyfin refresh is a bare /Library/Refresh with a comment explaining why it
must stay that way. A normal scan makes zero media probes; FullRefresh does probe,
and at 400 episodes that is 400 cold starts.
Not yet done: no systemd units are installed (needs root — deploy/deploy.sh), the
admin UI has no routes for sources or the approval queue yet, and nothing has been
pointed at the real media root.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Tom FluxandClaude Opus 5
|
f640c064c6 |
Phase 0 complete: all three API assumptions verified, and 119 subscriptions
The key works. All three assumptions the plan rested on hold, so nothing in the
design had to change — but the measurements moved several numbers by a lot, and
one of them exposed a design problem the plan did not have an answer for.
Verified against the live API:
subscriptions.list?channelId= 200, 119 channels over 3 pages. An API key is
sufficient; no OAuth, no consent screen.
playlistItems.list on UULF Accepts the undocumented long-form playlist id
and returns exact videoPublishedAt. 1,249 items
against 2,724 in UU, so it excludes 54% of the
catalogue. The UU-plus-duration-filter fallback
is not needed.
videos.list durations 50 ids in one call, 50 back, 0 unparseable. Of
those 50 consecutive UU uploads, 38 were <=120s,
which is the Shorts filter earning its keep.
The account has 119 subscriptions, not the ~20 this plan assumed, so rather than
extrapolate I probed all 119 UULF feeds: 441 long-form videos in the last 30
days, median 1 per channel, and all 119 feeds polled in 7.9s on 8 threads. The
library is therefore smaller than the previous 20-channel estimate despite six
times the channels, because the distribution is severely skewed. Quota, scan cost
and poll cost are all now measured rather than guessed, and none of them is a
constraint: ~130 units to build, ~75/day steady state, under 1% of the budget.
The problem that fell out: 52 of 117 channels uploaded nothing long-form in 30
days, so they would appear in Jellyfin as empty series, and a channel uploading
every six weeks would flicker in and out as its one video crossed the retention
line. With a median of 1 upload a month that is the common case, not an edge
case. Recommended fix in §5 — retention becomes max(30 days, N most recent
videos) via min_keep_videos, default 5, which costs ~260 extra episodes and
removes the flicker entirely because a video only leaves once 5 newer ones exist.
Flagged as a decision rather than applied unilaterally.
Also recorded, because it cost three attempts to get the key working and every
failure was an indistinguishable 403 forbidden: SERVICE_DISABLED means the API is
not enabled on the project, API_KEY_SERVICE_BLOCKED means the key's own
restrictions exclude it, they are fixed on different console screens, and the
signal is in error.details[].reason rather than error.errors[].reason. The
ordering matters — the API must be enabled before the key can be restricted to
it, because it is absent from the picker until then. And after the fix, calls
still failed at 0s, 30s and 60s and succeeded at 90s; nothing was wrong except
impatience.
One more trap worth having in writing: pageInfo.totalResults reported 127 while
pagination returned 119 distinct channels, and 2 of those 119 have feeds that
HTTP-error. Terminated and private channels are still counted as subscriptions,
so anything sized off totalResults will be wrong.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Tom FluxandClaude Opus 5
|
f3d70f1c87 |
Tell the two API-key setup failures apart in verify_api.py
Ran Phase 0 against a live key and hit both of the ways a fresh Google Cloud
project can be wrong, in sequence. Google reports them as the same 403
`forbidden`, so the first version of this script printed reason='forbidden'
three times and buried the one sentence that said what to do.
The distinguishing signal is in error.details[].reason, not
error.errors[].reason:
SERVICE_DISABLED YouTube Data API v3 is not enabled on the project.
Carries an activationUrl naming the project number.
API_KEY_SERVICE_BLOCKED The API is enabled, but this key's API restrictions
exclude it.
They are fixed on different console screens, so they are now separate exception
types with separate advice, and a one-call preflight reports either before the
three real checks run and fail identically.
The ordering between them is a trap worth writing down: YouTube Data API v3 does
not appear in a key's API-restriction picker until the API is enabled on the
project, so creating the key and restricting it first yields a key that blocks
the only API it exists for. That is precisely what happened here. §4.1 step 4
now says to enable before restricting.
Nothing has yet reached YouTube's own privacy check, so whether the brother's
subscriptions are readable is still untested — every call so far failed at the
key.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Tom FluxandClaude Opus 5
|
339232c7f9 |
Phase 0: rolling 30-day window, delete on unsubscribe, shared password
Phase 0 is done apart from one external dependency, and what it measured changed the shape of the plan more than the three answers it was meant to confirm. Measured the actual upload rates from the live UULF feeds rather than guessing from catalogue size, and the numbers make the whole scale section boring. Pitch Side publishes 0.60 long-form videos a day, The Pyramid Podcast 0.77 — so a 30-day window is about 20 episodes per channel and roughly 400 across a 20-channel library, not the 20,000 the previous draft braced for. The UULF feed is also doing more work than expected: it excludes 60-74% of what these channels publish, all of it Shorts and livestreams nobody wants as Jellyfin episodes. A side effect worth recording: Pitch Side's 15-entry UULF feed spans 23.3 days, because the feed holds 15 long-form items rather than 15 uploads. Free RSS therefore covers most of a 30-day retention window on its own, which narrows what the API is for. It still earns its place — RSS carries no duration, and subscription reading has no alternative at all — but the honest version is that it now buys correctness far more than it buys safety, and the plan says so instead of keeping the more flattering 4-hours-of-yt-dlp argument. Three decisions folded in. Retention is a rolling 30 days, superseding the earlier "3 months or 300 videos". Those answered different questions — backfill depth versus retention — and holding both would mean backfilling 90 days and deleting two thirds of it on the next sweep. One number now governs both ends, so the library cannot grow. This resurrects reap.py, which the previous draft deleted, and makes tombstones load-bearing in two ways: without them the next poll re-materialises everything the sweep just deleted, and episode ordinals for a given day would shift as videos disappear. aged_out is deliberately never revivable, or raising retention_days would resurrect months of episodes into Jellyfin as new. Unsubscribing now deletes the channel rather than deactivating it, which is reasonable when rebuilding costs one API page and ~20 files. The consequence is that the removal-detection rules stop being precautionary and become the only thing standing between a transient 403 and a wiped library, so the plan now says that explicitly next to them. Auth reverts to the single shared admin_password_hash, matching every other service on susan. web/auth.py carries over untouched. What that gives up — independent revocation, and knowing who approved what — is named once and then dropped. Added tools/verify_api.py so the remaining Phase 0 work is one command. It runs all three API checks, paginates the subscription list, prints the subsync_max_new value derived from totalResults, distinguishes 403 subscriptionForbidden from every other failure with the fix in the message, and exits non-zero so it can be gated on. The ISO-8601 duration parser is unit-checked. Blocked on: there is no Google API key anywhere on this machine (searched /opt/*, /home/susan, the settings table and the config trees for AIza-shaped strings). Creating one needs a browser and a Google account. That plus the brother unchecking "Keep all my subscriptions private" are the two external prerequisites, and §16 records exactly what each unverified assumption costs if it turns out false. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tom FluxandClaude Opus 5
|
8cca4fe1e2 |
Plan ytstream as a service, and mirror a YouTube account's subscriptions
The streaming proof of concept works — FINDINGS.md records the measurements —
but it runs by hand out of a home directory. This is the plan for making it a
real service, plus the design for the new requirement: my brother subscribes to
a channel on YouTube and it appears in Jellyfin without anyone touching an
admin page.
Three things drove the shape of the plan.
It is a fork of youtube-automate, not a green-field build. That project is 3,543
lines and most of it — RSS discovery, episode numbering, NFO schema, scrypt auth,
the admin UI — has nothing to do with downloading and was verified on this
machine on 2026-08-11. Only download.py and reap.py actually die. Re-deriving the
rest would mean re-earning knowledge we already paid for.
Getting a Google API key for the subscription feature turns out to pay for
itself twice over, so the plan leans on it much harder than the feature needs.
subscriptions.list, playlistItems.list and videos.list are 1 unit each against
10,000/day, which means the whole metadata path can move off yt-dlp: exact upload
dates from contentDetails.videoPublishedAt instead of approximate_date (measured
wrong by up to 2 days, and episode numbers are derived from it), durations
without an extraction per video, and no residential-IP request storm when a
channel is added. yt-dlp is then only ever invoked by the proxy, for one video,
because a human pressed play. That is a much better boundary than the one we
have.
The brother's channel is the sole source of truth. An earlier draft modelled
subscriptions as multi-source claims so a manually-added channel could not be
deleted by someone else's list; with one source that is dead weight, and
speculative generality in the code path that deletes things is the wrong place to
spend it. What survives instead is paranoia about removals, which now matter more
rather than less: a 403, a timeout or an empty response is never read as an
unsubscribe, a channel must be absent from three consecutive healthy syncs, and
even then the tree stays on disk and the channel merely goes inactive.
Verified while writing this, rather than assumed:
- subscriptions.list accepts a channelId filter and is not documented as
needing an authorized request, so an API key is enough. It returns 403 —
not an empty list — when subscriptions are private, which is what makes the
removal safeguards able to tell "he made them private again" apart from "he
unsubscribed from everything".
- There is no scraping fallback. /@cflux1030/channels returns 200 but silently
serves the Home tab: four tabs, zero channel ids in ytInitialData. YouTube
retired the public subscriptions tab, so the privacy checkbox is mandatory
rather than merely convenient.
- @cflux1030 resolves to UCPcTWaLV8zwx4WP4QExHj4Q, so no channels.list call is
needed at build time.
- OAuth as a fallback carries a trap worth writing down: a consent screen in
"Testing" status issues refresh tokens that expire in 7 days, and
youtube.readonly is not one of the exempt basic scopes.
Catalogue depth is bounded at 3 months or 300 videos per channel, whichever
comes first. The plan says plainly that this bounds the initial build and not the
library, which grows forward indefinitely — roughly 15,000 episodes a year at
20 channels — so the Jellyfin scan-cost measurement in Phase 3 is what tells us
when that becomes a problem. Deliberately not solved now.
One number is still open: subsync_max_new depends on how many channels he
actually follows, which only the day-one API call can tell us. Phase 0 closes it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|