04721a78eac9a10efa4fdd30f2006b4b1e459766
14
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Claude
|
04721a78ea |
Measure parallel fetching: it does not help
The ceiling is the internet connection at ~8 MB/s. Parallel chunks buy 10-30%, concurrency across different videos buys nothing, and --concurrent-fragments is inapplicable because these formats carry no fragments at all. yt-dlp is already doing the thing that matters: the formats advertise http_chunk_size=10485760 and that chunking is worth 13x, because a single long range request gets throttled to 0.60 MB/s. Also tested option 3 while here: NFO <streamdetails> is ignored for episodes -- RunTimeTicks stays null and MediaStreams empty -- so Jellyfin cannot be talked out of transcoding that way either. The useful number: download is 8 MB/s against a 0.39 MB/s playback bitrate, 20x headroom. Streaming while downloading was never bandwidth bound; it failed purely on Jellyfin's probe decision. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Claude
|
22d8828080 |
Never serve a partial file: it makes Jellyfin transcode
The 12s first-byte grace was the wrong trade and a real play found it within the hour. A 2-hour upload took 3 minutes to start, played 6 seconds, and stalled. Jellyfin had run ffmpeg with -probesize 1G against the growing stream and then transcoded to HLS with libx264. The cause is the container. A fragmented MP4 with empty_moov has no duration in its header, so the only way to get one is to sum every fragment -- probing a growing file reads all of it. Jellyfin cannot establish duration, codec or bitrate, so it abandons direct play and transcodes a stream it also cannot seek. It was targeting 4.83 Mbps against a source measured at 3.29: re-encoding a stream that already fit, because it could not measure it. The same video once complete reports SupportsDirectPlay with the exact runtime and bitrate. So FIRST_BYTE_GRACE defaults to infinite again, with --wait-timeout raised to 600s for a 2-hour upload. A cold long video is slow to start, which is accepted: the fetch outlives the request so a retry is instant, and a retryable stall beats a transcode that wastes a gigabyte and cannot work. Both failure modes are recorded at the constant in the order measured so the 12s cap is not reintroduced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Claude
|
c12837cbb9 |
Record the completed decommission
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Claude
|
28d0e83130 |
Fix the Jellyfin rename call: it takes a name, not an id
POST /Library/VirtualFolders/Name is the odd one out in that controller -- most of /Library/VirtualFolders/* takes an id, and passing one here returns a bare "HTTP 400: Error processing request." that says nothing about why. Verified against 10.11.4: name -> 204. Also records that renaming re-ids the library, because Jellyfin derives the ItemId from the name. ytstream is unaffected because find_library matches on path -- confirmed by a full refresh-metadata over 257 NFOs with 0 proxy requests straight after the rename. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Claude
|
89c3644d57 |
Script the Jellyfin library retirement
Matches libraries by path, never by name, and reads the name to delete back from the API rather than assuming it: the endpoint takes a name, matches loosely on some versions, and "YouTube" is a prefix of "YouTube (stream)". Refuses to retire the old library unless ytstream's has episodes, so a broken replacement cannot leave the server with no YouTube library. Dry run by default. Also records that /opt/youtube-automate is verified fully pushed to its bare repo -- clean tree, specs.md tracked -- so it is safe to delete, while /var/lib/youtube-automate holds subs.db, which is state and not in the repo. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Claude
|
c7a80d200e |
Hand cron over from youtube-automate, and script the rest
Cron now calls `ytstream run` hourly and ytstream's update-ytdlp.sh weekly. Both inherit youtube-automate's healthchecks UUIDs and keep its schedules unchanged: a check may be configured with a cron expression rather than a simple period, so moving to the :23/04:50 slots the fragment proposed could have alerted on a job that ran fine. Inheriting also means the placeholder UUIDs never needed filling in. The old service turned out to track only 2 channels, and one of them -- The Pyramid Podcast -- was sitting unresolved in ytstream's approval queue. Decommissioning without checking would have silently dropped half of what the old service existed to follow. Approved and backfilled. decommission.sh does the two steps needing root (nginx repoint, disable the unit) and refuses until an admin password is set, because the UI fails closed and the hostname would otherwise serve a login nobody can pass. Deleting the 2.0 GB of old downloads and touching the Jellyfin libraries are left out on purpose. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Claude
|
b35b6a3689 |
Clear the stranded tmpfs cache at startup
The session map is memory-only, so session directories surviving a restart can never be served and never be evicted -- and the work root is a tmpfs, so that is leaked RAM until reboot. The restart that ships the TTFB fix would have stranded 1.56 GB. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Claude
|
d3616e7532 |
Bound time-to-first-byte so a first play actually plays
Playback failed in Jellyfin the day after deployment, for uncached videos only. The handler blocked on the full download-and-mux before sending anything at all -- not even response headers -- so a 46-minute upload sat silent for 79 seconds and the client gave up. The proxy counted it a success, which is why /healthz and doctor both looked fine. Yesterday's "DirectPlay verified" only ever ran against videos already pulled during testing, so the first-play path was never exercised. The output is already a fragmented MP4, so it is readable while being written; a finished file only buys a correct duration and working seeks. Cap the wait at --first-byte-grace (12s, explicit in the unit) and stream whatever has not muxed by then. Measured: 156s -> 12.0s TTFB on a 66-minute upload, with ranges on a complete file unchanged. The cost is a first play with no seek bar when the grace is missed. Every later play of that video is perfect. Also fixes a hang found while testing this: the streaming loop waited on "finished and good" rather than "finished", so a producer that died after writing some bytes held the connection for the full 45s stall timeout and then dropped it. The stale yt-dlp warning pointed at youtube-automate's venv, the tree we are decommissioning; it now names ytstream's. /healthz reports the serving mode and grace. Eight new proxy tests, 353 passing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tom FluxandClaude Opus 5
|
d3bf8d6f19 |
Deploy it, and fix the six things installation found
Both units are installed and running, 10 of 119 channels approved, 251 episodes live in Jellyfin with verified DirectPlay. 345 tests. Six problems surfaced that no test could have, and two of them were mine in the deploy scripts. deploy.sh had a circular dependency with bootstrap.sh: deploy started the units and told the operator to run bootstrap, but bootstrap refused to run until the state directory existed, which only deploy creates. The units started against a non-existent venv, failed 203/EXEC and restart-looped 17 and 21 times. deploy.sh now creates the directory, calls bootstrap itself through runuser so the venv is not left root-owned, and refuses to start units when the venv is still missing. Deno was absent, and `doctor` is the only reason we know. It is mandatory rather than nice-to-have — without a JS runtime yt-dlp cannot solve the n challenge, which youtube-automate measured on this machine as 22 formats instead of 29 plus throttling. Nothing else would have complained; playback would just have quietly degraded. bootstrap.sh now installs it and asserts yt-dlp reports it. Episodes had no synopsis at all, because materialise passed plot=None while both sources hand us descriptions for free. Now plumbed through from RSS (media:group/media:description) and from videos.list, which carries snippet.description in the call already being made for durations — so the ~40% of episodes older than RSS reaches get one too. That needed a schema v2 migration; v1 was left exactly as shipped so a fresh install and a migrated one are identical, and a test asserts it. `materialise --all` — the documented recovery from a Jellyfin metadata wipe — was itself creating duplicates. Episode numbers were re-derived each run, and next_episode() excludes the row being numbered, so re-materialising a day's videos in a different order renumbered them and orphaned the old files. One run left 102 orphaned NFOs against 251 episodes. An episode number is now permanent once assigned, and a video whose rel_path changes has its old files removed first. Running it twice is now a no-op. Two Jellyfin behaviours worth having in writing. It ignores <runtime> and <durationinseconds> for episodes while reading the rest of the NFO happily, so a .strm shows no duration until first played — not fixable without probing, which is the one thing this design exists to avoid. And a plain /Library/Refresh does not reliably re-read a rewritten NFO: after rewriting all 251, fifty kept their old empty metadata. The fix is metadataRefreshMode=Default with replaceAllMetadata=false, which took plots from 201 to 251 while the proxy served zero requests. §5's prohibition on replaceAllMetadata=true still stands — that one probes. Exposed as `ytstream refresh-metadata` and run automatically after `materialise --all`. The measurement §5 has been waiting for: a full Jellyfin scan of 251 .strm files took ~119 s, about 8 minutes per 1,000 episodes, and made zero media probes. That last number is the fact the whole design rests on, now confirmed at scale on the real library rather than on seven PoC files. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tom FluxandClaude Opus 5
|
61cc1672ec |
Admin routes for the subscription queue, and record the build in plan.md
Phase 4's acceptance criterion is that my brother approves the first import himself, which needs a UI, so /pending now carries source management, sync-now, and multi-select approve/reject. Driven over real HTTP rather than only through the templates: unauthenticated requests redirect to login, all four new routes reject a missing or forged CSRF token, and the live account rendered 117 checkboxes. Approving three at once added three channels, which is the point of the fix underneath. _form() collapses repeated fields to the last value, which is correct for every single-value field but silently wrong for a form of checkboxes all named `id` — it would have approved only the last box ticked. Added _form_list(), with the parsed body cached because rfile can only be read once and the approval path needs both views of it. The approval page is deliberately its own page rather than a section on the index: the first sync of the real account queued 119 channels, and that does not belong inline under the channel table. Source errors are shown in full rather than truncated, because the useful ones say exactly what to do — "subscriptions are private, uncheck Keep all my subscriptions private" — and hiding that behind a log file defeats the purpose of surfacing it. plan.md §13 now reflects what is actually built rather than what was intended, and a new §17 records the three bugs the build turned up, including which of them a test caught and which two needed real data. 337 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tom FluxandClaude Opus 5
|
f640c064c6 |
Phase 0 complete: all three API assumptions verified, and 119 subscriptions
The key works. All three assumptions the plan rested on hold, so nothing in the
design had to change — but the measurements moved several numbers by a lot, and
one of them exposed a design problem the plan did not have an answer for.
Verified against the live API:
subscriptions.list?channelId= 200, 119 channels over 3 pages. An API key is
sufficient; no OAuth, no consent screen.
playlistItems.list on UULF Accepts the undocumented long-form playlist id
and returns exact videoPublishedAt. 1,249 items
against 2,724 in UU, so it excludes 54% of the
catalogue. The UU-plus-duration-filter fallback
is not needed.
videos.list durations 50 ids in one call, 50 back, 0 unparseable. Of
those 50 consecutive UU uploads, 38 were <=120s,
which is the Shorts filter earning its keep.
The account has 119 subscriptions, not the ~20 this plan assumed, so rather than
extrapolate I probed all 119 UULF feeds: 441 long-form videos in the last 30
days, median 1 per channel, and all 119 feeds polled in 7.9s on 8 threads. The
library is therefore smaller than the previous 20-channel estimate despite six
times the channels, because the distribution is severely skewed. Quota, scan cost
and poll cost are all now measured rather than guessed, and none of them is a
constraint: ~130 units to build, ~75/day steady state, under 1% of the budget.
The problem that fell out: 52 of 117 channels uploaded nothing long-form in 30
days, so they would appear in Jellyfin as empty series, and a channel uploading
every six weeks would flicker in and out as its one video crossed the retention
line. With a median of 1 upload a month that is the common case, not an edge
case. Recommended fix in §5 — retention becomes max(30 days, N most recent
videos) via min_keep_videos, default 5, which costs ~260 extra episodes and
removes the flicker entirely because a video only leaves once 5 newer ones exist.
Flagged as a decision rather than applied unilaterally.
Also recorded, because it cost three attempts to get the key working and every
failure was an indistinguishable 403 forbidden: SERVICE_DISABLED means the API is
not enabled on the project, API_KEY_SERVICE_BLOCKED means the key's own
restrictions exclude it, they are fixed on different console screens, and the
signal is in error.details[].reason rather than error.errors[].reason. The
ordering matters — the API must be enabled before the key can be restricted to
it, because it is absent from the picker until then. And after the fix, calls
still failed at 0s, 30s and 60s and succeeded at 90s; nothing was wrong except
impatience.
One more trap worth having in writing: pageInfo.totalResults reported 127 while
pagination returned 119 distinct channels, and 2 of those 119 have feeds that
HTTP-error. Terminated and private channels are still counted as subscriptions,
so anything sized off totalResults will be wrong.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Tom FluxandClaude Opus 5
|
f3d70f1c87 |
Tell the two API-key setup failures apart in verify_api.py
Ran Phase 0 against a live key and hit both of the ways a fresh Google Cloud
project can be wrong, in sequence. Google reports them as the same 403
`forbidden`, so the first version of this script printed reason='forbidden'
three times and buried the one sentence that said what to do.
The distinguishing signal is in error.details[].reason, not
error.errors[].reason:
SERVICE_DISABLED YouTube Data API v3 is not enabled on the project.
Carries an activationUrl naming the project number.
API_KEY_SERVICE_BLOCKED The API is enabled, but this key's API restrictions
exclude it.
They are fixed on different console screens, so they are now separate exception
types with separate advice, and a one-call preflight reports either before the
three real checks run and fail identically.
The ordering between them is a trap worth writing down: YouTube Data API v3 does
not appear in a key's API-restriction picker until the API is enabled on the
project, so creating the key and restricting it first yields a key that blocks
the only API it exists for. That is precisely what happened here. §4.1 step 4
now says to enable before restricting.
Nothing has yet reached YouTube's own privacy check, so whether the brother's
subscriptions are readable is still untested — every call so far failed at the
key.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
||
|
Tom FluxandClaude Opus 5
|
339232c7f9 |
Phase 0: rolling 30-day window, delete on unsubscribe, shared password
Phase 0 is done apart from one external dependency, and what it measured changed the shape of the plan more than the three answers it was meant to confirm. Measured the actual upload rates from the live UULF feeds rather than guessing from catalogue size, and the numbers make the whole scale section boring. Pitch Side publishes 0.60 long-form videos a day, The Pyramid Podcast 0.77 — so a 30-day window is about 20 episodes per channel and roughly 400 across a 20-channel library, not the 20,000 the previous draft braced for. The UULF feed is also doing more work than expected: it excludes 60-74% of what these channels publish, all of it Shorts and livestreams nobody wants as Jellyfin episodes. A side effect worth recording: Pitch Side's 15-entry UULF feed spans 23.3 days, because the feed holds 15 long-form items rather than 15 uploads. Free RSS therefore covers most of a 30-day retention window on its own, which narrows what the API is for. It still earns its place — RSS carries no duration, and subscription reading has no alternative at all — but the honest version is that it now buys correctness far more than it buys safety, and the plan says so instead of keeping the more flattering 4-hours-of-yt-dlp argument. Three decisions folded in. Retention is a rolling 30 days, superseding the earlier "3 months or 300 videos". Those answered different questions — backfill depth versus retention — and holding both would mean backfilling 90 days and deleting two thirds of it on the next sweep. One number now governs both ends, so the library cannot grow. This resurrects reap.py, which the previous draft deleted, and makes tombstones load-bearing in two ways: without them the next poll re-materialises everything the sweep just deleted, and episode ordinals for a given day would shift as videos disappear. aged_out is deliberately never revivable, or raising retention_days would resurrect months of episodes into Jellyfin as new. Unsubscribing now deletes the channel rather than deactivating it, which is reasonable when rebuilding costs one API page and ~20 files. The consequence is that the removal-detection rules stop being precautionary and become the only thing standing between a transient 403 and a wiped library, so the plan now says that explicitly next to them. Auth reverts to the single shared admin_password_hash, matching every other service on susan. web/auth.py carries over untouched. What that gives up — independent revocation, and knowing who approved what — is named once and then dropped. Added tools/verify_api.py so the remaining Phase 0 work is one command. It runs all three API checks, paginates the subscription list, prints the subsync_max_new value derived from totalResults, distinguishes 403 subscriptionForbidden from every other failure with the fix in the message, and exits non-zero so it can be gated on. The ISO-8601 duration parser is unit-checked. Blocked on: there is no Google API key anywhere on this machine (searched /opt/*, /home/susan, the settings table and the config trees for AIza-shaped strings). Creating one needs a browser and a Google account. That plus the brother unchecking "Keep all my subscriptions private" are the two external prerequisites, and §16 records exactly what each unverified assumption costs if it turns out false. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tom FluxandClaude Opus 5
|
8cca4fe1e2 |
Plan ytstream as a service, and mirror a YouTube account's subscriptions
The streaming proof of concept works — FINDINGS.md records the measurements —
but it runs by hand out of a home directory. This is the plan for making it a
real service, plus the design for the new requirement: my brother subscribes to
a channel on YouTube and it appears in Jellyfin without anyone touching an
admin page.
Three things drove the shape of the plan.
It is a fork of youtube-automate, not a green-field build. That project is 3,543
lines and most of it — RSS discovery, episode numbering, NFO schema, scrypt auth,
the admin UI — has nothing to do with downloading and was verified on this
machine on 2026-08-11. Only download.py and reap.py actually die. Re-deriving the
rest would mean re-earning knowledge we already paid for.
Getting a Google API key for the subscription feature turns out to pay for
itself twice over, so the plan leans on it much harder than the feature needs.
subscriptions.list, playlistItems.list and videos.list are 1 unit each against
10,000/day, which means the whole metadata path can move off yt-dlp: exact upload
dates from contentDetails.videoPublishedAt instead of approximate_date (measured
wrong by up to 2 days, and episode numbers are derived from it), durations
without an extraction per video, and no residential-IP request storm when a
channel is added. yt-dlp is then only ever invoked by the proxy, for one video,
because a human pressed play. That is a much better boundary than the one we
have.
The brother's channel is the sole source of truth. An earlier draft modelled
subscriptions as multi-source claims so a manually-added channel could not be
deleted by someone else's list; with one source that is dead weight, and
speculative generality in the code path that deletes things is the wrong place to
spend it. What survives instead is paranoia about removals, which now matter more
rather than less: a 403, a timeout or an empty response is never read as an
unsubscribe, a channel must be absent from three consecutive healthy syncs, and
even then the tree stays on disk and the channel merely goes inactive.
Verified while writing this, rather than assumed:
- subscriptions.list accepts a channelId filter and is not documented as
needing an authorized request, so an API key is enough. It returns 403 —
not an empty list — when subscriptions are private, which is what makes the
removal safeguards able to tell "he made them private again" apart from "he
unsubscribed from everything".
- There is no scraping fallback. /@cflux1030/channels returns 200 but silently
serves the Home tab: four tabs, zero channel ids in ytInitialData. YouTube
retired the public subscriptions tab, so the privacy checkbox is mandatory
rather than merely convenient.
- @cflux1030 resolves to UCPcTWaLV8zwx4WP4QExHj4Q, so no channels.list call is
needed at build time.
- OAuth as a fallback carries a trap worth writing down: a consent screen in
"Testing" status issues refresh tokens that expire in 7 days, and
youtube.readonly is not one of the exempt basic scopes.
Catalogue depth is bounded at 3 months or 300 videos per channel, whichever
comes first. The plan says plainly that this bounds the initial build and not the
library, which grows forward indefinitely — roughly 15,000 episodes a year at
20 channels — so the Jellyfin scan-cost measurement in Phase 3 is what tells us
when that becomes a problem. Deliberately not solved now.
One number is still open: subsync_max_new depends on how many channels he
actually follows, which only the day-one API call can tell us. Phase 0 closes it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|