04721a78eac9a10efa4fdd30f2006b4b1e459766
7
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Claude
|
22d8828080 |
Never serve a partial file: it makes Jellyfin transcode
The 12s first-byte grace was the wrong trade and a real play found it within the hour. A 2-hour upload took 3 minutes to start, played 6 seconds, and stalled. Jellyfin had run ffmpeg with -probesize 1G against the growing stream and then transcoded to HLS with libx264. The cause is the container. A fragmented MP4 with empty_moov has no duration in its header, so the only way to get one is to sum every fragment -- probing a growing file reads all of it. Jellyfin cannot establish duration, codec or bitrate, so it abandons direct play and transcodes a stream it also cannot seek. It was targeting 4.83 Mbps against a source measured at 3.29: re-encoding a stream that already fit, because it could not measure it. The same video once complete reports SupportsDirectPlay with the exact runtime and bitrate. So FIRST_BYTE_GRACE defaults to infinite again, with --wait-timeout raised to 600s for a 2-hour upload. A cold long video is slow to start, which is accepted: the fetch outlives the request so a retry is instant, and a retryable stall beats a transcode that wastes a gigabyte and cannot work. Both failure modes are recorded at the constant in the order measured so the 12s cap is not reintroduced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Claude
|
28d0e83130 |
Fix the Jellyfin rename call: it takes a name, not an id
POST /Library/VirtualFolders/Name is the odd one out in that controller -- most of /Library/VirtualFolders/* takes an id, and passing one here returns a bare "HTTP 400: Error processing request." that says nothing about why. Verified against 10.11.4: name -> 204. Also records that renaming re-ids the library, because Jellyfin derives the ItemId from the name. ytstream is unaffected because find_library matches on path -- confirmed by a full refresh-metadata over 257 NFOs with 0 proxy requests straight after the rename. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Claude
|
89c3644d57 |
Script the Jellyfin library retirement
Matches libraries by path, never by name, and reads the name to delete back from the API rather than assuming it: the endpoint takes a name, matches loosely on some versions, and "YouTube" is a prefix of "YouTube (stream)". Refuses to retire the old library unless ytstream's has episodes, so a broken replacement cannot leave the server with no YouTube library. Dry run by default. Also records that /opt/youtube-automate is verified fully pushed to its bare repo -- clean tree, specs.md tracked -- so it is safe to delete, while /var/lib/youtube-automate holds subs.db, which is state and not in the repo. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Claude
|
c7a80d200e |
Hand cron over from youtube-automate, and script the rest
Cron now calls `ytstream run` hourly and ytstream's update-ytdlp.sh weekly. Both inherit youtube-automate's healthchecks UUIDs and keep its schedules unchanged: a check may be configured with a cron expression rather than a simple period, so moving to the :23/04:50 slots the fragment proposed could have alerted on a job that ran fine. Inheriting also means the placeholder UUIDs never needed filling in. The old service turned out to track only 2 channels, and one of them -- The Pyramid Podcast -- was sitting unresolved in ytstream's approval queue. Decommissioning without checking would have silently dropped half of what the old service existed to follow. Approved and backfilled. decommission.sh does the two steps needing root (nginx repoint, disable the unit) and refuses until an admin password is set, because the UI fails closed and the hostname would otherwise serve a login nobody can pass. Deleting the 2.0 GB of old downloads and touching the Jellyfin libraries are left out on purpose. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Claude
|
d3616e7532 |
Bound time-to-first-byte so a first play actually plays
Playback failed in Jellyfin the day after deployment, for uncached videos only. The handler blocked on the full download-and-mux before sending anything at all -- not even response headers -- so a 46-minute upload sat silent for 79 seconds and the client gave up. The proxy counted it a success, which is why /healthz and doctor both looked fine. Yesterday's "DirectPlay verified" only ever ran against videos already pulled during testing, so the first-play path was never exercised. The output is already a fragmented MP4, so it is readable while being written; a finished file only buys a correct duration and working seeks. Cap the wait at --first-byte-grace (12s, explicit in the unit) and stream whatever has not muxed by then. Measured: 156s -> 12.0s TTFB on a 66-minute upload, with ranges on a complete file unchanged. The cost is a first play with no seek bar when the grace is missed. Every later play of that video is perfect. Also fixes a hang found while testing this: the streaming loop waited on "finished and good" rather than "finished", so a producer that died after writing some bytes held the connection for the full 45s stall timeout and then dropped it. The stale yt-dlp warning pointed at youtube-automate's venv, the tree we are decommissioning; it now names ytstream's. /healthz reports the serving mode and grace. Eight new proxy tests, 353 passing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tom FluxandClaude Opus 5
|
d3bf8d6f19 |
Deploy it, and fix the six things installation found
Both units are installed and running, 10 of 119 channels approved, 251 episodes live in Jellyfin with verified DirectPlay. 345 tests. Six problems surfaced that no test could have, and two of them were mine in the deploy scripts. deploy.sh had a circular dependency with bootstrap.sh: deploy started the units and told the operator to run bootstrap, but bootstrap refused to run until the state directory existed, which only deploy creates. The units started against a non-existent venv, failed 203/EXEC and restart-looped 17 and 21 times. deploy.sh now creates the directory, calls bootstrap itself through runuser so the venv is not left root-owned, and refuses to start units when the venv is still missing. Deno was absent, and `doctor` is the only reason we know. It is mandatory rather than nice-to-have — without a JS runtime yt-dlp cannot solve the n challenge, which youtube-automate measured on this machine as 22 formats instead of 29 plus throttling. Nothing else would have complained; playback would just have quietly degraded. bootstrap.sh now installs it and asserts yt-dlp reports it. Episodes had no synopsis at all, because materialise passed plot=None while both sources hand us descriptions for free. Now plumbed through from RSS (media:group/media:description) and from videos.list, which carries snippet.description in the call already being made for durations — so the ~40% of episodes older than RSS reaches get one too. That needed a schema v2 migration; v1 was left exactly as shipped so a fresh install and a migrated one are identical, and a test asserts it. `materialise --all` — the documented recovery from a Jellyfin metadata wipe — was itself creating duplicates. Episode numbers were re-derived each run, and next_episode() excludes the row being numbered, so re-materialising a day's videos in a different order renumbered them and orphaned the old files. One run left 102 orphaned NFOs against 251 episodes. An episode number is now permanent once assigned, and a video whose rel_path changes has its old files removed first. Running it twice is now a no-op. Two Jellyfin behaviours worth having in writing. It ignores <runtime> and <durationinseconds> for episodes while reading the rest of the NFO happily, so a .strm shows no duration until first played — not fixable without probing, which is the one thing this design exists to avoid. And a plain /Library/Refresh does not reliably re-read a rewritten NFO: after rewriting all 251, fifty kept their old empty metadata. The fix is metadataRefreshMode=Default with replaceAllMetadata=false, which took plots from 201 to 251 while the proxy served zero requests. §5's prohibition on replaceAllMetadata=true still stands — that one probes. Exposed as `ytstream refresh-metadata` and run automatically after `materialise --all`. The measurement §5 has been waiting for: a full Jellyfin scan of 251 .strm files took ~119 s, about 8 minutes per 1,000 episodes, and made zero media probes. That last number is the fact the whole design rests on, now confirmed at scale on the real library rather than on seven PoC files. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> |
||
|
Tom FluxandClaude Opus 5
|
155f05773d |
Build ytstream: catalogue, retention, subscription mirror, proxy
Phases 1-4 of plan.md §13. Forked from youtube-automate as planned rather than
written from scratch: naming, NFO, auth, the admin UI, settings and the DB layer
came across largely unchanged, download.py is gone, and the pieces that only make
sense for a streaming library are new.
api.py YouTube Data API v3 client. The whole metadata path.
strm.py Materialising: a .strm, an .nfo and a thumbnail. Replaces the
330-line download.py, because the job is writing a URL to a file.
subsync.py The subscription mirror, most of which is refusals.
reap.py Retention, rewritten around the 30-day window and min_keep_videos.
discovery.py RSS polling plus an API-backed, resumable, bounded backfill.
proxy/ The verified PoC, moved in with a systemd unit.
330 tests, all passing, no network and no yt-dlp in any of them. The suite leans
towards the failure paths, because that is where this design can actually hurt
someone: a 403 that looks like an unsubscribe, a video that ages out and comes
back, a title that never arrives. tests/test_proxy.py replaces the two standalone
scripts under proxy/ and now drives the real make_handler(mgr, ...) rather than the
PoC's make_handler(path, done), so routing and video-id validation are covered too.
Ran it end to end against the live API and it found three real bugs.
The first was mine and the tests caught it: strm.remove pruned empty directories
up to the media root, so a channel directory whose tvshow.nfo happened to be
missing would be deleted along with the season. It only looked safe because
tvshow.nfo normally stops the walk. The prune boundary is now the channel
directory explicitly.
The other two only showed up against real data, and they compounded. The backfill
inserted rows with no title and left the RSS poll to fill them in — but RSS returns
15 entries, which for Pitch Side spans 23 days against a 30-day window, so five of
twenty episodes were named after their video ids. Worse, strm.materialise wrote
that fallback back to the database as the title, which made the row look titled and
permanently disabled the repair path. Both fixed: playlistItems.list now requests
snippet as well as contentDetails, which costs the same single quota unit and
carries the title alongside the exact publish date, and the fallback is used for the
filename without being persisted. A title that does arrive late now also removes the
badly-named files and re-queues, so the episode is rewritten rather than keeping its
video-id name forever. Verified against the live API: all twenty Pitch Side episodes
now carry real titles.
Measured on the real account: 119 subscriptions queued for approval and none added
on the first sync, then a two-channel run backfilled and materialised 26 episodes in
under seven seconds.
Two deliberate departures from plan.md, both recorded there:
min_keep_videos defaults to 5 rather than being left as an open question. Without
it 52 of 117 measured channels are empty Jellyfin series that flicker in and out as
their single video crosses the retention line, and the plan already recommended it.
The Jellyfin refresh is a bare /Library/Refresh with a comment explaining why it
must stay that way. A normal scan makes zero media probes; FullRefresh does probe,
and at 400 episodes that is 400 cold starts.
Not yet done: no systemd units are installed (needs root — deploy/deploy.sh), the
admin UI has no routes for sources or the approval queue yet, and nothing has been
pointed at the real media root.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|