tom/ytstream
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Never serve a partial file: it makes Jellyfin transcode

Browse Source 
The 12s first-byte grace was the wrong trade and a real play found it
within the hour. A 2-hour upload took 3 minutes to start, played 6
seconds, and stalled. Jellyfin had run ffmpeg with -probesize 1G against
the growing stream and then transcoded to HLS with libx264.

The cause is the container. A fragmented MP4 with empty_moov has no
duration in its header, so the only way to get one is to sum every
fragment -- probing a growing file reads all of it. Jellyfin cannot
establish duration, codec or bitrate, so it abandons direct play and
transcodes a stream it also cannot seek. It was targeting 4.83 Mbps
against a source measured at 3.29: re-encoding a stream that already fit,
because it could not measure it.

The same video once complete reports SupportsDirectPlay with the exact
runtime and bitrate.

So FIRST_BYTE_GRACE defaults to infinite again, with --wait-timeout raised
to 600s for a 2-hour upload. A cold long video is slow to start, which is
accepted: the fetch outlives the request so a retry is instant, and a
retryable stall beats a transcode that wastes a gigabyte and cannot work.
Both failure modes are recorded at the constant in the order measured so
the 12s cap is not reintroduced.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Claude
2026-08-13 11:32:12 +01:00
parent c12837cbb9
commit 22d8828080
10 changed files with 514 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
proxy/ytstream_proxy.py
+65 -34
View File
@@ -26,25 +26,20 @@ Safety rails, because a Jellyfin library scan can ask for every episode at once:
hour; a refresh storm hits the cap in seconds.
--cache-gb tmpfs budget; least-recently-used complete files are evicted.
Serving modes. The output is a fragmented MP4, so it is readable while it is
still being written; the only thing a finished file buys is a correct duration
and working seeks, because ffmpeg patches the real duration into the moov on
close and ignores both mvhd.duration and an injected mehd box before then.
Serving modes. The output is a fragmented MP4, so it is *readable* while being
written -- but do not serve it that way to Jellyfin. See FIRST_BYTE_GRACE below:
a growing fragmented MP4 has no duration in its header, so Jellyfin drags up to a
gigabyte through this proxy trying to probe one and then transcodes a stream it
cannot seek. A slow start beats that.
default wait for the mux, but only for --first-byte-grace seconds,
then serve what exists. Short videos finish inside the
grace and keep the exact duration and ranges; long ones
start immediately and regain both when the mux lands.
--first-byte-grace raise it above --wait-timeout for the old strict behaviour
(finished file or 504), which is only safe for a client
that will wait minutes for its first byte.
default wait for the complete mux, bounded by --wait-timeout, then
serve it with real ranges and the exact duration. A cold
long video is slow to start; the fetch continues after the
client gives up, so the next attempt is instant.
--first-byte-grace finite value: stop waiting after N seconds and stream the
partial file. Fast start, but only for a client that
tolerates an unseekable, duration-less stream.
--growing grace of zero: serve as soon as any byte exists.
Why the grace exists at all, measured 2026-08-13 on this machine: the wait is
~3s to extract plus roughly duration/50..110 to pull and mux, so a 46-minute
upload took 79 seconds during which the handler sent nothing -- not even
response headers. Jellyfin reported "playback error" on the first play of every
long video while already-cached ones played perfectly.
"""
import argparse
@@ -73,15 +68,35 @@ POT_ARGS = "youtubepot-bgutilhttp:base_url=http://127.0.0.1:4416"
CLIENT_ARGS = "youtube:player_client=default"
MAX_HEIGHT = 720
STALL_TIMEOUT = 45.0
# How long a request may block waiting for a *complete* mux before it gives up and
# streams the partial file instead. This is a cap on time-to-first-byte, not a
# prediction: measured cold on 2026-08-13, completion ran from 10s (8-minute
# upload) to 156s (66-minute upload), and it does not follow the duration closely
# because YouTube's per-format throttling varies and one transient failure plus a
# retry costs ~5s of extraction before any byte is pulled. So do not tune this
# expecting to cover "videos up to N minutes" -- it catches the quick cases,
# absorbs a single retry, and bounds the bad case at something a player tolerates.
FIRST_BYTE_GRACE = 12.0
# How long a request may block waiting for a *complete* mux before giving up and
# streaming the partial file instead. Infinite by default, i.e. wait for the whole
# thing (bounded by --wait-timeout), because serving a partial file to Jellyfin is
# WORSE than making it wait. Measured 2026-08-13, in this order:
#
# 1. Waiting for the complete file with no cap meant a 46-minute upload sent
# nothing for 79s and the client gave up -> "playback error".
# 2. So this was capped at 12s and the partial file streamed instead. A
# 2-hour upload then took 3 minutes to start, played 6 seconds, and stalled.
# Jellyfin had run:
# ffmpeg -analyzeduration 200M -probesize 1G -i .../watch/<id> ... libx264
# It TRANSCODED, after dragging up to a gigabyte through the proxy to probe.
#
# The reason is the container. A fragmented MP4 with empty_moov carries no
# duration in its header -- the only way to get one is to sum every fragment, so
# probing a growing file reads all of it. Jellyfin cannot establish duration,
# codec or bitrate, so it stops trusting direct play and transcodes a stream it
# also cannot seek. On the *complete* file ffmpeg patches the real duration into
# the moov on close, the probe is cheap, and Jellyfin reported SupportsDirectPlay
# with the exact runtime and a 3.29 Mbps bitrate -- comfortably under the
# 4.83 Mbps cap it had been transcoding down to.
#
# So a slow start is the correct failure: the fetch continues after the client
# gives up, and the next attempt is instant. A partial file is a fast start
# followed by a transcode that cannot work.
#
# Set a finite value only for a client that tolerates an unseekable, duration-less
# stream. Jellyfin does not.
FIRST_BYTE_GRACE = float("inf")
VIDEO_ID_RE = re.compile(r"^[A-Za-z0-9_-]{11}$")
_log_lock = threading.Lock()
@@ -330,6 +345,10 @@ class Manager:
# Reported by /healthz so the serving mode is visible without reading the
# unit file. Set by main(); the handler holds the value it actually uses.
self.first_byte_grace = 0.0 if growing else FIRST_BYTE_GRACE
# True when a request waits for the whole mux rather than ever serving a
# partial file. Overwritten by main(); the default matches the default
# grace so a Manager built directly (tests) reports the same thing.
self.strict = not growing
self.lock = threading.Lock()
self.sessions = {}
self.counters = {"requests": 0, "started": 0, "reused": 0,
@@ -460,8 +479,16 @@ class Manager:
def status(self):
with self.lock:
return {
"mode": "growing" if self.growing else "wait-then-stream",
"first_byte_grace_s": self.first_byte_grace,
# `null` rather than Infinity: json.dumps would happily emit the
# latter, and it is not valid JSON for whoever reads this.
"mode": ("growing" if self.growing
else "wait-for-complete" if self.strict
else "wait-then-stream"),
# `null` rather than a number when strict: there is no cap, and
# json.dumps would otherwise emit Infinity, which is not JSON.
"first_byte_grace_s": (
None if self.strict or self.first_byte_grace == float("inf")
else self.first_byte_grace),
"no_fetch": self.no_fetch,
"max_pipelines": self.max_pipelines,
"max_starts": self.max_starts,
@@ -768,9 +795,11 @@ def main():
help="hard cap on blocking for the mux; only reachable when "
"--first-byte-grace is raised to meet it")
ap.add_argument("--first-byte-grace", type=float, default=FIRST_BYTE_GRACE,
help=f"seconds to wait for a complete mux before streaming "
f"the partial file instead (default {FIRST_BYTE_GRACE:g}). "
f"Raise to --wait-timeout for finished-file-or-504")
help="seconds to wait for a complete mux before streaming the "
"partial file instead. Default is to wait for the whole "
"thing: a partial fragmented MP4 makes Jellyfin transcode "
"after a 1 GB probe. Only set this for a client that "
"tolerates an unseekable, duration-less stream")
ap.add_argument("--max-starts", type=int, default=20,
help="max cold starts per window; bounds a runaway library "
"refresh (default 20)")
@@ -814,6 +843,7 @@ def main():
args.max_retries, args.max_starts, args.starts_window)
grace = 0.0 if args.growing else min(args.first_byte_grace, args.wait_timeout)
mgr.strict = not args.growing and grace >= args.wait_timeout
mgr.first_byte_grace = grace
if args.no_fetch:
@@ -822,9 +852,10 @@ def main():
if grace <= 0:
log("growing mode: a first play seeks badly and shows no duration until "
"the mux lands")
elif grace >= args.wait_timeout:
log(f"strict mode: a request blocks up to {grace:g}s for a complete mux "
f"and 504s otherwise -- long videos will fail to start")
elif mgr.strict:
log(f"waiting for a complete mux, up to {grace:g}s. A cold long video is "
f"slow to start and the fetch outlives the request, so a retry is "
f"instant -- see FIRST_BYTE_GRACE for why partial is worse")
else:
log(f"waiting up to {grace:g}s for a complete mux, then streaming the "
f"partial file")