Never serve a partial file: it makes Jellyfin transcode
The 12s first-byte grace was the wrong trade and a real play found it within the hour. A 2-hour upload took 3 minutes to start, played 6 seconds, and stalled. Jellyfin had run ffmpeg with -probesize 1G against the growing stream and then transcoded to HLS with libx264. The cause is the container. A fragmented MP4 with empty_moov has no duration in its header, so the only way to get one is to sum every fragment -- probing a growing file reads all of it. Jellyfin cannot establish duration, codec or bitrate, so it abandons direct play and transcodes a stream it also cannot seek. It was targeting 4.83 Mbps against a source measured at 3.29: re-encoding a stream that already fit, because it could not measure it. The same video once complete reports SupportsDirectPlay with the exact runtime and bitrate. So FIRST_BYTE_GRACE defaults to infinite again, with --wait-timeout raised to 600s for a 2-hour upload. A cold long video is slow to start, which is accepted: the fetch outlives the request so a retry is instant, and a retryable stall beats a transcode that wastes a gigabyte and cannot work. Both failure modes are recorded at the constant in the order measured so the 12s cap is not reintroduced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
@@ -21,11 +21,15 @@ UMask=0002
|
|||||||
Environment=PATH=/var/lib/ytstream/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
|
Environment=PATH=/var/lib/ytstream/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
|
||||||
Environment=PYTHONUNBUFFERED=1
|
Environment=PYTHONUNBUFFERED=1
|
||||||
|
|
||||||
# --first-byte-grace is the difference between "plays" and "playback error".
|
# No --first-byte-grace: requests wait for the COMPLETE mux, bounded by
|
||||||
# Waiting for a complete mux costs ~3s to extract plus duration/50..110 to pull,
|
# --wait-timeout. Do not add a grace here without reading FIRST_BYTE_GRACE in the
|
||||||
# so a 46-minute upload blocked this socket for 79 silent seconds and Jellyfin
|
# proxy. Short version, both measured 2026-08-13: waiting sends nothing for ~80s
|
||||||
# gave up long before. Do not raise this to --wait-timeout without re-reading
|
# on a 46-minute upload and the client gives up, but serving the partial file
|
||||||
# the module docstring: that restores the behaviour that broke first plays.
|
# instead makes Jellyfin drag a gigabyte through this proxy to probe a fragmented
|
||||||
|
# MP4 that has no duration in its header, and then TRANSCODE a stream it cannot
|
||||||
|
# seek -- 3 minutes to start, 6 seconds of video, then a permanent stall.
|
||||||
|
# A slow cold start is the better failure: the fetch outlives the request, so the
|
||||||
|
# retry is instant. --wait-timeout is 600 to cover a 2-hour upload (2.4 GB).
|
||||||
|
|
||||||
WorkingDirectory=/opt/ytstream
|
WorkingDirectory=/opt/ytstream
|
||||||
ExecStart=/var/lib/ytstream/venv/bin/python3 /opt/ytstream/proxy/ytstream_proxy.py \
|
ExecStart=/var/lib/ytstream/venv/bin/python3 /opt/ytstream/proxy/ytstream_proxy.py \
|
||||||
@@ -35,7 +39,7 @@ ExecStart=/var/lib/ytstream/venv/bin/python3 /opt/ytstream/proxy/ytstream_proxy.
|
|||||||
--max-pipelines 2 \
|
--max-pipelines 2 \
|
||||||
--max-retries 2 \
|
--max-retries 2 \
|
||||||
--max-starts 20 --starts-window 3600 \
|
--max-starts 20 --starts-window 3600 \
|
||||||
--first-byte-grace 12
|
--wait-timeout 600
|
||||||
|
|
||||||
Restart=always
|
Restart=always
|
||||||
RestartSec=5
|
RestartSec=5
|
||||||
|
|||||||
@@ -1272,3 +1272,77 @@ he subscribes to something and nothing happens, which is the feature not working
|
|||||||
Reversible per channel with `ytstream unsubscribe`.
|
Reversible per channel with `ytstream unsubscribe`.
|
||||||
|
|
||||||
`approve --all`: **108 added, 0 failed, 70 s.**
|
`approve --all`: **108 added, 0 failed, 70 s.**
|
||||||
|
|
||||||
|
## 22. Serving a partial file to Jellyfin is worse than making it wait — 2026-08-13
|
||||||
|
|
||||||
|
§19 capped the wait for a complete mux at 12s and streamed the partial file after
|
||||||
|
that, to stop long videos failing to start. That was the wrong trade, and a real
|
||||||
|
play found it within the hour: a 2-hour upload took **3 minutes to start, played 6
|
||||||
|
seconds, and stalled permanently**.
|
||||||
|
|
||||||
|
### What Jellyfin actually did
|
||||||
|
|
||||||
|
```
|
||||||
|
ffmpeg -analyzeduration 200M -probesize 1G -i http://127.0.0.1:8099/watch/JYZYnkXMxdU
|
||||||
|
-codec:v:0 libx264 -preset veryfast -maxrate 4830438 ... -f hls
|
||||||
|
```
|
||||||
|
|
||||||
|
It **transcoded**, after dragging up to a gigabyte through the proxy to probe. The
|
||||||
|
Jellyfin log closes it out: `Playback stopped ... Stopped at "6016" ms`.
|
||||||
|
|
||||||
|
### Why — and it is the container, not the client
|
||||||
|
|
||||||
|
A fragmented MP4 written with `empty_moov` **has no duration in its header**. The
|
||||||
|
only way to obtain one is to sum every fragment, so probing a *growing* file means
|
||||||
|
reading all of it — hence `-probesize 1G`. Jellyfin therefore cannot establish
|
||||||
|
duration, codec or bitrate, stops trusting direct play, and transcodes a stream it
|
||||||
|
also cannot seek.
|
||||||
|
|
||||||
|
The same video once complete, via `PlaybackInfo`:
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| `SupportsDirectPlay` | **True** |
|
||||||
|
| `RunTimeTicks` | 1278 s (database says 1279) |
|
||||||
|
| `Bitrate` | 3,290,533 |
|
||||||
|
| Streams | h264 720p 3.16 Mbps + aac 128 kbps |
|
||||||
|
|
||||||
|
The transcode had been targeting `-maxrate 4830438` — **4.83 Mbps, above the
|
||||||
|
source's 3.29**. Jellyfin was re-encoding a stream that already fitted, purely
|
||||||
|
because it could not measure it. ffmpeg patches the real duration into the moov on
|
||||||
|
close, which is what makes the complete file cheap to probe and safe to direct-play.
|
||||||
|
|
||||||
|
### The decision
|
||||||
|
|
||||||
|
`FIRST_BYTE_GRACE` now defaults to **infinite** — wait for the complete mux,
|
||||||
|
bounded by `--wait-timeout` (raised to 600s to cover a 2-hour, 2.4 GB upload). A
|
||||||
|
cold long video is slow to start, and that is accepted deliberately:
|
||||||
|
|
||||||
|
* the fetch outlives the request, so a **retry is instant**;
|
||||||
|
* the failure is a stall the user can retry, not a runaway transcode that wastes a
|
||||||
|
gigabyte of transfer and cannot succeed.
|
||||||
|
|
||||||
|
Both failure modes are now recorded at the constant, in the order they were
|
||||||
|
measured, so nobody re-tries the 12s cap and rediscovers this. `/healthz` reports
|
||||||
|
`mode: wait-for-complete` and `first_byte_grace_s: null` — `null` rather than a
|
||||||
|
number because `json.dumps` renders an infinite float as `Infinity`, which is not
|
||||||
|
valid JSON.
|
||||||
|
|
||||||
|
### Still unresolved
|
||||||
|
|
||||||
|
**A cold long video does not start on the first press.** The grace was the wrong
|
||||||
|
fix for it, and the right one is not in yet. The options, none free:
|
||||||
|
|
||||||
|
1. **Pre-warm** the newest episode per channel after each run — turns the common
|
||||||
|
case ("watch the latest") instant, at the cost of fetching videos nobody asked
|
||||||
|
for, which is the premise the whole design rejects. Bounded, though: 119 videos.
|
||||||
|
2. **Faster pull.** `yt-dlp -o -` uses a single connection; measured 4.3 MB/s on a
|
||||||
|
525 MB upload. If concurrent ranged fetches work on these formats, a 2-minute
|
||||||
|
wait could become 20 seconds and the problem mostly disappears.
|
||||||
|
3. **Give Jellyfin the metadata up front** so it never probes: duration, codec and
|
||||||
|
bitrate are all known before a byte is fetched. `<streamdetails>` in the NFO was
|
||||||
|
measured not to affect scan probing, but its effect on `PlaybackInfo`
|
||||||
|
specifically has not been tested — and that is the one that decides transcoding.
|
||||||
|
|
||||||
|
Option 2 is the one to measure first: it is the only one that costs nothing and
|
||||||
|
helps every case.
|
||||||
|
|||||||
+65
-34
@@ -26,25 +26,20 @@ Safety rails, because a Jellyfin library scan can ask for every episode at once:
|
|||||||
hour; a refresh storm hits the cap in seconds.
|
hour; a refresh storm hits the cap in seconds.
|
||||||
--cache-gb tmpfs budget; least-recently-used complete files are evicted.
|
--cache-gb tmpfs budget; least-recently-used complete files are evicted.
|
||||||
|
|
||||||
Serving modes. The output is a fragmented MP4, so it is readable while it is
|
Serving modes. The output is a fragmented MP4, so it is *readable* while being
|
||||||
still being written; the only thing a finished file buys is a correct duration
|
written -- but do not serve it that way to Jellyfin. See FIRST_BYTE_GRACE below:
|
||||||
and working seeks, because ffmpeg patches the real duration into the moov on
|
a growing fragmented MP4 has no duration in its header, so Jellyfin drags up to a
|
||||||
close and ignores both mvhd.duration and an injected mehd box before then.
|
gigabyte through this proxy trying to probe one and then transcodes a stream it
|
||||||
|
cannot seek. A slow start beats that.
|
||||||
|
|
||||||
default wait for the mux, but only for --first-byte-grace seconds,
|
default wait for the complete mux, bounded by --wait-timeout, then
|
||||||
then serve what exists. Short videos finish inside the
|
serve it with real ranges and the exact duration. A cold
|
||||||
grace and keep the exact duration and ranges; long ones
|
long video is slow to start; the fetch continues after the
|
||||||
start immediately and regain both when the mux lands.
|
client gives up, so the next attempt is instant.
|
||||||
--first-byte-grace raise it above --wait-timeout for the old strict behaviour
|
--first-byte-grace finite value: stop waiting after N seconds and stream the
|
||||||
(finished file or 504), which is only safe for a client
|
partial file. Fast start, but only for a client that
|
||||||
that will wait minutes for its first byte.
|
tolerates an unseekable, duration-less stream.
|
||||||
--growing grace of zero: serve as soon as any byte exists.
|
--growing grace of zero: serve as soon as any byte exists.
|
||||||
|
|
||||||
Why the grace exists at all, measured 2026-08-13 on this machine: the wait is
|
|
||||||
~3s to extract plus roughly duration/50..110 to pull and mux, so a 46-minute
|
|
||||||
upload took 79 seconds during which the handler sent nothing -- not even
|
|
||||||
response headers. Jellyfin reported "playback error" on the first play of every
|
|
||||||
long video while already-cached ones played perfectly.
|
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
@@ -73,15 +68,35 @@ POT_ARGS = "youtubepot-bgutilhttp:base_url=http://127.0.0.1:4416"
|
|||||||
CLIENT_ARGS = "youtube:player_client=default"
|
CLIENT_ARGS = "youtube:player_client=default"
|
||||||
MAX_HEIGHT = 720
|
MAX_HEIGHT = 720
|
||||||
STALL_TIMEOUT = 45.0
|
STALL_TIMEOUT = 45.0
|
||||||
# How long a request may block waiting for a *complete* mux before it gives up and
|
# How long a request may block waiting for a *complete* mux before giving up and
|
||||||
# streams the partial file instead. This is a cap on time-to-first-byte, not a
|
# streaming the partial file instead. Infinite by default, i.e. wait for the whole
|
||||||
# prediction: measured cold on 2026-08-13, completion ran from 10s (8-minute
|
# thing (bounded by --wait-timeout), because serving a partial file to Jellyfin is
|
||||||
# upload) to 156s (66-minute upload), and it does not follow the duration closely
|
# WORSE than making it wait. Measured 2026-08-13, in this order:
|
||||||
# because YouTube's per-format throttling varies and one transient failure plus a
|
#
|
||||||
# retry costs ~5s of extraction before any byte is pulled. So do not tune this
|
# 1. Waiting for the complete file with no cap meant a 46-minute upload sent
|
||||||
# expecting to cover "videos up to N minutes" -- it catches the quick cases,
|
# nothing for 79s and the client gave up -> "playback error".
|
||||||
# absorbs a single retry, and bounds the bad case at something a player tolerates.
|
# 2. So this was capped at 12s and the partial file streamed instead. A
|
||||||
FIRST_BYTE_GRACE = 12.0
|
# 2-hour upload then took 3 minutes to start, played 6 seconds, and stalled.
|
||||||
|
# Jellyfin had run:
|
||||||
|
# ffmpeg -analyzeduration 200M -probesize 1G -i .../watch/<id> ... libx264
|
||||||
|
# It TRANSCODED, after dragging up to a gigabyte through the proxy to probe.
|
||||||
|
#
|
||||||
|
# The reason is the container. A fragmented MP4 with empty_moov carries no
|
||||||
|
# duration in its header -- the only way to get one is to sum every fragment, so
|
||||||
|
# probing a growing file reads all of it. Jellyfin cannot establish duration,
|
||||||
|
# codec or bitrate, so it stops trusting direct play and transcodes a stream it
|
||||||
|
# also cannot seek. On the *complete* file ffmpeg patches the real duration into
|
||||||
|
# the moov on close, the probe is cheap, and Jellyfin reported SupportsDirectPlay
|
||||||
|
# with the exact runtime and a 3.29 Mbps bitrate -- comfortably under the
|
||||||
|
# 4.83 Mbps cap it had been transcoding down to.
|
||||||
|
#
|
||||||
|
# So a slow start is the correct failure: the fetch continues after the client
|
||||||
|
# gives up, and the next attempt is instant. A partial file is a fast start
|
||||||
|
# followed by a transcode that cannot work.
|
||||||
|
#
|
||||||
|
# Set a finite value only for a client that tolerates an unseekable, duration-less
|
||||||
|
# stream. Jellyfin does not.
|
||||||
|
FIRST_BYTE_GRACE = float("inf")
|
||||||
VIDEO_ID_RE = re.compile(r"^[A-Za-z0-9_-]{11}$")
|
VIDEO_ID_RE = re.compile(r"^[A-Za-z0-9_-]{11}$")
|
||||||
|
|
||||||
_log_lock = threading.Lock()
|
_log_lock = threading.Lock()
|
||||||
@@ -330,6 +345,10 @@ class Manager:
|
|||||||
# Reported by /healthz so the serving mode is visible without reading the
|
# Reported by /healthz so the serving mode is visible without reading the
|
||||||
# unit file. Set by main(); the handler holds the value it actually uses.
|
# unit file. Set by main(); the handler holds the value it actually uses.
|
||||||
self.first_byte_grace = 0.0 if growing else FIRST_BYTE_GRACE
|
self.first_byte_grace = 0.0 if growing else FIRST_BYTE_GRACE
|
||||||
|
# True when a request waits for the whole mux rather than ever serving a
|
||||||
|
# partial file. Overwritten by main(); the default matches the default
|
||||||
|
# grace so a Manager built directly (tests) reports the same thing.
|
||||||
|
self.strict = not growing
|
||||||
self.lock = threading.Lock()
|
self.lock = threading.Lock()
|
||||||
self.sessions = {}
|
self.sessions = {}
|
||||||
self.counters = {"requests": 0, "started": 0, "reused": 0,
|
self.counters = {"requests": 0, "started": 0, "reused": 0,
|
||||||
@@ -460,8 +479,16 @@ class Manager:
|
|||||||
def status(self):
|
def status(self):
|
||||||
with self.lock:
|
with self.lock:
|
||||||
return {
|
return {
|
||||||
"mode": "growing" if self.growing else "wait-then-stream",
|
# `null` rather than Infinity: json.dumps would happily emit the
|
||||||
"first_byte_grace_s": self.first_byte_grace,
|
# latter, and it is not valid JSON for whoever reads this.
|
||||||
|
"mode": ("growing" if self.growing
|
||||||
|
else "wait-for-complete" if self.strict
|
||||||
|
else "wait-then-stream"),
|
||||||
|
# `null` rather than a number when strict: there is no cap, and
|
||||||
|
# json.dumps would otherwise emit Infinity, which is not JSON.
|
||||||
|
"first_byte_grace_s": (
|
||||||
|
None if self.strict or self.first_byte_grace == float("inf")
|
||||||
|
else self.first_byte_grace),
|
||||||
"no_fetch": self.no_fetch,
|
"no_fetch": self.no_fetch,
|
||||||
"max_pipelines": self.max_pipelines,
|
"max_pipelines": self.max_pipelines,
|
||||||
"max_starts": self.max_starts,
|
"max_starts": self.max_starts,
|
||||||
@@ -768,9 +795,11 @@ def main():
|
|||||||
help="hard cap on blocking for the mux; only reachable when "
|
help="hard cap on blocking for the mux; only reachable when "
|
||||||
"--first-byte-grace is raised to meet it")
|
"--first-byte-grace is raised to meet it")
|
||||||
ap.add_argument("--first-byte-grace", type=float, default=FIRST_BYTE_GRACE,
|
ap.add_argument("--first-byte-grace", type=float, default=FIRST_BYTE_GRACE,
|
||||||
help=f"seconds to wait for a complete mux before streaming "
|
help="seconds to wait for a complete mux before streaming the "
|
||||||
f"the partial file instead (default {FIRST_BYTE_GRACE:g}). "
|
"partial file instead. Default is to wait for the whole "
|
||||||
f"Raise to --wait-timeout for finished-file-or-504")
|
"thing: a partial fragmented MP4 makes Jellyfin transcode "
|
||||||
|
"after a 1 GB probe. Only set this for a client that "
|
||||||
|
"tolerates an unseekable, duration-less stream")
|
||||||
ap.add_argument("--max-starts", type=int, default=20,
|
ap.add_argument("--max-starts", type=int, default=20,
|
||||||
help="max cold starts per window; bounds a runaway library "
|
help="max cold starts per window; bounds a runaway library "
|
||||||
"refresh (default 20)")
|
"refresh (default 20)")
|
||||||
@@ -814,6 +843,7 @@ def main():
|
|||||||
args.max_retries, args.max_starts, args.starts_window)
|
args.max_retries, args.max_starts, args.starts_window)
|
||||||
|
|
||||||
grace = 0.0 if args.growing else min(args.first_byte_grace, args.wait_timeout)
|
grace = 0.0 if args.growing else min(args.first_byte_grace, args.wait_timeout)
|
||||||
|
mgr.strict = not args.growing and grace >= args.wait_timeout
|
||||||
mgr.first_byte_grace = grace
|
mgr.first_byte_grace = grace
|
||||||
|
|
||||||
if args.no_fetch:
|
if args.no_fetch:
|
||||||
@@ -822,9 +852,10 @@ def main():
|
|||||||
if grace <= 0:
|
if grace <= 0:
|
||||||
log("growing mode: a first play seeks badly and shows no duration until "
|
log("growing mode: a first play seeks badly and shows no duration until "
|
||||||
"the mux lands")
|
"the mux lands")
|
||||||
elif grace >= args.wait_timeout:
|
elif mgr.strict:
|
||||||
log(f"strict mode: a request blocks up to {grace:g}s for a complete mux "
|
log(f"waiting for a complete mux, up to {grace:g}s. A cold long video is "
|
||||||
f"and 504s otherwise -- long videos will fail to start")
|
f"slow to start and the fetch outlives the request, so a retry is "
|
||||||
|
f"instant -- see FIRST_BYTE_GRACE for why partial is worse")
|
||||||
else:
|
else:
|
||||||
log(f"waiting up to {grace:g}s for a complete mux, then streaming the "
|
log(f"waiting up to {grace:g}s for a complete mux, then streaming the "
|
||||||
f"partial file")
|
f"partial file")
|
||||||
|
|||||||
@@ -413,3 +413,132 @@ def test_reclassifying_a_materialised_video_removes_its_files(
|
|||||||
assert stats["shorts"] == 1
|
assert stats["shorts"] == 1
|
||||||
assert videos.get(conn, "vid00000001")["state"] == videos.SKIPPED_SHORT
|
assert videos.get(conn, "vid00000001")["state"] == videos.SKIPPED_SHORT
|
||||||
assert not path.exists()
|
assert not path.exists()
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------- the min_keep_videos floor
|
||||||
|
#
|
||||||
|
# §5 defines retention as max(retention_days, min_keep_videos newest) and gives
|
||||||
|
# the reason: a channel that uploads every six weeks has nothing inside a 30-day
|
||||||
|
# window and would appear in Jellyfin as an empty series. reap.py implemented the
|
||||||
|
# second half; discovery did not, so it only protected videos that had already
|
||||||
|
# been materialised. Measured on the real 119 subscriptions: 441 episodes but 57
|
||||||
|
# channels with none, 55 of them holding skipped_old rows.
|
||||||
|
|
||||||
|
|
||||||
|
def _old(conn, channel, count, *, start=1):
|
||||||
|
"""`count` skipped_old videos, newest first at 2026-01-{start}..."""
|
||||||
|
made = []
|
||||||
|
for index in range(count):
|
||||||
|
made.append(add_video(
|
||||||
|
conn, channel["id"], f"old{index + start:08d}",
|
||||||
|
upload_date=f"2026-01-{index + start:02d}",
|
||||||
|
state=videos.SKIPPED_OLD,
|
||||||
|
))
|
||||||
|
return made
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_channel_with_nothing_in_the_window_keeps_its_newest(conn, settings, channel):
|
||||||
|
_old(conn, channel, 8)
|
||||||
|
|
||||||
|
promoted = discovery.top_up_to_min_keep(conn, settings, channel)
|
||||||
|
|
||||||
|
assert promoted == 5
|
||||||
|
listed = conn.execute(
|
||||||
|
"SELECT video_id FROM video WHERE state = ? ORDER BY upload_date DESC",
|
||||||
|
(videos.LISTED,),
|
||||||
|
).fetchall()
|
||||||
|
# The five NEWEST, not the first five found.
|
||||||
|
assert [row["video_id"] for row in listed] == [
|
||||||
|
"old00000008", "old00000007", "old00000006", "old00000005", "old00000004",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def test_the_floor_counts_what_is_already_there(conn, settings, channel):
|
||||||
|
"""A channel with 3 in-window videos needs only 2 older ones."""
|
||||||
|
for index in range(3):
|
||||||
|
add_video(conn, channel["id"], f"new{index:08d}", upload_date="2026-08-10")
|
||||||
|
_old(conn, channel, 6)
|
||||||
|
|
||||||
|
assert discovery.top_up_to_min_keep(conn, settings, channel) == 2
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_busy_channel_is_untouched(conn, settings, channel):
|
||||||
|
for index in range(9):
|
||||||
|
add_video(conn, channel["id"], f"new{index:08d}", upload_date="2026-08-10")
|
||||||
|
_old(conn, channel, 4)
|
||||||
|
|
||||||
|
assert discovery.top_up_to_min_keep(conn, settings, channel) == 0
|
||||||
|
assert conn.execute(
|
||||||
|
"SELECT count(*) FROM video WHERE state = ?", (videos.SKIPPED_OLD,)
|
||||||
|
).fetchone()[0] == 4
|
||||||
|
|
||||||
|
|
||||||
|
def test_topping_up_is_idempotent(conn, settings, channel):
|
||||||
|
"""It runs every poll, every hour. Twice must not mean ten episodes."""
|
||||||
|
_old(conn, channel, 8)
|
||||||
|
|
||||||
|
first = discovery.top_up_to_min_keep(conn, settings, channel)
|
||||||
|
second = discovery.top_up_to_min_keep(conn, settings, channel)
|
||||||
|
|
||||||
|
assert (first, second) == (5, 0)
|
||||||
|
assert conn.execute(
|
||||||
|
"SELECT count(*) FROM video WHERE state = ?", (videos.LISTED,)
|
||||||
|
).fetchone()[0] == 5
|
||||||
|
|
||||||
|
|
||||||
|
def test_aged_out_videos_are_never_revived(conn, settings, channel):
|
||||||
|
"""They were on disk and were deleted. Reviving them presents months of old
|
||||||
|
episodes to Jellyfin as new, which is what the tombstone exists to stop."""
|
||||||
|
for index in range(6):
|
||||||
|
add_video(conn, channel["id"], f"gone{index:08d}",
|
||||||
|
upload_date=f"2026-02-{index + 1:02d}", state=videos.AGED_OUT)
|
||||||
|
|
||||||
|
assert discovery.top_up_to_min_keep(conn, settings, channel) == 0
|
||||||
|
assert conn.execute(
|
||||||
|
"SELECT count(*) FROM video WHERE state = ?", (videos.AGED_OUT,)
|
||||||
|
).fetchone()[0] == 6
|
||||||
|
|
||||||
|
|
||||||
|
def test_shorts_and_livestreams_do_not_count_towards_the_floor(conn, settings, channel):
|
||||||
|
"""They are excluded by policy and can never be episodes, so a channel whose
|
||||||
|
newest uploads are all Shorts must reach further back for long-form ones."""
|
||||||
|
for index in range(4):
|
||||||
|
add_video(conn, channel["id"], f"shrt{index:08d}",
|
||||||
|
upload_date="2026-08-11", state=videos.SKIPPED_SHORT)
|
||||||
|
add_video(conn, channel["id"], "live0000001",
|
||||||
|
upload_date="2026-08-11", state=videos.SKIPPED_LIVE)
|
||||||
|
_old(conn, channel, 7)
|
||||||
|
|
||||||
|
assert discovery.top_up_to_min_keep(conn, settings, channel) == 5
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_floor_of_zero_disables_it(conn, settings, channel):
|
||||||
|
settings.set("min_keep_videos", "0")
|
||||||
|
_old(conn, channel, 6)
|
||||||
|
|
||||||
|
assert discovery.top_up_to_min_keep(conn, settings, channel) == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_a_channel_with_no_videos_at_all_stays_empty(conn, settings, channel):
|
||||||
|
"""Two of the real 119 have no long-form uploads whatsoever (their UULF
|
||||||
|
playlist 404s). There is nothing to promote and no directory should appear."""
|
||||||
|
assert discovery.top_up_to_min_keep(conn, settings, channel) == 0
|
||||||
|
|
||||||
|
|
||||||
|
def test_promoted_videos_survive_the_next_reap(conn, settings, channel):
|
||||||
|
"""The whole point: the two halves of max(window, N newest) must agree. If
|
||||||
|
reap deleted what discovery just promoted, channels would flicker hourly."""
|
||||||
|
from ytstream import reap
|
||||||
|
|
||||||
|
_old(conn, channel, 5)
|
||||||
|
discovery.top_up_to_min_keep(conn, settings, channel)
|
||||||
|
for row in conn.execute(
|
||||||
|
"SELECT video_id FROM video WHERE state = ?", (videos.LISTED,)
|
||||||
|
).fetchall():
|
||||||
|
videos.mark_materialised(
|
||||||
|
conn, row["video_id"], rel_path=f"c/{row['video_id']}.strm",
|
||||||
|
season=2026, episode=1, upload_date="2026-01-01", duration=900,
|
||||||
|
title="t",
|
||||||
|
)
|
||||||
|
|
||||||
|
assert reap.candidates(conn, settings) == []
|
||||||
|
|||||||
+27
-4
@@ -341,14 +341,37 @@ def test_growing_flag_means_no_wait_at_all(tmp_path):
|
|||||||
server.close()
|
server.close()
|
||||||
|
|
||||||
|
|
||||||
def test_status_reports_the_serving_mode_and_grace(manager_factory):
|
def test_the_default_is_to_wait_for_a_complete_file(manager_factory):
|
||||||
"""/healthz has to show this: it is the difference between 'plays' and
|
"""The default must not serve a partial file. Jellyfin responds to one by
|
||||||
'playback error', and it is otherwise invisible without the unit file."""
|
dragging a gigabyte through the proxy to probe a fragmented MP4 that has no
|
||||||
|
duration in its header, then transcoding a stream it cannot seek: measured
|
||||||
|
2026-08-13, 3 minutes to start, 6 seconds of video, permanent stall."""
|
||||||
|
assert proxy.FIRST_BYTE_GRACE == float("inf")
|
||||||
|
|
||||||
|
status = manager_factory().status()
|
||||||
|
assert status["mode"] == "wait-for-complete"
|
||||||
|
assert status["first_byte_grace_s"] is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_status_is_valid_json_with_no_grace_cap(manager_factory):
|
||||||
|
"""json.dumps emits `Infinity` for an infinite float, which is not JSON and
|
||||||
|
breaks any client that parses /healthz strictly."""
|
||||||
|
payload = json.dumps(manager_factory().status())
|
||||||
|
assert "Infinity" not in payload
|
||||||
|
assert json.loads(payload)["first_byte_grace_s"] is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_status_reports_a_finite_grace_when_one_is_set(manager_factory):
|
||||||
manager = manager_factory()
|
manager = manager_factory()
|
||||||
|
manager.strict = False
|
||||||
|
manager.first_byte_grace = 12.0
|
||||||
|
|
||||||
status = manager.status()
|
status = manager.status()
|
||||||
assert status["mode"] == "wait-then-stream"
|
assert status["mode"] == "wait-then-stream"
|
||||||
assert status["first_byte_grace_s"] == proxy.FIRST_BYTE_GRACE
|
assert status["first_byte_grace_s"] == 12.0
|
||||||
|
|
||||||
|
|
||||||
|
def test_status_reports_growing_mode(manager_factory):
|
||||||
growing = manager_factory(growing=True)
|
growing = manager_factory(growing=True)
|
||||||
assert growing.status()["mode"] == "growing"
|
assert growing.status()["mode"] == "growing"
|
||||||
assert growing.status()["first_byte_grace_s"] == 0.0
|
assert growing.status()["first_byte_grace_s"] == 0.0
|
||||||
|
|||||||
@@ -230,3 +230,37 @@ def test_summarise_is_a_single_line():
|
|||||||
assert "channels=119" in text
|
assert "channels=119" in text
|
||||||
assert "materialised=7" in text
|
assert "materialised=7" in text
|
||||||
assert "aged_out=8" in text
|
assert "aged_out=8" in text
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_prunes_channels_that_have_nothing_to_show(conn, settings, media_root,
|
||||||
|
monkeypatch):
|
||||||
|
"""End to end: a subscribed channel whose directory exists only because
|
||||||
|
subscribe() wrote tvshow.nfo must not survive a run as an empty series."""
|
||||||
|
from conftest import add_channel
|
||||||
|
from ytstream import runner
|
||||||
|
|
||||||
|
empty = add_channel(conn, "UC" + "s" * 22, "Dead Channel", "dead-channel")
|
||||||
|
tree = media_root / "dead-channel"
|
||||||
|
tree.mkdir()
|
||||||
|
(tree / "tvshow.nfo").write_text("<tvshow/>")
|
||||||
|
|
||||||
|
assert runner.prune_empty_channels(conn) == 1
|
||||||
|
assert not tree.exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_run_keeps_a_channel_with_a_materialised_video(conn, settings, media_root):
|
||||||
|
from conftest import add_channel, add_video
|
||||||
|
from ytstream import runner, videos
|
||||||
|
|
||||||
|
row = add_channel(conn, "UC" + "t" * 22, "Alive", "alive")
|
||||||
|
tree = media_root / "alive"
|
||||||
|
(tree / "Season 2026").mkdir(parents=True)
|
||||||
|
(tree / "Season 2026" / "ep.strm").write_text("url")
|
||||||
|
video = add_video(conn, row["id"], "vid00000009")
|
||||||
|
videos.mark_materialised(conn, "vid00000009",
|
||||||
|
rel_path="alive/Season 2026/ep.strm", season=2026,
|
||||||
|
episode=1, upload_date="2026-08-01", duration=900,
|
||||||
|
title="t")
|
||||||
|
|
||||||
|
assert runner.prune_empty_channels(conn) == 0
|
||||||
|
assert tree.exists()
|
||||||
|
|||||||
@@ -310,3 +310,55 @@ def test_a_renamed_episode_removes_its_old_files(conn, settings, media_root,
|
|||||||
assert not old.exists()
|
assert not old.exists()
|
||||||
assert (media_root / second["rel_path"]).exists()
|
assert (media_root / second["rel_path"]).exists()
|
||||||
assert len(list(media_root.rglob("*.strm"))) == 1
|
assert len(list(media_root.rglob("*.strm"))) == 1
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------- pruning channels with nothing to show
|
||||||
|
#
|
||||||
|
# §5: a channel with nothing inside the window must not leave an empty series in
|
||||||
|
# Jellyfin. runner creates directories lazily to honour that, but
|
||||||
|
# channels.subscribe() mkdirs to write tvshow.nfo and a poster, so every
|
||||||
|
# subscription got one regardless. Measured after approving all 119 real
|
||||||
|
# subscriptions: 57 empty series.
|
||||||
|
|
||||||
|
|
||||||
|
def test_prune_removes_a_channel_with_no_episodes(conn, media_root, channel):
|
||||||
|
tree = media_root / "clabretro"
|
||||||
|
tree.mkdir()
|
||||||
|
(tree / "tvshow.nfo").write_text("<tvshow/>")
|
||||||
|
(tree / "poster.jpg").write_bytes(b"x" * 100)
|
||||||
|
|
||||||
|
assert strm.prune_if_no_episodes(channel) is True
|
||||||
|
assert not tree.exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_prune_leaves_a_channel_that_has_episodes(conn, settings, media_root,
|
||||||
|
channel, video, no_thumbs):
|
||||||
|
strm.materialise(conn, settings, channel, video)
|
||||||
|
|
||||||
|
assert strm.prune_if_no_episodes(channel) is False
|
||||||
|
assert (media_root / "clabretro").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_prune_finds_episodes_in_any_season(conn, media_root, channel):
|
||||||
|
"""The .strm lives a directory down, so a shallow check would delete it."""
|
||||||
|
deep = media_root / "clabretro" / "Season 2019"
|
||||||
|
deep.mkdir(parents=True)
|
||||||
|
(deep / "clabretro - S2019E1010 - Old [dQw4w9WgXcQ].strm").write_text("url")
|
||||||
|
|
||||||
|
assert strm.prune_if_no_episodes(channel) is False
|
||||||
|
assert deep.exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_prune_is_a_no_op_when_there_is_no_directory(conn, channel):
|
||||||
|
assert strm.prune_if_no_episodes(channel) is False
|
||||||
|
|
||||||
|
|
||||||
|
def test_prune_refuses_a_blank_channel_directory(conn, media_root):
|
||||||
|
from conftest import add_channel
|
||||||
|
|
||||||
|
row = add_channel(conn, "UC" + "r" * 22, "Blank", "")
|
||||||
|
keep = media_root / "keep"
|
||||||
|
keep.mkdir()
|
||||||
|
|
||||||
|
assert strm.prune_if_no_episodes(row) is False
|
||||||
|
assert keep.exists()
|
||||||
|
|||||||
+68
-2
@@ -424,6 +424,62 @@ def rescan_channel(
|
|||||||
return cursor.rowcount
|
return cursor.rowcount
|
||||||
|
|
||||||
|
|
||||||
|
def top_up_to_min_keep(
|
||||||
|
conn: sqlite3.Connection, settings: Settings, channel: sqlite3.Row
|
||||||
|
) -> int:
|
||||||
|
"""Promote the newest `skipped_old` rows until the channel has `min_keep_videos`
|
||||||
|
videos that will end up on disk. Returns how many were promoted.
|
||||||
|
|
||||||
|
§5 defines retention as **`max(retention_days, min_keep_videos newest)`**, and
|
||||||
|
`reap.candidates()` implements the second half — but only for videos that were
|
||||||
|
materialised in the first place. Discovery judges every video against the
|
||||||
|
window alone and tombstones the rest as `skipped_old`, so a channel that has
|
||||||
|
not uploaded inside the window materialises *nothing* and appears in Jellyfin
|
||||||
|
as an empty series. That is the precise outcome §5 introduced this setting to
|
||||||
|
prevent: "keeps its last 5 videos permanently visible instead of showing an
|
||||||
|
empty shelf".
|
||||||
|
|
||||||
|
Measured 2026-08-13 on the real 119 subscriptions: 441 videos materialised —
|
||||||
|
exactly the in-window count Phase 0 predicted — but **57 channels had zero
|
||||||
|
episodes**, and 55 of those were sitting on `skipped_old` rows. §5's own cost
|
||||||
|
estimate was `441 + (52 x 5) ~= 700`, so the shortfall was in the code, not in
|
||||||
|
the measurement.
|
||||||
|
|
||||||
|
Only `materialised` and `listed` count towards the floor: `skipped_short` and
|
||||||
|
`skipped_live` are excluded by policy and can never be episodes, so a channel
|
||||||
|
whose five newest uploads are all Shorts correctly reaches further back for
|
||||||
|
long-form ones.
|
||||||
|
|
||||||
|
`aged_out` rows are deliberately not revived, for the reason `rescan_channel`
|
||||||
|
gives. reap() protects the newest `min_keep_videos` on disk, so anything
|
||||||
|
promoted here sits inside its protected set and will not be deleted straight
|
||||||
|
back out — the two halves of the rule now agree.
|
||||||
|
"""
|
||||||
|
keep = max(0, settings.get_int("min_keep_videos"))
|
||||||
|
if not keep:
|
||||||
|
return 0
|
||||||
|
|
||||||
|
have = conn.execute(
|
||||||
|
"SELECT count(*) FROM video WHERE channel_pk = ? AND state IN (?, ?)",
|
||||||
|
(channel["id"], videos.MATERIALISED, videos.LISTED),
|
||||||
|
).fetchone()[0]
|
||||||
|
shortfall = keep - have
|
||||||
|
if shortfall <= 0:
|
||||||
|
return 0
|
||||||
|
|
||||||
|
with conn:
|
||||||
|
cursor = conn.execute(
|
||||||
|
"UPDATE video SET state = ? WHERE id IN ("
|
||||||
|
" SELECT id FROM video WHERE channel_pk = ? AND state = ?"
|
||||||
|
" ORDER BY upload_date DESC, id DESC LIMIT ?)",
|
||||||
|
(videos.LISTED, channel["id"], videos.SKIPPED_OLD, shortfall),
|
||||||
|
)
|
||||||
|
if cursor.rowcount:
|
||||||
|
log.info("%s: kept %d older video(s) to reach the %d-video floor",
|
||||||
|
channel["title"], cursor.rowcount, keep)
|
||||||
|
return cursor.rowcount
|
||||||
|
|
||||||
|
|
||||||
def poll_all(
|
def poll_all(
|
||||||
conn: sqlite3.Connection, settings: Settings, channel_pk: int | None = None
|
conn: sqlite3.Connection, settings: Settings, channel_pk: int | None = None
|
||||||
) -> dict:
|
) -> dict:
|
||||||
@@ -434,7 +490,7 @@ def poll_all(
|
|||||||
rows = channels.all_channels(conn)
|
rows = channels.all_channels(conn)
|
||||||
|
|
||||||
totals = {"channels": 0, "queued": 0, "old": 0, "known": 0, "repaired": 0,
|
totals = {"channels": 0, "queued": 0, "old": 0, "known": 0, "repaired": 0,
|
||||||
"titled": 0, "shorts": 0, "live": 0, "failed": 0}
|
"titled": 0, "shorts": 0, "live": 0, "kept": 0, "failed": 0}
|
||||||
for channel in rows:
|
for channel in rows:
|
||||||
totals["channels"] += 1
|
totals["channels"] += 1
|
||||||
if not channel["backfilled"]:
|
if not channel["backfilled"]:
|
||||||
@@ -450,7 +506,17 @@ def poll_all(
|
|||||||
|
|
||||||
if "error" in stats:
|
if "error" in stats:
|
||||||
totals["failed"] += 1
|
totals["failed"] += 1
|
||||||
for key in ("queued", "old", "known", "repaired", "titled", "shorts", "live"):
|
else:
|
||||||
|
# After the poll, so it sees this run's discoveries and only reaches
|
||||||
|
# back for older videos when the window genuinely left the channel
|
||||||
|
# short. Skipped on a failed poll: a channel we could not read looks
|
||||||
|
# empty, and topping it up from tombstones would be guessing.
|
||||||
|
kept = top_up_to_min_keep(conn, settings, channel)
|
||||||
|
if kept:
|
||||||
|
stats["kept"] = kept
|
||||||
|
|
||||||
|
for key in ("queued", "old", "known", "repaired", "titled", "shorts",
|
||||||
|
"live", "kept"):
|
||||||
totals[key] += stats.get(key, 0)
|
totals[key] += stats.get(key, 0)
|
||||||
log.info("%s: %s", channel["title"], stats)
|
log.info("%s: %s", channel["title"], stats)
|
||||||
return totals
|
return totals
|
||||||
|
|||||||
@@ -80,6 +80,23 @@ def materialise_all(
|
|||||||
return stats
|
return stats
|
||||||
|
|
||||||
|
|
||||||
|
def prune_empty_channels(conn: sqlite3.Connection) -> int:
|
||||||
|
"""Remove directories for subscribed channels that have no episodes to show.
|
||||||
|
|
||||||
|
Runs after materialising and reaping, so it only sees the settled state. See
|
||||||
|
`strm.prune_if_no_episodes` for why these directories exist at all.
|
||||||
|
"""
|
||||||
|
pruned = 0
|
||||||
|
for channel in conn.execute(
|
||||||
|
"SELECT c.* FROM channel c WHERE NOT EXISTS ("
|
||||||
|
" SELECT 1 FROM video v WHERE v.channel_pk = c.id AND v.state = ?)",
|
||||||
|
(videos.MATERIALISED,),
|
||||||
|
).fetchall():
|
||||||
|
if strm.prune_if_no_episodes(channel):
|
||||||
|
pruned += 1
|
||||||
|
return pruned
|
||||||
|
|
||||||
|
|
||||||
def run(
|
def run(
|
||||||
conn: sqlite3.Connection, settings: Settings, channel_pk: int | None = None
|
conn: sqlite3.Connection, settings: Settings, channel_pk: int | None = None
|
||||||
) -> dict:
|
) -> dict:
|
||||||
@@ -94,6 +111,7 @@ def run(
|
|||||||
result["poll"] = discovery.poll_all(conn, settings, channel_pk)
|
result["poll"] = discovery.poll_all(conn, settings, channel_pk)
|
||||||
result["materialise"] = materialise_all(conn, settings)
|
result["materialise"] = materialise_all(conn, settings)
|
||||||
result["reap"] = reap.run(conn, settings)
|
result["reap"] = reap.run(conn, settings)
|
||||||
|
result["pruned"] = prune_empty_channels(conn)
|
||||||
|
|
||||||
if result["materialise"]["materialised"] or result["reap"]["aged_out"]:
|
if result["materialise"]["materialised"] or result["reap"]["aged_out"]:
|
||||||
jellyfin.from_settings(settings).refresh()
|
jellyfin.from_settings(settings).refresh()
|
||||||
@@ -124,6 +142,10 @@ def summarise(result: dict) -> str:
|
|||||||
]
|
]
|
||||||
if sync.get("refused"):
|
if sync.get("refused"):
|
||||||
parts.append(f"SYNC_REFUSED={sync['refused']}")
|
parts.append(f"SYNC_REFUSED={sync['refused']}")
|
||||||
|
if poll.get("kept"):
|
||||||
|
parts.append(f"kept_for_floor={poll['kept']}")
|
||||||
|
if result.get("pruned"):
|
||||||
|
parts.append(f"pruned_empty={result['pruned']}")
|
||||||
if made.get("errors"):
|
if made.get("errors"):
|
||||||
parts.append(f"errors={made['errors']}")
|
parts.append(f"errors={made['errors']}")
|
||||||
return " ".join(parts)
|
return " ".join(parts)
|
||||||
|
|||||||
@@ -207,6 +207,39 @@ def remove(video: sqlite3.Row) -> int:
|
|||||||
return removed
|
return removed
|
||||||
|
|
||||||
|
|
||||||
|
def prune_if_no_episodes(channel: sqlite3.Row) -> bool:
|
||||||
|
"""Remove a subscribed channel's directory if it holds no episodes.
|
||||||
|
|
||||||
|
§5 says a channel with nothing inside the window must not leave an empty
|
||||||
|
series in Jellyfin, and `runner.materialise_pending` honours that by creating
|
||||||
|
the directory lazily, from the first episode. But `channels.subscribe()` calls
|
||||||
|
`_write_show_metadata()`, which mkdirs to write tvshow.nfo and the poster —
|
||||||
|
so every subscription got a directory whether or not it had anything to show.
|
||||||
|
Measured 2026-08-13 after approving all 119 subscriptions: **57 empty series**.
|
||||||
|
|
||||||
|
`top_up_to_min_keep()` fixes 55 of those by keeping older videos. This handles
|
||||||
|
the remainder — the channels with no long-form uploads at all, whose UULF
|
||||||
|
playlist 404s (2 of the real 119). It runs after materialising, so "no
|
||||||
|
episodes" means none were written this run either.
|
||||||
|
|
||||||
|
Only ever removes a directory with no `.strm` in it, so an active channel is
|
||||||
|
never touched. `subscribe()` runs once per channel, so a pruned directory does
|
||||||
|
not come back every hour — it reappears only when the channel finally uploads
|
||||||
|
something and `write_show()` recreates it.
|
||||||
|
"""
|
||||||
|
directory = channel_dir(channel)
|
||||||
|
if directory == config.MEDIA_ROOT or not str(channel["dir_name"]).strip():
|
||||||
|
log.error("refusing to prune %s: unsafe channel directory", directory)
|
||||||
|
return False
|
||||||
|
if not directory.is_dir():
|
||||||
|
return False
|
||||||
|
if any(directory.rglob("*.strm")):
|
||||||
|
return False
|
||||||
|
shutil.rmtree(directory)
|
||||||
|
log.info("pruned %s: subscribed but no episodes to show", directory)
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
def remove_channel_tree(channel: sqlite3.Row) -> bool:
|
def remove_channel_tree(channel: sqlite3.Row) -> bool:
|
||||||
"""Delete a whole channel directory, on unsubscribe.
|
"""Delete a whole channel directory, on unsubscribe.
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user