Measure parallel fetching: it does not help

The ceiling is the internet connection at ~8 MB/s. Parallel chunks buy 10-30%, concurrency across different videos buys nothing, and --concurrent-fragments is inapplicable because these formats carry no fragments at all. yt-dlp is already doing the thing that matters: the formats advertise http_chunk_size=10485760 and that chunking is worth 13x, because a single long range request gets throttled to 0.60 MB/s. Also tested option 3 while here: NFO <streamdetails> is ignored for episodes -- RunTimeTicks stays null and MediaStreams empty -- so Jellyfin cannot be talked out of transcoding that way either. The useful number: download is 8 MB/s against a 0.39 MB/s playback bitrate, 20x headroom. Streaming while downloading was never bandwidth bound; it failed purely on Jellyfin's probe decision. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Never serve a partial file: it makes Jellyfin transcode
2026-08-13 12:08:33 +01:00 · 2026-08-13 11:32:12 +01:00 · 2026-08-13 11:10:24 +01:00 · 2026-08-13 11:05:28 +01:00 · 2026-08-13 11:00:47 +01:00 · 2026-08-13 10:53:48 +01:00
25 changed files with 2186 additions and 118 deletions
@@ -1,5 +1,9 @@
 #!/bin/bash
-# Everything that does NOT need root. Run as susan, before deploy.sh.
+# Everything that does NOT need root: the venv, yt-dlp, the POT container, the
 # database.
 #
 # Normally invoked for you by deploy.sh, which creates the state directory first.
 # Safe to run directly as susan afterwards — for instance to rebuild the venv:
 #
 #     /opt/ytstream/deploy/bootstrap.sh
 set -euo pipefail
@@ -11,8 +15,12 @@ REPO=/opt/ytstream
 say() { printf '\n\033[1m==> %s\033[0m\n' "$1"; }
 if [[ ! -d $STATE ]]; then
-    echo "$STATE does not exist yet — run 'sudo $REPO/deploy/deploy.sh' first," >&2
+    echo "$STATE does not exist. Run 'sudo $REPO/deploy/deploy.sh' — it creates the" >&2
-    echo "or create it with: sudo install -d -o susan -g automation -m 0770 $STATE" >&2
+    echo "directory and then calls this script." >&2
    exit 1
 fi
 if [[ ! -w $STATE ]]; then
    echo "$STATE is not writable by $(id -un). Expected mode 0770 susan:automation." >&2
    exit 1
 fi
@@ -32,6 +40,40 @@ print("POT plugin:", "MISSING" if missing else "present")
 raise SystemExit(1 if missing else 0)
 PY
 say "Installing the Deno JS runtime"
 # MANDATORY, not optional. Without a JS runtime yt-dlp cannot solve the `n`
 # challenge, and youtube-automate measured the consequence on this machine: 22
 # formats instead of 29, and throttled downloads. yt-dlp finds it by PATH, and the
 # unit pins PATH to this venv's bin, so it has to live here rather than anywhere
 # more natural.
 if [[ -x $VENV/bin/deno ]]; then
    echo "    already present: $("$VENV/bin/deno" --version | head -1)"
 elif [[ -x /var/lib/youtube-automate/venv/bin/deno ]]; then
    # Same machine, same architecture, already verified working. Prefer it over a
    # download while youtube-automate is still installed.
    install -m 0755 /var/lib/youtube-automate/venv/bin/deno "$VENV/bin/deno"
    echo "    copied from the youtube-automate venv: $("$VENV/bin/deno" --version | head -1)"
 else
    tmp=$(mktemp -d)
    curl -fsSL -o "$tmp/deno.zip" \
        https://github.com/denoland/deno/releases/latest/download/deno-x86_64-unknown-linux-gnu.zip
    unzip -q "$tmp/deno.zip" -d "$tmp"
    install -m 0755 "$tmp/deno" "$VENV/bin/deno"
    rm -rf "$tmp"
    echo "    installed $("$VENV/bin/deno" --version | head -1)"
 fi
 say "Confirming yt-dlp can solve JS challenges"
 # All three lines must appear, per youtube-automate specs.md §3.
 if "$VENV/bin/yt-dlp" -v --simulate --no-warnings \
        "https://www.youtube.com/watch?v=dQw4w9WgXcQ" 2>&1 \
        | grep -q "JS runtimes: deno"; then
    echo "    yt-dlp reports the deno runtime"
 else
    echo "    WARNING: yt-dlp did not report a deno JS runtime — playback may be" >&2
    echo "    throttled or missing formats. Check: ytstream doctor" >&2
 fi
 say "Starting the POT provider container if it is not already up"
 if ! curl -sf --max-time 5 http://127.0.0.1:4416/ping >/dev/null 2>&1; then
    docker run -d --name bgutil-provider --restart unless-stopped \
@@ -43,4 +85,4 @@ fi
 say "Initialising the database"
 "$VENV/bin/ytstream" status || true
-say "Done — now run 'sudo $REPO/deploy/deploy.sh'"
+say "Virtualenv ready at $VENV"
@@ -1,15 +1,27 @@
-# ytstream — add these to susan's crontab (`crontab -e`).
+# ytstream — these are INSTALLED in susan's crontab as of 2026-08-13.
 # Kept here as the record of what should be there (`crontab -l` to confirm).
 #
 # HC_API_URL is already set at the top of susan's crontab; these follow the
 # existing one-UUID-per-job convention. `sg mediaserver` guarantees new files are
 # group-owned by mediaserver even if the invoking shell's primary group differs.
 #
-# Get fresh UUIDs from hc.jihakuz.xyz before enabling these.
+# The two UUIDs are INHERITED from youtube-automate's entries rather than freshly
 # created, and the schedules are unchanged for the same reason: a healthchecks
 # check may be configured with a cron expression rather than a simple period, so
 # moving :17 to :23 could have alerted on a job that ran perfectly. Inheriting
 # also keeps the ping history continuous across the handover. The checks are still
 # *named* after youtube-automate in the hc.jihakuz.xyz UI — rename them there, it
 # has no effect on anything here.
-# Sync subscriptions, poll, materialise, reap. Hourly. Runs at :23 to stay clear
+# Sync subscriptions, poll, materialise, reap. Hourly at :17.
-# of youtube-automate's :17 entry during the overlap period.
+# Measured 2026-08-13: a full run over 11 channels takes ~8s.
-23 * * * * runitor -uuid REPLACE-WITH-UUID -- sg mediaserver "/usr/local/bin/ytstream run"
+17 * * * * runitor -uuid 41a4d61a-7743-43d9-9b5d-d37d536e4726 -- sg mediaserver "/usr/local/bin/ytstream run"
 # Keep yt-dlp current — this is the thing that keeps playback working as YouTube
-# changes. Mondays 04:50, after youtube-automate's 04:40 slot.
+# changes. Mondays 04:40.
-50 4 * * 1 runitor -uuid REPLACE-WITH-UUID -- /opt/ytstream/deploy/update-ytdlp.sh
+#
 # The script tries to restart ytstream-proxy and cannot, because cron runs it as
 # susan. That is harmless: the proxy shells out to `yt-dlp` per request, so a pip
 # upgrade takes effect on the next fetch with no restart. Leaving the attempt in
 # means the message shows up in the healthchecks output if it ever does matter.
 40 4 * * 1 runitor -uuid 721e4cf0-d796-48e7-a184-79d21e1ba373 -- /opt/ytstream/deploy/update-ytdlp.sh
@@ -0,0 +1,91 @@
 #!/bin/bash
 # Retire youtube-automate and hand tube.jihakuz.xyz to ytstream.
 #
 #     sudo /opt/ytstream/deploy/decommission.sh
 #
 # This is plan.md §12 steps 4 and 5 — the two that need root. Everything it does
 # is reversible: the nginx change is backed up next to the original, and the
 # service is disabled rather than removed.
 #
 # Deliberately NOT done here, because each one destroys something:
 #   * deleting /disks/Plex/YouTube (§12 step 6)
 #   * removing or renaming the Jellyfin libraries (§12 step 3)
 #   * removing /opt/youtube-automate, its repo, subs.db or specs.md (§12 step 7)
 set -euo pipefail
 CONF=/etc/nginx/sites-available/jihakuz.xyz
 OLD_PORT=8085          # youtube-automate
 NEW_PORT=8086          # ytstream admin
 DB=/var/lib/ytstream/ytstream.db
 if [[ $EUID -ne 0 ]]; then
    echo "This script needs root. Run: sudo $0" >&2
    exit 1
 fi
 # ---------------------------------------------------------------- preflight
 # Repointing a public hostname at a service that cannot be logged into wastes an
 # evening working out why. The admin UI fails closed with no password set: every
 # login attempt is rejected, including your brother's.
 if ! sqlite3 "$DB" \
        "SELECT 1 FROM setting WHERE key='admin_password_hash' AND value <> '';" \
        2>/dev/null | grep -q 1; then
    echo "!! No admin password is set, so nobody can log in to the UI this would" >&2
    echo "   expose. Run this first, then re-run me:" >&2
    echo "       sudo -u susan /var/lib/ytstream/venv/bin/ytstream set-password" >&2
    exit 1
 fi
 # And do not hand the hostname to a port nothing is listening on.
 code=$(curl -s -o /dev/null -w '%{http_code}' -m 10 "http://127.0.0.1:${NEW_PORT}/" || true)
 if [[ "$code" != "200" && "$code" != "303" && "$code" != "302" ]]; then
    echo "!! ytstream-admin is not answering on ${NEW_PORT} (got '${code}')." >&2
    echo "   systemctl status ytstream-admin" >&2
    exit 1
 fi
 echo "==> ytstream-admin answers on ${NEW_PORT} (HTTP ${code}) and has a password set"
 # ------------------------------------------------------------------- nginx
 if grep -q "127.0.0.1:${NEW_PORT}" "$CONF"; then
    echo "==> nginx already points tube.jihakuz.xyz at ${NEW_PORT}; skipping"
 elif ! grep -q "127.0.0.1:${OLD_PORT}" "$CONF"; then
    echo "!! Found neither ${OLD_PORT} nor ${NEW_PORT} in $CONF. Not guessing." >&2
    exit 1
 else
    BACKUP="${CONF}.bak-$(date +%Y%m%d%H%M%S)"
    cp -a "$CONF" "$BACKUP"
    echo "==> Backed up $CONF to $BACKUP"
    # One line. The proxy_set_header block the youtube-automate fix added is
    # already there and still correct — X-Forwarded-For in particular, because
    # the login throttle keys on it and without it one attacker locks out all.
    sed -i "s|proxy_pass http://127.0.0.1:${OLD_PORT};|proxy_pass http://127.0.0.1:${NEW_PORT};|" "$CONF"
    echo "==> Repointed tube.jihakuz.xyz at 127.0.0.1:${NEW_PORT}"
    if ! nginx -t; then
        echo "!! nginx config test failed — restoring the backup" >&2
        cp -a "$BACKUP" "$CONF"
        nginx -t
        exit 1
    fi
    systemctl reload nginx
    echo "==> nginx reloaded"
 fi
 # ---------------------------------------------------------------- the service
 if systemctl is-enabled --quiet youtube-automate.service 2>/dev/null \
   || systemctl is-active --quiet youtube-automate.service 2>/dev/null; then
    systemctl disable --now youtube-automate.service
    echo "==> Stopped and disabled youtube-automate.service"
    echo "    Unit left in place at /etc/systemd/system/ — 'systemctl enable --now"
    echo "    youtube-automate' brings it back if this turns out to be premature."
 else
    echo "==> youtube-automate.service already stopped and disabled"
 fi
 echo
 echo "Done. tube.jihakuz.xyz now serves ytstream's admin UI."
 echo "Still holding, on purpose:"
 echo "  * /disks/Plex/YouTube  ($(du -sh /disks/Plex/YouTube 2>/dev/null | cut -f1)) — the old downloads"
 echo "  * both Jellyfin libraries — remove/rename by hand when you are ready"
 echo "  * /opt/youtube-automate, its repo, subs.db and specs.md — keep these"
@@ -6,14 +6,21 @@
 #
 #     sudo /opt/ytstream/deploy/deploy.sh
 #
-# Everything that does NOT need root — the venv, the database, the POT provider
+# This is the only command needed. It creates the state directory, builds the venv
-# container, subscriptions, the API key — is handled by `bootstrap.sh` and the
+# by calling bootstrap.sh as the service user, and only then installs and starts
-# application itself. Run bootstrap.sh (as susan) first.
+# the units.
 #
 # That ordering is not cosmetic. An earlier version installed the units first and
 # told the operator to run bootstrap.sh separately — but bootstrap.sh needs the
 # state directory, which only this script can create, so neither could go first.
 # The units started, failed 203/EXEC against a venv that did not exist yet, and
 # restart-looped until the venv appeared.
 set -euo pipefail
 REPO=/opt/ytstream
 STATE=/var/lib/ytstream
 VENV=$STATE/venv
 SERVICE_USER=susan
 HOSTNAME_=tube.jihakuz.xyz
 if [[ $EUID -ne 0 ]]; then
@@ -26,7 +33,23 @@ say() { printf '\n\033[1m==> %s\033[0m\n' "$1"; }
 say "Creating $STATE"
 # root-owned directory, group-writable by `automation` so susan's cron job and the
 # admin server can both write the database.
-install -d -o susan -g automation -m 0770 "$STATE"
+install -d -o "$SERVICE_USER" -g automation -m 0770 "$STATE"
 say "Building the virtualenv"
 if [[ -x $VENV/bin/ytstream ]]; then
    echo "    already present at $VENV"
 else
    # As the service user, so the venv is not left root-owned. Absolute path:
    # runuser lives in /sbin, which is not always on the invoking PATH.
    /sbin/runuser -u "$SERVICE_USER" -- bash "$REPO/deploy/bootstrap.sh"
 fi
 # Refuse to start units that cannot possibly work. Restart=always would otherwise
 # turn a missing venv into a restart loop in the journal.
 if [[ ! -x $VENV/bin/ytstream ]]; then
    echo "The virtualenv was not built. Fix that, then re-run $0." >&2
    exit 1
 fi
 say "Installing the /usr/local/bin shim"
 cat > /usr/local/bin/ytstream <<EOF
@@ -0,0 +1,174 @@
 #!/usr/bin/env python3
 """Retire the old *YouTube* Jellyfin library and rename ytstream's to take its name.
    sudo -u susan /opt/ytstream/deploy/retire-jellyfin-library.py          # show the plan
    sudo -u susan /opt/ytstream/deploy/retire-jellyfin-library.py --yes    # do it
 plan.md §12 step 3. This is the one decommission step that throws something away:
 removing a library discards Jellyfin's own state for those items — watch history,
 resume positions, favourites. The *files* are untouched, and re-adding the library
 at the same path returns the same ItemId and reuses the old items (measured in the
 PoC), so the loss is bounded, but it is a loss.
 Libraries are matched by PATH, never by name, and the name to delete is read back
 from the API rather than assumed. Deleting by a guessed name is how you remove the
 wrong library: the endpoint takes a name, matches loosely on some versions, and
 "YouTube" is a prefix of "YouTube (stream)".
 """
 from __future__ import annotations
 import argparse
 import json
 import sqlite3
 import sys
 import urllib.error
 import urllib.parse
 import urllib.request
 BASE = "http://127.0.0.1:8096"
 DB = "/var/lib/ytstream/ytstream.db"
 OLD_PATH = "/disks/Plex/YouTube"      # youtube-automate's downloads
 NEW_PATH = "/disks/Plex/_ytstream"    # ytstream's .strm tree
 FINAL_NAME = "YouTube"
 def api_key() -> str:
    try:
        with sqlite3.connect(f"file:{DB}?mode=ro", uri=True) as conn:
            row = conn.execute(
                "SELECT value FROM setting WHERE key='jellyfin_api_key'").fetchone()
    except sqlite3.Error as exc:
        sys.exit(f"cannot read {DB}: {exc}\nRun me as susan or root.")
    if not row or not row[0]:
        sys.exit("no jellyfin_api_key in the settings table")
    return row[0]
 def request(key: str, method: str, path: str, params: dict | None = None):
    url = BASE + path
    if params:
        url += "?" + urllib.parse.urlencode(params)
    req = urllib.request.Request(
        url, method=method, headers={"X-Emby-Token": key, "Accept": "application/json"})
    try:
        with urllib.request.urlopen(req, timeout=60) as response:
            body = response.read()
    except urllib.error.HTTPError as exc:
        sys.exit(f"{method} {path} -> HTTP {exc.code}: {exc.read()[:200]!r}")
    except OSError as exc:
        sys.exit(f"{method} {path} -> {exc}\nIs Jellyfin running?")
    return json.loads(body) if body else None
 def folders(key: str) -> list[dict]:
    return request(key, "GET", "/Library/VirtualFolders") or []
 def by_path(all_folders: list[dict], target: str) -> dict | None:
    target = target.rstrip("/")
    for folder in all_folders:
        for location in folder.get("Locations") or []:
            if str(location).rstrip("/") == target:
                return folder
    return None
 def episode_count(key: str, library: dict) -> int:
    result = request(key, "GET", "/Items", {
        "parentId": library["ItemId"],
        "includeItemTypes": "Episode",
        "recursive": "true",
        "limit": "0",
    })
    return (result or {}).get("TotalRecordCount", 0)
 def main() -> None:
    ap = argparse.ArgumentParser(description=__doc__,
                                 formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("--yes", action="store_true", help="actually make the changes")
    args = ap.parse_args()
    key = api_key()
    current = folders(key)
    old = by_path(current, OLD_PATH)
    new = by_path(current, NEW_PATH)
    if new is None:
        sys.exit(f"no Jellyfin library covers {NEW_PATH}. Refusing to touch anything "
                 f"-- retiring the old library without a working new one would leave "
                 f"no YouTube library at all.")
    # A healthy replacement is the whole precondition. An empty new library means
    # the scan never ran or the tree is unreadable, and removing the old one then
    # would leave nothing to watch.
    episodes = episode_count(key, new)
    print(f"ytstream library  {new['Name']!r} at {NEW_PATH}: {episodes} episodes")
    if episodes == 0:
        sys.exit("the ytstream library has 0 episodes -- fix that first "
                 "(`ytstream materialise --all`, then a Jellyfin scan)")
    if old is None:
        print(f"old library       none found at {OLD_PATH} (already retired)")
    else:
        print(f"old library       {old['Name']!r} at {OLD_PATH}: "
              f"{episode_count(key, old)} episodes")
    renaming = new["Name"] != FINAL_NAME
    print()
    print("Planned changes:")
    if old is not None:
        print(f"  1. DELETE library {old['Name']!r}  (files in {OLD_PATH} untouched)")
    if renaming:
        print(f"  {'2' if old is not None else '1'}. RENAME {new['Name']!r} -> {FINAL_NAME!r}")
    if old is None and not renaming:
        print("  nothing to do.")
        return
    print()
    print("Jellyfin's watch history and resume positions for the deleted library go")
    print("with it. The video files do not.")
    if not args.yes:
        print()
        print("Dry run. Re-run with --yes to apply.")
        return
    if old is not None:
        # Name read back from the API, not guessed. refreshLibrary=false: a scan
        # here would be pointless work and, on the ytstream side, unwanted traffic.
        request(key, "DELETE", "/Library/VirtualFolders",
                {"name": old["Name"], "refreshLibrary": "false"})
        print(f"==> deleted {old['Name']!r}")
    if renaming:
        # `name`, NOT `id`. This endpoint is the odd one out -- most of
        # /Library/VirtualFolders/* takes an id, and passing one here returns a
        # bare "HTTP 400: Error processing request." that says nothing about why.
        # Verified against Jellyfin 10.11.4 on 2026-08-13: name -> 204.
        request(key, "POST", "/Library/VirtualFolders/Name",
                {"name": new["Name"], "newName": FINAL_NAME})
        print(f"==> renamed {new['Name']!r} to {FINAL_NAME!r}")
        # A library's ItemId is derived from its name, so renaming re-ids it --
        # measured, 98e74a0c… became 34f331a8…, which was the *deleted* library's
        # id, because that one had this name. Nothing here caches an ItemId, and
        # ytstream's own find_library() matches on path, so both survive it. Any
        # future caller that stores an ItemId will not.
        print("    note: the library's ItemId changed (Jellyfin derives it from "
              "the name)")
    after = folders(key)
    print()
    print("Now:")
    for folder in after:
        locations = folder.get("Locations") or []
        if any(str(p).rstrip("/") in (OLD_PATH, NEW_PATH) for p in locations):
            print(f"  {folder['Name']!r}  {locations}")
    if by_path(after, OLD_PATH) is None and old is not None:
        print(f"\n{OLD_PATH} is no longer a library. The files are still there:")
        print(f"  rm -rf {OLD_PATH}    # when you are ready")
 if __name__ == "__main__":
    main()
@@ -21,6 +21,16 @@ UMask=0002
 Environment=PATH=/var/lib/ytstream/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
 Environment=PYTHONUNBUFFERED=1
 # No --first-byte-grace: requests wait for the COMPLETE mux, bounded by
 # --wait-timeout. Do not add a grace here without reading FIRST_BYTE_GRACE in the
 # proxy. Short version, both measured 2026-08-13: waiting sends nothing for ~80s
 # on a 46-minute upload and the client gives up, but serving the partial file
 # instead makes Jellyfin drag a gigabyte through this proxy to probe a fragmented
 # MP4 that has no duration in its header, and then TRANSCODE a stream it cannot
 # seek -- 3 minutes to start, 6 seconds of video, then a permanent stall.
 # A slow cold start is the better failure: the fetch outlives the request, so the
 # retry is instant. --wait-timeout is 600 to cover a 2-hour upload (2.4 GB).
 WorkingDirectory=/opt/ytstream
 ExecStart=/var/lib/ytstream/venv/bin/python3 /opt/ytstream/proxy/ytstream_proxy.py \
    --host 127.0.0.1 --port 8099 \
@@ -28,7 +38,8 @@ ExecStart=/var/lib/ytstream/venv/bin/python3 /opt/ytstream/proxy/ytstream_proxy.
    --cache-gb 8 \
    --max-pipelines 2 \
    --max-retries 2 \
-    --max-starts 20 --starts-window 3600
+    --max-starts 20 --starts-window 3600 \
    --wait-timeout 600
 Restart=always
 RestartSec=5
@@ -1,13 +1,17 @@
 # `ytstream` — implementation plan
 **Target machine:** `susan`
-**Status:** streaming PoC verified end to end against real videos and real Jellyfin — every
+**Status:** **deployed and running.** Both systemd units are installed and active, 10 of the 119
-measurement behind this plan is written up in **`FINDINGS.md`** alongside this file. Nothing is
+mirrored channels are approved, and 251 episodes are live in Jellyfin with correct metadata and
-installed as a service yet. This document is the plan for turning it into one.
+verified DirectPlay. 345 tests pass. What remains is the cron entries, curating the rest of the
 subscription list, and the cut-over in §12.
-**Repo:** `/opt/ytstream`, pushed to `/disks/git-repos/ytstream.git`, branch `main`. The PoC code
+The streaming PoC measurements this plan was designed around are in **`FINDINGS.md`** alongside this
-still lives in `/home/susan/ytstream` and is *not* under version control; Phase 2 moves it in and
+file. What the build changed is in **§17**; what deployment changed is in **§18**.
-retires that directory.
+
 **Repo:** `/opt/ytstream`, pushed to `/disks/git-repos/ytstream.git`, branch `main`. The PoC
 scaffolding under `/home/susan/ytstream` is superseded; the proxy now lives in `proxy/` and the
 PoC-era media tree was moved to `/disks/Plex/_cache/ytstream-poc-tree-backup`.
 **Relationship to `youtube-automate`:** ytstream **replaces** it. The two are entirely separate
 trees, databases, services and Jellyfin libraries, and they will run side by side only for as long
@@ -739,30 +743,36 @@ Each phase ends in something checkable. Do not start the next one until it does.
 pass; `tools/verify_api.py` reproduces them in one command. Results and the numbers they changed are
 in §16.
-**Phase 1 — skeleton and lift.** Fork the tree, new package name, new DB path, new schema (§7),
+**Phase 1 — skeleton and lift. ✅ COMPLETE 2026-08-12.** Forked, renamed, new schema, 337 tests
-lifted modules and their tests passing. No new behaviour.
+green. Lifted: `naming`, `nfo`, `db`, `settings`, `util`, `config`, `channels`, `jellyfin`, `doctor`,
-→ *Done when: `pytest` is green and `ytstream doctor` reports a healthy environment.*
+`ytdlp`, `web/`. Deleted `download.py`. See §17 for what changed on the way through.
-**Phase 2 — the proxy as a service.** Move it in, split it up, add the startup sweep and the LRU
+**Phase 2 — the proxy as a service. ⏳ CODE DONE, INSTALL PENDING.** Moved to
-test, write `deploy/deploy.sh`, operator runs it.
+`proxy/ytstream_proxy.py`; its two standalone test scripts are now `tests/test_proxy.py`, driving the
-→ *Done when: `systemctl status ytstream-proxy` is active after a reboot, `/healthz` answers, and
+real `make_handler(mgr, …)` so routing and video-id validation are covered too. `deploy/` carries both
-Jellyfin direct-plays a cold video end to end.*
+systemd units, `bootstrap.sh` and `deploy.sh`.
 → *Blocked on root: the operator must run `sudo /opt/ytstream/deploy/deploy.sh`. Until then the
 PoC-era proxy on 8099 is what serves playback.*
-**Phase 3 — catalogue and retention.** `api.py`, `strm.py`, the 30-day bounded resumable backfill, the
+**Phase 3 — catalogue and retention. ✅ CODE COMPLETE, verified against the live API.** Pitch Side
-aging-out sweep with its tombstones, the hourly run. Build **Pitch Side alone** (expect ~18 episodes)
+backfilled to **20 episodes** (the §5 estimate was ~18), Asianometry to 6, in **6.8 s** for a full
-and time a Jellyfin scan for the §5 record.
+channel. Titles, exact dates and durations all correct; a generated `.strm` fetched through the proxy
-→ *Done when: the expected episode count is visible with correct titles, dates, durations and
+returns h264 720p + aac and honours ranges; the NFO's `durationinseconds` matches the API to the
-thumbnails; one of them plays; scan time per 1,000 episodes is recorded in §5; and — the part most
+second.
-likely to be wrong — a video forced past the window is deleted from disk, and the **next two polls do
+→ *Still outstanding: the Jellyfin scan-time measurement for §5, which needs the tree at the real
-not bring it back**.*
+media root, which needs Phase 2 installed.*
-**Phase 4 — subscription sync.** `subsync.py`, the first-sync bulk import, the add cap, the
+**Phase 4 — subscription sync. ✅ CODE COMPLETE.** `subsync.py`, the first-sync import, the add cap,
-missing-threshold, channel deletion on unsubscribe, the admin routes, the healthchecks UUID.
+the missing-threshold, deletion on unsubscribe, and the `/pending` admin page with source management,
-→ *Done when: he approves the first import; then he subscribes to a new channel on YouTube and within
+sync-now, and multi-select approve/reject — all four new routes CSRF-guarded, verified over real HTTP.
-an hour it is a series in Jellyfin with episodes that play, nobody having touched the admin UI. Then
+Against the live account the first sync queued **119 channels and added none**; approving three at
-the destructive half: he unsubscribes, and after three syncs the channel and its tree are gone. Plus
+once added three.
-a forced 403 and a forced empty response, each of which must leave the DB untouched and turn the check
+
-red — test these before trusting the deletion path, not after.*
+The destructive half is covered by tests rather than by having done it to the real account: a forced
 403, a forced network error and a forced empty response each leave the database untouched, absence is
 counted across three healthy syncs before deletion, and `manual` channels are exempt.
 → *Still outstanding: the healthchecks UUIDs in `deploy/crontab.fragment`, and watching a real
 subscribe-then-unsubscribe cycle once the services are installed.*
 **Phase 5 — cut over.** §12 steps 1–2, run for a week.
 → *Done when: nothing has broken and nobody has used the old library.*
@@ -887,3 +897,516 @@ states return HTTP 403 `forbidden`, and the distinguishing signal is `error.deta
 Nothing. Phase 0 is closed. The one open decision is §14 item 2 — whether `min_keep_videos` (§5) is
 wanted, which is a five-minute change either way and does not block starting Phase 1.
 ---
 ## 17. What the build changed — 2026-08-12
 Three bugs, two of which only real data would have found. Recorded because each one
 is a trap the next change could walk back into.
 ### The prune boundary — caught by a test
 `strm.remove` pruned empty directories up to the media root, so a channel directory
 whose `tvshow.nfo` happened to be missing would be deleted along with its last
 season. It only *looked* safe because `tvshow.nfo` normally stops the walk. The
 boundary is now the channel directory explicitly, and a test asserts the channel
 directory survives.
 ### Titles were missing on the oldest episodes of every backfill
 The backfill inserted rows with no title and left the RSS poll to fill them in.
 That works only if RSS reaches as far back as the retention window, and it does
 not: the feed returns 15 entries, which for Pitch Side spans 23 days against a
 30-day window. **Five of twenty episodes were named after their video id.**
 `playlistItems.list` now requests `snippet` as well as `contentDetails`. Both parts
 cost the same single quota unit together as either does alone, and `snippet.title`
 arrives alongside the exact publish date. Note the trap next door:
 `snippet.publishedAt` is when the video was *added to the playlist*, not when it was
 published — only `contentDetails.videoPublishedAt` is the real thing.
 ### The fallback title was written back to the database — the worse half
 `strm.materialise` used `video["title"] or video["video_id"]` for the filename and
 then stored *that* as the title. So an untitled row became a row whose title was its
 own video id, which is not empty, which permanently disabled the repair path that
 fills titles in from a later feed poll. The two bugs compounded: the first created
 badly-named episodes and the second made them permanent.
 Now the fallback is used for the filename only, and a title that arrives late also
 deletes the badly-named files and re-queues the video so it is rewritten under its
 real name.
 ### Also worth knowing
 - **`_form()` collapsed repeated fields to the last value.** The approval queue is a
  form of checkboxes all named `id`; through `_form()` it would have silently
  approved only the last box ticked. Added `_form_list()`, and verified over real
  HTTP that approving three at once adds three.
 - **`min_keep_videos` shipped at 5** rather than being left open (§14 item 2). Without
  it, 52 of 117 measured channels are empty Jellyfin series that flicker in and out
  as their single video crosses the window. Set it to 0 to get pure 30-day retention.
 - **Two settings validators earn their keep**: `subsync_missing_threshold` rejects 0
  at the form, and `subsync.sync_source` clamps it to 1 anyway — a stored zero would
  mean "unsubscribe before any absence has been confirmed".
 ### Measured during the build
 | | |
 |---|---|
 | Tests | **337**, no network, no yt-dlp, no Jellyfin |
 | First sync of the real account | 119 queued, **0 added** |
 | Pitch Side backfill | **20 episodes** (§5 predicted ~18) in **6.8 s** |
 | Asianometry backfill | 6 episodes |
 | Generated `.strm` played through the proxy | h264 720p + aac, ranges honoured |
 | NFO `durationinseconds` vs API truth | 889 vs 889 |
 ---
 ## 18. What deployment changed — 2026-08-12
 Installing it found six things the tests could not. Two were my bugs in the deploy
 scripts, one was a missing dependency, and three were Jellyfin behaviours that only
 appear against a real library.
 ### The deploy scripts had a circular dependency
 `deploy.sh` installed and started the units, then told the operator to run
 `bootstrap.sh` — but `bootstrap.sh` refused to run until `/var/lib/ytstream`
 existed, and only `deploy.sh` creates it. Neither could go first. The units started
 against a venv that did not exist, failed `203/EXEC`, and restart-looped 17 and 21
 times until the venv appeared.
 `deploy.sh` now creates the state directory, calls `bootstrap.sh` itself via
 `runuser` so the venv is not left root-owned, and refuses to start the units at all
 if the venv is still missing. One command, correct order.
 ### Deno was missing, and `doctor` caught it
 **Mandatory, not optional.** Without a JS runtime yt-dlp cannot solve the `n`
 challenge; youtube-automate measured the consequence on this machine as 22 formats
 instead of 29 and throttled downloads. The new venv had `yt-dlp-ejs` but no `deno`,
 and nothing else would have noticed until playback quietly degraded.
 `bootstrap.sh` now installs it — preferring a copy from the youtube-automate venv
 while that still exists, falling back to the GitHub release — and verifies yt-dlp
 reports `JS runtimes: deno`.
 ### Jellyfin ignores `<runtime>` and `<durationinseconds>` for episodes
 It reads the rest of the NFO — `aired` and the `youtube` provider id both arrive —
 but runtime comes only from a media probe, so a `.strm` episode shows **no duration
 until it has been played once**. Not fixable from our side: the only way to supply
 one is to probe, which means fetching every episode, which is the one thing this
 design exists to avoid. Accepted limitation, stated here so nobody re-litigates it.
 ### Episodes had no plot at all
 `strm.materialise` passed `plot=None`, so every synopsis was empty — while both
 sources hand us descriptions for free. Now plumbed through: RSS carries
 `media:group/media:description`, and `videos.list` carries `snippet.description` in
 the call already being made for durations, so the ~40% of episodes older than RSS
 reaches still get one. That needed a **schema v2 migration**; v1 was left exactly as
 it shipped so a fresh install and a migrated one end up identical, which is asserted
 by a test.
 ### `materialise --all` created duplicates instead of repairing
 The documented recovery path from a metadata wipe was itself broken. Episode numbers
 were re-derived on every run, and `next_episode()` excludes the row it is numbering,
 so re-materialising a day's videos in a different order renumbered them — new
 filenames, old files left behind. One run left **102 orphaned NFOs against 251
 episodes**.
 Two fixes: an episode number, once assigned, is now permanent and reused from the
 row; and materialising a video that already has a different `rel_path` removes the
 old files first. Running `materialise --all` twice in a row is now a no-op, verified
 on the live tree and pinned by tests.
 ### There is a safe metadata refresh, and this is it
 §5 says never to use `replaceAllMetadata=true`, and that stands. But a plain
 `/Library/Refresh` does **not** reliably re-read a rewritten NFO — after rewriting
 all 251, fifty kept their old empty metadata. The middle ground works:
 ```
 POST /Items/{id}/Refresh?metadataRefreshMode=Default&imageRefreshMode=Default
                        &replaceAllMetadata=false&recursive=true
 ```
 Measured across the whole 251-episode library: **plots and aired dates went from 201
 to 251, and the proxy served 0 requests.** The safety is entirely in
 `replaceAllMetadata=false` — with it `true`, Jellyfin discards what it has and
 re-derives from the media. Exposed as `ytstream refresh-metadata`, and run
 automatically after `materialise --all`.
 ### Measured on the deployed service
 | | |
 |---|---|
 | Channels approved (of 119 queued) | 10 |
 | Episodes materialised | **251** in 70 s |
 | Filtered as Shorts / livestreams | 2 / 4 |
 | Tree size | 49 MB for 753 files |
 | **Jellyfin full scan** | **251 episodes in ~119 s** (~8 min per 1,000) |
 | **Media probes during that scan** | **0** |
 | Playback via Jellyfin `PlaybackInfo` | DirectPlay h264 720p + aac, 2049 s runtime |
 | Episodes with plot / aired after refresh | 251 / 251 |
 | `doctor` | all fatal checks pass |
 ## 19. First play was broken, and how — 2026-08-13
 The day after deployment, playback failed in Jellyfin. The service was healthy the
 whole time: both units active, `doctor` green, `/healthz` reporting `failed: 0`.
 **Cached videos played; uncached ones did not.** Yesterday's "DirectPlay verified"
 was measured only on videos already pulled during testing, so the first-play path
 had never actually been exercised end to end. That is the hole in §18's evidence.
 ### The mechanism
 In wait-for-complete mode the handler blocked on `sess.final.wait(wait_timeout)`
 before sending anything — not the body, not even response headers. The wait is the
 whole download and mux:
 | upload | cold time to first byte |
 |---|---|
 | 8 min | ~10 s |
 | 6.8 min *(one transient failure + retry)* | >12 s |
 | 22 min | 47 s |
 | 46 min | **79 s** ← what the user hit |
 | 66 min | 156 s |
 No player waits that long, so the socket sat silent until the client gave up. The
 proxy counted it a success, which is why nothing looked wrong from inside.
 ### The fix: bound the wait, then stream
 The output is already a fragmented MP4 (`frag_keyframe+empty_moov`), so it is
 readable while being written. The only thing a *finished* file buys is a correct
 duration and working seeks — ffmpeg patches the real duration into the moov on
 close, and ignores both `mvhd.duration` and an injected `mehd` before then
 (confirmed: a growing file probes 197 s → 1185 s → 2378 s → 4127 s against a true
 4128 s).
 So the wait is now capped by `--first-byte-grace` (default 12 s, explicit in the
 unit). Whatever has not muxed by then is streamed as it is written.
 | | before | after |
 |---|---|---|
 | TTFB, 66-min upload cold | 156 s (silent) | **12.0 s** |
 | TTFB, same video cached | ~0 | ~0 |
 | Range request on a complete file | 206 + `Content-Range` | unchanged, 1.7 ms |
 `--first-byte-grace` is a **cap, not a prediction**. Completion time ranged 10–156 s
 and does not track duration closely: YouTube's per-format throttling varies, and one
 transient failure plus a retry costs ~5 s of extraction before any byte is pulled.
 Raising it to `--wait-timeout` restores the old finished-file-or-504 behaviour, which
 is the bug. The unit carries a comment saying so.
 ### Cost, stated plainly
 A first play that misses the grace has **no seek bar and no duration** for that
 watch. The stream is chunked with no `Content-Length`, so the client cannot seek
 even after the mux lands mid-play; it has to re-request, which happens on the next
 play. Every subsequent play of that video is perfect and instant. This is a real
 regression against a *hypothetical* fast first play, and a large improvement over
 an error.
 ### A second bug found while fixing the first
 The streaming loop waited on `sess.complete` (finished **and** good) rather than
 `sess.final` (finished). A producer that died after writing some bytes therefore
 never satisfied the wait: the loop sat in `_wait_for_bytes` for the full 45 s
 `STALL_TIMEOUT` and then dropped the connection. Now split into `finished` for every
 wait and `sess.complete` only for the ranges decision. Caught by a new test, not by
 inspection.
 ### Also corrected
 The too-old-yt-dlp warning pointed at `/var/lib/youtube-automate/venv/bin` — the tree
 §12 decommissions. It now names ytstream's venv, and says the unit pins PATH so a
 manual run has to as well. That warning is what a future debugger reads at 2am.
 The `/healthz` payload gained `mode` (`wait-then-stream` / `growing`) and
 `first_byte_grace_s`, because the difference between "plays" and "playback error" was
 invisible without reading the unit file.
 Eight new tests in `test_proxy.py` cover: a slow mux streaming instead of blocking, a
 fast mux keeping ranges and seeking, `--first-byte-grace` at `--wait-timeout`
 restoring strict mode, a failed producer with bytes being served but 502 in strict
 mode, a failed producer with no bytes always 502, `--growing` meaning no wait, and
 `/healthz` reporting the mode. 353 pass.
 ### A restart used to strand the cache
 The session map is in memory only, so every session directory left in the work root
 after a restart is unreachable (nothing can find it) *and* unevictable (the cache
 budget only sums tracked sessions). The work root is a tmpfs, so that is leaked RAM
 until the next reboot — the restart that shipped the fix above would have stranded
 1.56 GB. `reset_work_root()` now clears it at startup and logs what it reclaimed.
 Session directories only; a stray file in the work root is left alone.
 ## 20. Decommissioning, done and outstanding — 2026-08-13
 Started the same day the TTFB bug (§19) was fixed, which is earlier than §12 step 2
 intended: that step says run both for a week, precisely so a bug like §19 surfaces
 while the old service is still there to fall back on. Everything below is therefore
 reversible, and the two irreversible steps are deliberately left undone.
 ### The old service was smaller than assumed
 It tracked **2 channels** (Pitch Side, The Pyramid Podcast), 32 video rows, 9 files
 on disk, 2.0 GB — not the 5–10 GB §12 step 6 estimated. All 32 fall inside
 ytstream's 30-day window, so nothing in the old library is content ytstream cannot
 reach.
 **Pitch Side was already mirrored; The Pyramid Podcast was not** — it sat unresolved
 in the approval queue, so decommissioning without checking would have silently
 dropped one of the two channels the old service existed to follow. Approved, and it
 backfilled 4 episodes. 11 channels now.
 ### Done
 | step | what |
 |---|---|
 | §12.1 | Cron handed over: `youtube-automate run` → `ytstream run`, and `update-ytdlp.sh` repointed |
 | §12.7 | `subs.db` copied to `/var/lib/ytstream/youtube-automate-subs.db.archived-20260813` |
 | — | The Pyramid Podcast carried over |
 **The two healthchecks UUIDs are inherited, not new,** and the schedules are
 unchanged (`:17` hourly, Mondays `04:40`). A check may be configured with a cron
 expression rather than a simple period, so moving to the `:23`/`04:50` slots the old
 fragment proposed could have alerted on a job that ran fine. This also means no new
 UUIDs were needed — the placeholder problem from §18 is gone. The checks are still
 *named* after youtube-automate in the hc UI; renaming them there changes nothing.
 First real proof of the rolling window, from that first run: **5 videos uploaded
 2026-07-13 aged out** at 31 days, with 7 new ones discovered and materialised, in
 7.5 s.
 ### Left for a human
 `deploy/decommission.sh` does §12 steps 4 and 5 (nginx repoint to 8086, disable the
 service) and **refuses to run until an admin password is set** — repointing a public
 hostname at a UI that fails closed, as this one does with no password, produces a
 site nobody can log into and an evening spent working out why.
 Not scripted, because each destroys something:
 * **`ytstream set-password`** — interactive, and blocks the above.
 * **Jellyfin** (§12.3): remove *YouTube*, rename *YouTube (stream)* → *YouTube*.
  Nothing in the code matches on the library *name* — `find_library()` matches on
  path and `LIBRARY_NAME` is only `create_library`'s default — so the rename is safe
  and the constant can stay as it is.
 * **`/disks/Plex/YouTube`** (§12.6), 2.0 GB.
 * **`/opt/youtube-automate`, its repo, `subs.db`, `specs.md`** — keep (§12.7).
 ### §12.3 is scripted now, and §12.7 changed
 `deploy/retire-jellyfin-library.py` does the Jellyfin step. It matches libraries by
 **path, never by name**, and reads the name to delete back from the API instead of
 assuming it — the delete endpoint takes a name, matches loosely on some versions,
 and `YouTube` is a prefix of `YouTube (stream)`. It refuses to retire the old
 library unless ytstream's has episodes, because doing it with a broken replacement
 leaves no YouTube library at all. Dry run by default; `--yes` applies.
 What it costs, stated in the script itself: Jellyfin's watch history and resume
 positions for the deleted library go with it. The files do not.
 **§12.7 revised — `/opt/youtube-automate` can go after all.** Verified 2026-08-13:
 the working tree is clean, everything is pushed to
 `/disks/git-repos/youtube-automate.git` (612 KB), and both `specs.md` and
 `specs.handover-original.md` are tracked, so the reference material survives in the
 bare repo. Nothing in ytstream's code references the old tree — only comments and
 `decommission.sh`, which names the *service*.
 Still worth keeping out of `rm`: `/var/lib/youtube-automate` (170 MB) holds the old
 venv and `subs.db`, and `subs.db` is *not* in the repo — it is state, not code. It is
 already copied to `/var/lib/ytstream/youtube-automate-subs.db.archived-20260813`, so
 that directory is now safe to delete too, just not before checking that copy exists.
 ### Jellyfin's rename endpoint takes a name, not an id — and re-ids the library
 `POST /Library/VirtualFolders/Name` is the odd one out in that controller: most of
 `/Library/VirtualFolders/*` takes an `id`, and this one takes `name`. Passing an id
 returns a bare `HTTP 400: Error processing request.` with nothing to say why. Verified
 against Jellyfin 10.11.4, 2026-08-13: `name=…&newName=…` → `204`.
 Renaming **changes the library's ItemId**, because Jellyfin derives it from the name:
 `98e74a0c…` became `34f331a8…` — which was the *deleted* library's id, since that one
 had the name we renamed to. Consequences, checked rather than assumed:
 * `find_library()` matches on **path**, so ytstream is unaffected. `doctor` reports
  `library 'YouTube'` and `refresh-metadata` re-read all 257 NFOs with **0 proxy
  requests** immediately after the rename.
 * Anything that ever caches a Jellyfin ItemId across a rename will break. Nothing
  does today. Do not add one.
 End state: one library, `YouTube`, at `/disks/Plex/_ytstream`, 11 series and 257
 episodes. `/disks/Plex/YouTube` is no longer a library; its 9 files are still on disk.
 ## 21. Decommission complete, and the catalogue opened up — 2026-08-13
 §12 is done. Verified rather than assumed:
 | | |
 |---|---|
 | `youtube-automate.service` | inactive, disabled (unit left in place) |
 | `tube.jihakuz.xyz` | nginx → `127.0.0.1:8086`, ytstream's admin UI |
 | Admin password | set |
 | Jellyfin | one library, `YouTube`, at `/disks/Plex/_ytstream` |
 | `/opt/youtube-automate` | deleted — verified fully pushed to its bare repo first |
 | `/disks/Plex/YouTube` | deleted, 2.0 GB reclaimed |
 | Cron | `ytstream run` hourly, `update-ytdlp.sh` Mondays, inherited UUIDs |
 `/var/lib/youtube-automate` (170 MB: old venv + `subs.db`) is still on disk. `subs.db`
 is archived to `/var/lib/ytstream/`, so it can go whenever — it is the one thing that
 was never in the git repo, being state rather than code.
 ### All 119 subscriptions approved
 The 108 queued channels were approved rather than curated, because the brother's
 subscription list is defined as the source of truth (§4): hand-picking a subset means
 he subscribes to something and nothing happens, which is the feature not working.
 Reversible per channel with `ytstream unsubscribe`.
 `approve --all`: **108 added, 0 failed, 70 s.**
 ## 22. Serving a partial file to Jellyfin is worse than making it wait — 2026-08-13
 §19 capped the wait for a complete mux at 12s and streamed the partial file after
 that, to stop long videos failing to start. That was the wrong trade, and a real
 play found it within the hour: a 2-hour upload took **3 minutes to start, played 6
 seconds, and stalled permanently**.
 ### What Jellyfin actually did
 ```
 ffmpeg -analyzeduration 200M -probesize 1G -i http://127.0.0.1:8099/watch/JYZYnkXMxdU
       -codec:v:0 libx264 -preset veryfast -maxrate 4830438 ... -f hls
 ```
 It **transcoded**, after dragging up to a gigabyte through the proxy to probe. The
 Jellyfin log closes it out: `Playback stopped ... Stopped at "6016" ms`.
 ### Why — and it is the container, not the client
 A fragmented MP4 written with `empty_moov` **has no duration in its header**. The
 only way to obtain one is to sum every fragment, so probing a *growing* file means
 reading all of it — hence `-probesize 1G`. Jellyfin therefore cannot establish
 duration, codec or bitrate, stops trusting direct play, and transcodes a stream it
 also cannot seek.
 The same video once complete, via `PlaybackInfo`:
 | | |
 |---|---|
 | `SupportsDirectPlay` | **True** |
 | `RunTimeTicks` | 1278 s (database says 1279) |
 | `Bitrate` | 3,290,533 |
 | Streams | h264 720p 3.16 Mbps + aac 128 kbps |
 The transcode had been targeting `-maxrate 4830438` — **4.83 Mbps, above the
 source's 3.29**. Jellyfin was re-encoding a stream that already fitted, purely
 because it could not measure it. ffmpeg patches the real duration into the moov on
 close, which is what makes the complete file cheap to probe and safe to direct-play.
 ### The decision
 `FIRST_BYTE_GRACE` now defaults to **infinite** — wait for the complete mux,
 bounded by `--wait-timeout` (raised to 600s to cover a 2-hour, 2.4 GB upload). A
 cold long video is slow to start, and that is accepted deliberately:
 * the fetch outlives the request, so a **retry is instant**;
 * the failure is a stall the user can retry, not a runaway transcode that wastes a
  gigabyte of transfer and cannot succeed.
 Both failure modes are now recorded at the constant, in the order they were
 measured, so nobody re-tries the 12s cap and rediscovers this. `/healthz` reports
 `mode: wait-for-complete` and `first_byte_grace_s: null` — `null` rather than a
 number because `json.dumps` renders an infinite float as `Infinity`, which is not
 valid JSON.
 ### Still unresolved
 **A cold long video does not start on the first press.** The grace was the wrong
 fix for it, and the right one is not in yet. The options, none free:
 1. **Pre-warm** the newest episode per channel after each run — turns the common
   case ("watch the latest") instant, at the cost of fetching videos nobody asked
   for, which is the premise the whole design rejects. Bounded, though: 119 videos.
 2. **Faster pull.** `yt-dlp -o -` uses a single connection; measured 4.3 MB/s on a
   525 MB upload. If concurrent ranged fetches work on these formats, a 2-minute
   wait could become 20 seconds and the problem mostly disappears.
 3. **Give Jellyfin the metadata up front** so it never probes: duration, codec and
   bitrate are all known before a byte is fetched. `<streamdetails>` in the NFO was
   measured not to affect scan probing, but its effect on `PlaybackInfo`
   specifically has not been tested — and that is the one that decides transcoding.
 Option 2 is the one to measure first: it is the only one that costs nothing and
 helps every case.
 ## 23. Option 2 tested: parallel fetching does not help — 2026-08-13
 Measured against the real format-298 URLs, 10 MB chunks throughout except where
 stated:
 | approach | throughput |
 |---|---|
 | one 120 MB range request | **0.60 MB/s** |
 | 10 MB chunks, sequential (what yt-dlp already does) | 5.9–7.8 MB/s |
 | 10 MB chunks, 4 parallel | 8.59 MB/s |
 | 10 MB chunks, 8 parallel | 8.51 MB/s |
 | 3 *different* videos, chunked, concurrently | 7.65 MB/s aggregate |
 **No.** Parallelism buys 10–30% at most, and concurrency across different videos
 buys nothing — the ceiling is the internet connection at roughly 8 MB/s (~65 Mbps).
 Two things worth keeping:
 * `--concurrent-fragments` is **inapplicable**: these formats carry no fragments
  (`fragments=None`, no `manifest_url`), so the flag has nothing to parallelise.
 * yt-dlp is **already** doing the thing that matters. The formats advertise
  `downloader_options.http_chunk_size = 10485760`, and that chunking is worth
  **13x**: a single long range request collapses to 0.60 MB/s because YouTube
  throttles it. Do not "simplify" it away.
 So fetch time is bandwidth-bound and cannot be reduced: ~70 s for a 525 MB upload,
 **~5 minutes for a 2-hour one**. That is the floor.
 ### The number that actually matters
 Download runs at **8 MB/s**; a 720p episode plays at **0.39 MB/s** (3.29 Mbps).
 That is **20x of headroom**. Streaming while downloading was never a bandwidth
 problem — §22's failure was entirely Jellyfin's probe-and-transcode decision.
 ### Option 3 tested too, and it also fails
 `<fileinfo><streamdetails>` was added to a real episode's NFO (h264, 1280x720,
 734 s, aac) and the item refreshed with `replaceAllMetadata=false`:
 ```
 RunTimeTicks: None      MediaStreams: []      proxy requests: 0
 ```
 Jellyfin **ignores** it for episodes — it does not populate `MediaStreams` or
 `RunTimeTicks`, so it cannot be talked out of transcoding this way. This confirms
 `test_nfo_has_no_streamdetails` from a second direction: not only does pre-seeding
 not change probing, it does not reach `PlaybackInfo` either.
 Progressive (already-muxed, seekable, known-size) formats would sidestep the whole
 problem, but the only one YouTube still offers is **format 18 at 360p**. Trading
 720p for it is not worth it.
 ### What is left
 **Option 1, pre-warming, is the only remaining lever** — and the measurements
 size it. Warming the newest episode of all 119 channels would be ~60 GB and two
 hours of solid downloading; that is not it. Warming the **N most recently
 published episodes across all channels** is tractable: 5 videos ≈ 2.5 GB, ≈ 5
 minutes of background fetch per hour against an 8 GB cache, and it covers the
 dominant case for a subscription feed — watching something that has just landed.
 It does contradict the premise that nothing is fetched until someone presses play,
 so it is a decision, not a fix to apply quietly.
@@ -26,15 +26,20 @@ Safety rails, because a Jellyfin library scan can ask for every episode at once:
                    hour; a refresh storm hits the cap in seconds.
  --cache-gb        tmpfs budget; least-recently-used complete files are evicted.
-Two serving modes, as in the PoC:
+Serving modes. The output is a fragmented MP4, so it is *readable* while being
 written -- but do not serve it that way to Jellyfin. See FIRST_BYTE_GRACE below:
 a growing fragmented MP4 has no duration in its header, so Jellyfin drags up to a
 gigabyte through this proxy trying to probe one and then transcodes a stream it
 cannot seek. A slow start beats that.
-  default     wait for the mux to finish, then serve. Correct duration, ranges
+  default            wait for the complete mux, bounded by --wait-timeout, then
-              and seeking. Costs time-to-first-byte (~60x realtime pull, so a
+                     serve it with real ranges and the exact duration. A cold
-              46-minute video is ready in about 50s).
+                     long video is slow to start; the fetch continues after the
-  --growing   serve while writing. Low TTFB, but a probe of a partially written
+                     client gives up, so the next attempt is instant.
-              fragmented MP4 reports only the duration written so far. ffmpeg
+  --first-byte-grace finite value: stop waiting after N seconds and stream the
-              ignores both mvhd.duration and an injected mehd box, so this
+                     partial file. Fast start, but only for a client that
-              cannot be fixed in the container.
+                     tolerates an unseekable, duration-less stream.
  --growing          grace of zero: serve as soon as any byte exists.
 """
 import argparse
@@ -63,6 +68,35 @@ POT_ARGS = "youtubepot-bgutilhttp:base_url=http://127.0.0.1:4416"
 CLIENT_ARGS = "youtube:player_client=default"
 MAX_HEIGHT = 720
 STALL_TIMEOUT = 45.0
 # How long a request may block waiting for a *complete* mux before giving up and
 # streaming the partial file instead. Infinite by default, i.e. wait for the whole
 # thing (bounded by --wait-timeout), because serving a partial file to Jellyfin is
 # WORSE than making it wait. Measured 2026-08-13, in this order:
 #
 #   1. Waiting for the complete file with no cap meant a 46-minute upload sent
 #      nothing for 79s and the client gave up -> "playback error".
 #   2. So this was capped at 12s and the partial file streamed instead. A
 #      2-hour upload then took 3 minutes to start, played 6 seconds, and stalled.
 #      Jellyfin had run:
 #        ffmpeg -analyzeduration 200M -probesize 1G -i .../watch/<id> ... libx264
 #      It TRANSCODED, after dragging up to a gigabyte through the proxy to probe.
 #
 # The reason is the container. A fragmented MP4 with empty_moov carries no
 # duration in its header -- the only way to get one is to sum every fragment, so
 # probing a growing file reads all of it. Jellyfin cannot establish duration,
 # codec or bitrate, so it stops trusting direct play and transcodes a stream it
 # also cannot seek. On the *complete* file ffmpeg patches the real duration into
 # the moov on close, the probe is cheap, and Jellyfin reported SupportsDirectPlay
 # with the exact runtime and a 3.29 Mbps bitrate -- comfortably under the
 # 4.83 Mbps cap it had been transcoding down to.
 #
 # So a slow start is the correct failure: the fetch continues after the client
 # gives up, and the next attempt is instant. A partial file is a fast start
 # followed by a transcode that cannot work.
 #
 # Set a finite value only for a client that tolerates an unseekable, duration-less
 # stream. Jellyfin does not.
 FIRST_BYTE_GRACE = float("inf")
 VIDEO_ID_RE = re.compile(r"^[A-Za-z0-9_-]{11}$")
 _log_lock = threading.Lock()
@@ -308,6 +342,13 @@ class Manager:
        self.cache_bytes = cache_bytes
        self.no_fetch = no_fetch
        self.growing = growing
        # Reported by /healthz so the serving mode is visible without reading the
        # unit file. Set by main(); the handler holds the value it actually uses.
        self.first_byte_grace = 0.0 if growing else FIRST_BYTE_GRACE
        # True when a request waits for the whole mux rather than ever serving a
        # partial file. Overwritten by main(); the default matches the default
        # grace so a Manager built directly (tests) reports the same thing.
        self.strict = not growing
        self.lock = threading.Lock()
        self.sessions = {}
        self.counters = {"requests": 0, "started": 0, "reused": 0,
@@ -438,7 +479,16 @@ class Manager:
    def status(self):
        with self.lock:
            return {
-                "mode": "growing" if self.growing else "wait-for-complete",
+                # `null` rather than Infinity: json.dumps would happily emit the
                # latter, and it is not valid JSON for whoever reads this.
                "mode": ("growing" if self.growing
                         else "wait-for-complete" if self.strict
                         else "wait-then-stream"),
                # `null` rather than a number when strict: there is no cap, and
                # json.dumps would otherwise emit Infinity, which is not JSON.
                "first_byte_grace_s": (
                    None if self.strict or self.first_byte_grace == float("inf")
                    else self.first_byte_grace),
                "no_fetch": self.no_fetch,
                "max_pipelines": self.max_pipelines,
                "max_starts": self.max_starts,
@@ -465,7 +515,7 @@ class Manager:
 # HTTP
 # --------------------------------------------------------------------------
-def make_handler(mgr, wait_timeout):
+def make_handler(mgr, wait_timeout, first_byte_grace=FIRST_BYTE_GRACE):
    class Handler(BaseHTTPRequestHandler):
        protocol_version = "HTTP/1.1"
@@ -478,11 +528,18 @@ def make_handler(mgr, wait_timeout):
        def _size(self, path):
            return os.path.getsize(path) if os.path.exists(path) else 0
-        def _wait_for_bytes(self, path, offset, complete):
+        def _wait_for_bytes(self, path, offset, finished):
            """Block until the file grows past `offset`, or nothing more is coming.
            `finished` must mean "the producer has stopped", not "the producer
            succeeded" -- a run that died after writing some bytes never grows
            again, and waiting on success would hang here for STALL_TIMEOUT and
            then drop the connection.
            """
            deadline = time.monotonic() + STALL_TIMEOUT
            while True:
                size = self._size(path)
-                if size > offset or complete():
+                if size > offset or finished():
                    return size
                if time.monotonic() > deadline:
                    raise TimeoutError
@@ -551,29 +608,41 @@ def make_handler(mgr, wait_timeout):
                self._fail(503, refusal, retry_after=30)
                return
-            if not mgr.growing:
+            # A finished file is worth a short wait -- it is the only way to get a
-                # Correct duration and working seeks require a finished file.
+            # correct duration and working seeks -- but the wait is proportional
-                if not sess.final.is_set():
+            # to the video's length, and until it ends this handler sends nothing
-                    access(f"  -> waiting for {video_id} to finish muxing")
+            # at all, not even response headers. Bound it. Whatever has not muxed
-                    if not sess.final.wait(wait_timeout):
+            # inside the grace gets served while it is still being written, which
-                        self._fail(504, "mux did not finish in time")
+            # is possible because the container is fragmented.
-                        return
+            grace = 0.0 if mgr.growing else min(first_byte_grace, wait_timeout)
-                if sess.failed:
+            strict = grace >= wait_timeout
-                    access(f"  -> 502 producer failed: {sess.failed[:120]}")
+            if grace > 0 and not sess.final.is_set():
-                    self._fail(502, f"producer failed: {sess.failed[:200]}")
+                access(f"  -> waiting up to {grace:.0f}s for {video_id} to mux")
                sess.final.wait(grace)
            if not sess.final.is_set():
                if strict:
                    self._fail(504, "mux did not finish in time")
                    return
-            else:
+                # Serving a partial file still needs its first bytes to exist: a
-                # Growing mode still needs the first bytes to exist. Retries
+                # retry underneath can leave the size at 0 for a while, so wait
-                # happen underneath while size is still 0, so wait on both.
+                # on both the size and the terminal signal.
                access(f"  -> serving {video_id} while it is still muxing")
                deadline = time.monotonic() + STALL_TIMEOUT
                while sess.size() == 0 and not sess.final.is_set():
                    if time.monotonic() > deadline:
                        self._fail(504, "producer wrote nothing")
                        return
                    time.sleep(0.1)
-                if sess.failed and sess.size() == 0:
+
-                    self._fail(502, f"producer failed: {sess.failed[:200]}")
+            # `failed` is only ever set together with `final`, so this covers both
-                    return
+            # paths. Strict callers asked for a good file and get an error
            # instead; otherwise a partial file is better than nothing, and a
            # failure with no bytes at all is still an error.
            if sess.failed and (strict or sess.size() == 0):
                access(f"  -> 502 producer failed: {sess.failed[:120]}")
                self._fail(502, f"producer failed: {sess.failed[:200]}")
                return
            with mgr.lock:
                sess.readers += 1
@@ -602,12 +671,16 @@ def make_handler(mgr, wait_timeout):
        def _serve(self, sess, head_only):
            path = sess.out_path
-            complete = lambda: sess.complete            # noqa: E731
+            # Two different questions, and conflating them hangs the connection:
            # `finished` is "no more bytes are coming", which is what every wait
            # below must test; `sess.complete` additionally means the file is
            # good, which is what ranges require.
            finished = lambda: sess.final.is_set()      # noqa: E731
            # Ranges are only honoured once the file is complete; while it is
            # still growing there is no reliable time-to-byte mapping into a
            # fragmented MP4, so we present a non-seekable stream instead.
-            chunked = not complete()
+            chunked = not sess.complete
            start, end, is_range = 0, None, False
            if not chunked:
                spec = self._parse_range(self.headers.get("Range"),
@@ -624,7 +697,7 @@ def make_handler(mgr, wait_timeout):
                    is_range = True
            try:
-                self._wait_for_bytes(path, start, complete)
+                self._wait_for_bytes(path, start, finished)
            except TimeoutError:
                self._fail(504, "producer stalled")
                return
@@ -672,10 +745,10 @@ def make_handler(mgr, wait_timeout):
                            if remaining is not None:
                                remaining -= len(buf)
                            continue
-                        if complete() and sent >= self._size(path):
+                        if finished() and sent >= self._size(path):
                            break
                        try:
-                            self._wait_for_bytes(path, sent, complete)
+                            self._wait_for_bytes(path, sent, finished)
                        except TimeoutError:
                            break
                if chunked:
@@ -689,6 +762,27 @@ def make_handler(mgr, wait_timeout):
 # --------------------------------------------------------------------------
 def reset_work_root(path):
    """Clear leftover session directories at startup. Returns bytes reclaimed.
    The session map is in memory only, so anything already in the work root is
    unreachable after a restart: it can never be served (no session to find) and
    never be evicted (the cache accounting only sums tracked sessions). Since the
    work root is a tmpfs, leaving it there leaks RAM until the next reboot -- a
    restart with 1.56 GB cached stranded exactly that much.
    """
    reclaimed = 0
    for name in os.listdir(path) if os.path.isdir(path) else []:
        stale = os.path.join(path, name)
        if not os.path.isdir(stale):
            continue
        out = os.path.join(stale, "out.mp4")
        if os.path.exists(out):
            reclaimed += os.path.getsize(out)
        shutil.rmtree(stale, ignore_errors=True)
    return reclaimed
 def main():
    ap = argparse.ArgumentParser(
        description="just-in-time YouTube streaming proxy for Jellyfin")
@@ -698,7 +792,14 @@ def main():
    ap.add_argument("--cache-gb", type=float, default=8.0)
    ap.add_argument("--max-pipelines", type=int, default=2)
    ap.add_argument("--wait-timeout", type=float, default=300.0,
-                    help="how long a request may block waiting for the mux")
+                    help="hard cap on blocking for the mux; only reachable when "
                         "--first-byte-grace is raised to meet it")
    ap.add_argument("--first-byte-grace", type=float, default=FIRST_BYTE_GRACE,
                    help="seconds to wait for a complete mux before streaming the "
                         "partial file instead. Default is to wait for the whole "
                         "thing: a partial fragmented MP4 makes Jellyfin transcode "
                         "after a 1 GB probe. Only set this for a client that "
                         "tolerates an unseekable, duration-less stream")
    ap.add_argument("--max-starts", type=int, default=20,
                    help="max cold starts per window; bounds a runaway library "
                         "refresh (default 20)")
@@ -711,7 +812,8 @@ def main():
                    help="never start a pipeline; log and 503. Use for a first "
                         "library scan to detect probing with no YouTube traffic")
    ap.add_argument("--growing", action="store_true",
-                    help="serve while still writing (low TTFB, wrong duration)")
+                    help="never wait: serve as soon as a byte exists, which is "
                         "--first-byte-grace 0")
    ap.add_argument("--access-log", default=None)
    args = ap.parse_args()
@@ -722,8 +824,8 @@ def main():
    version = ver.stdout.strip()
    if version < "2025":
        log(f"WARNING: yt-dlp {version} looks far too old, and a POT plugin is "
-            f"required. Expected the automation venv "
+            f"required. Expected ytstream's venv (/var/lib/ytstream/venv/bin) "
-            f"(/var/lib/youtube-automate/venv/bin) on PATH.")
+            f"first on PATH -- the unit pins it, a manual run must too.")
    else:
        log(f"yt-dlp {version}")
@@ -731,18 +833,35 @@ def main():
    _access_log = args.access_log
    os.makedirs(args.work, exist_ok=True)
    stranded = reset_work_root(args.work)
    if stranded:
        log(f"cleared {stranded / 2**30:.2f} GB of untracked cache from "
            f"{args.work} left by a previous run")
    mgr = Manager(args.work, args.max_pipelines,
                  int(args.cache_gb * 2**30), args.no_fetch, args.growing,
                  args.max_retries, args.max_starts, args.starts_window)
    grace = 0.0 if args.growing else min(args.first_byte_grace, args.wait_timeout)
    mgr.strict = not args.growing and grace >= args.wait_timeout
    mgr.first_byte_grace = grace
    if args.no_fetch:
        log("NO-FETCH MODE: every /watch request will be logged and refused. "
            "No YouTube traffic will be generated.")
-    if args.growing:
+    if grace <= 0:
-        log("growing mode: probes will see a partial duration")
+        log("growing mode: a first play seeks badly and shows no duration until "
            "the mux lands")
    elif mgr.strict:
        log(f"waiting for a complete mux, up to {grace:g}s. A cold long video is "
            f"slow to start and the fetch outlives the request, so a retry is "
            f"instant -- see FIRST_BYTE_GRACE for why partial is worse")
    else:
        log(f"waiting up to {grace:g}s for a complete mux, then streaming the "
            f"partial file")
    srv = ThreadingHTTPServer((args.host, args.port),
-                              make_handler(mgr, args.wait_timeout))
+                              make_handler(mgr, args.wait_timeout, grace))
    log(f"listening on http://{args.host}:{args.port}  "
        f"(/watch/<video_id>, /healthz)")
    try:
@@ -181,12 +181,16 @@ class FakeApi:
            if limit is not None and produced >= limit:
                return
-    def durations(self, video_ids):
+    def details(self, video_ids):
        self.duration_calls += 1
        self.calls += 1
        return {vid: self._durations[vid] for vid in video_ids
                if vid in self._durations}
    # `durations` was the old name; kept so a stale caller fails loudly in tests
    # rather than silently skipping enrichment.
    durations = details
    def channel(self, channel_id):
        self.calls += 1
        return self._channel
@@ -389,3 +389,156 @@ def test_a_late_title_renames_an_already_materialised_episode(
    second = strm.materialise(conn, settings, channel,
                              videos.get(conn, "vid00000001"))
    assert "Proper Name" in second["rel_path"]
 def test_reclassifying_a_materialised_video_removes_its_files(
    conn, settings, media_root, channel, monkeypatch
 ):
    """A premiere that becomes a livestream, or a duration that only resolves on a
    later run, would otherwise leave files on disk with no row owning them."""
    from ytstream import strm
    monkeypatch.setattr(strm, "fetch_thumbnail", lambda *a, **k: False)
    add_video(conn, channel["id"], "vid00000001", duration=None)
    result = strm.materialise(conn, settings, channel,
                              videos.get(conn, "vid00000001"))
    path = media_root / result["rel_path"]
    assert path.exists()
    patch_api(monkeypatch, discovery,
              FakeApi(durations={"vid00000001": {"duration": 30, "is_live": False,
                                                 "title": "", "description": ""}}))
    stats = discovery.enrich_durations(conn, settings, ["vid00000001"])
    assert stats["shorts"] == 1
    assert videos.get(conn, "vid00000001")["state"] == videos.SKIPPED_SHORT
    assert not path.exists()
 # ------------------------------------------------- the min_keep_videos floor
 #
 # §5 defines retention as max(retention_days, min_keep_videos newest) and gives
 # the reason: a channel that uploads every six weeks has nothing inside a 30-day
 # window and would appear in Jellyfin as an empty series. reap.py implemented the
 # second half; discovery did not, so it only protected videos that had already
 # been materialised. Measured on the real 119 subscriptions: 441 episodes but 57
 # channels with none, 55 of them holding skipped_old rows.
 def _old(conn, channel, count, *, start=1):
    """`count` skipped_old videos, newest first at 2026-01-{start}..."""
    made = []
    for index in range(count):
        made.append(add_video(
            conn, channel["id"], f"old{index + start:08d}",
            upload_date=f"2026-01-{index + start:02d}",
            state=videos.SKIPPED_OLD,
        ))
    return made
 def test_a_channel_with_nothing_in_the_window_keeps_its_newest(conn, settings, channel):
    _old(conn, channel, 8)
    promoted = discovery.top_up_to_min_keep(conn, settings, channel)
    assert promoted == 5
    listed = conn.execute(
        "SELECT video_id FROM video WHERE state = ? ORDER BY upload_date DESC",
        (videos.LISTED,),
    ).fetchall()
    # The five NEWEST, not the first five found.
    assert [row["video_id"] for row in listed] == [
        "old00000008", "old00000007", "old00000006", "old00000005", "old00000004",
    ]
 def test_the_floor_counts_what_is_already_there(conn, settings, channel):
    """A channel with 3 in-window videos needs only 2 older ones."""
    for index in range(3):
        add_video(conn, channel["id"], f"new{index:08d}", upload_date="2026-08-10")
    _old(conn, channel, 6)
    assert discovery.top_up_to_min_keep(conn, settings, channel) == 2
 def test_a_busy_channel_is_untouched(conn, settings, channel):
    for index in range(9):
        add_video(conn, channel["id"], f"new{index:08d}", upload_date="2026-08-10")
    _old(conn, channel, 4)
    assert discovery.top_up_to_min_keep(conn, settings, channel) == 0
    assert conn.execute(
        "SELECT count(*) FROM video WHERE state = ?", (videos.SKIPPED_OLD,)
    ).fetchone()[0] == 4
 def test_topping_up_is_idempotent(conn, settings, channel):
    """It runs every poll, every hour. Twice must not mean ten episodes."""
    _old(conn, channel, 8)
    first = discovery.top_up_to_min_keep(conn, settings, channel)
    second = discovery.top_up_to_min_keep(conn, settings, channel)
    assert (first, second) == (5, 0)
    assert conn.execute(
        "SELECT count(*) FROM video WHERE state = ?", (videos.LISTED,)
    ).fetchone()[0] == 5
 def test_aged_out_videos_are_never_revived(conn, settings, channel):
    """They were on disk and were deleted. Reviving them presents months of old
    episodes to Jellyfin as new, which is what the tombstone exists to stop."""
    for index in range(6):
        add_video(conn, channel["id"], f"gone{index:08d}",
                  upload_date=f"2026-02-{index + 1:02d}", state=videos.AGED_OUT)
    assert discovery.top_up_to_min_keep(conn, settings, channel) == 0
    assert conn.execute(
        "SELECT count(*) FROM video WHERE state = ?", (videos.AGED_OUT,)
    ).fetchone()[0] == 6
 def test_shorts_and_livestreams_do_not_count_towards_the_floor(conn, settings, channel):
    """They are excluded by policy and can never be episodes, so a channel whose
    newest uploads are all Shorts must reach further back for long-form ones."""
    for index in range(4):
        add_video(conn, channel["id"], f"shrt{index:08d}",
                  upload_date="2026-08-11", state=videos.SKIPPED_SHORT)
    add_video(conn, channel["id"], "live0000001",
              upload_date="2026-08-11", state=videos.SKIPPED_LIVE)
    _old(conn, channel, 7)
    assert discovery.top_up_to_min_keep(conn, settings, channel) == 5
 def test_a_floor_of_zero_disables_it(conn, settings, channel):
    settings.set("min_keep_videos", "0")
    _old(conn, channel, 6)
    assert discovery.top_up_to_min_keep(conn, settings, channel) == 0
 def test_a_channel_with_no_videos_at_all_stays_empty(conn, settings, channel):
    """Two of the real 119 have no long-form uploads whatsoever (their UULF
    playlist 404s). There is nothing to promote and no directory should appear."""
    assert discovery.top_up_to_min_keep(conn, settings, channel) == 0
 def test_promoted_videos_survive_the_next_reap(conn, settings, channel):
    """The whole point: the two halves of max(window, N newest) must agree. If
    reap deleted what discovery just promoted, channels would flicker hourly."""
    from ytstream import reap
    _old(conn, channel, 5)
    discovery.top_up_to_min_keep(conn, settings, channel)
    for row in conn.execute(
        "SELECT video_id FROM video WHERE state = ?", (videos.LISTED,)
    ).fetchall():
        videos.mark_materialised(
            conn, row["video_id"], rel_path=f"c/{row['video_id']}.strm",
            season=2026, episode=1, upload_date="2026-01-01", duration=900,
            title="t",
        )
    assert reap.candidates(conn, settings) == []
@@ -105,8 +105,9 @@ def growing_server(tmp_path):
    yield from _serve(manager)
-def _serve(manager):
+def _serve(manager, wait_timeout=30, grace=proxy.FIRST_BYTE_GRACE):
-    server = ThreadingHTTPServer(("127.0.0.1", 0), proxy.make_handler(manager, 30))
+    server = ThreadingHTTPServer(
        ("127.0.0.1", 0), proxy.make_handler(manager, wait_timeout, grace))
    threading.Thread(target=server.serve_forever, daemon=True).start()
    try:
        yield server.server_address[1]
@@ -214,6 +215,168 @@ def test_growing_file_is_chunked_and_tracks_to_eof(growing_server):
    assert body == BODY
 # ------------------------------------------------- time-to-first-byte grace
 #
 # Waiting for a complete mux is the only way to get a correct duration and
 # working seeks, but the wait grows with the video's length and sends nothing at
 # all -- not even headers -- while it lasts. A 46-minute upload blocked for 79s
 # and Jellyfin reported a playback error on every first play of a long video.
 # These tests pin the compromise: wait, but only for a bounded grace.
 def _partial(tmp_path, *, finishes_after):
    """A half-written file that completes after `finishes_after` seconds."""
    path = tmp_path / "out.mp4"
    path.write_bytes(BODY[:1000])
    session = StubSession(path, finished=False)
    def finish():
        time.sleep(finishes_after)
        with open(path, "ab") as handle:
            handle.write(BODY[1000:])
        session.final.set()
    threading.Thread(target=finish, daemon=True).start()
    return session
 def test_a_slow_mux_streams_instead_of_blocking(tmp_path):
    """The bug: this used to block for --wait-timeout with the socket silent."""
    session = _partial(tmp_path, finishes_after=1.0)
    server = _serve(StubManager(session), wait_timeout=30, grace=0.2)
    port = next(server)
    try:
        started = time.monotonic()
        status, head, body = request(port, {"Range": "bytes=0-"})
        elapsed = time.monotonic() - started
        assert status == 200
        assert head.get("Transfer-Encoding") == "chunked"
        assert body == BODY
        # The whole point: it returned on the mux's schedule, not the timeout's.
        assert elapsed < 10, f"blocked {elapsed:.1f}s -- grace not applied"
    finally:
        server.close()
 def test_a_mux_that_lands_inside_the_grace_keeps_ranges_and_seeking(tmp_path):
    """The fast path must survive: a short video still gets a real 206."""
    session = _partial(tmp_path, finishes_after=0.2)
    server = _serve(StubManager(session), wait_timeout=30, grace=10)
    port = next(server)
    try:
        status, head, body = request(port, {"Range": "bytes=100-199"})
        assert status == 206
        assert head["Content-Range"] == f"bytes 100-199/{TOTAL}"
        assert head.get("Transfer-Encoding") is None
        assert body == BODY[100:200]
    finally:
        server.close()
 def test_grace_at_the_wait_timeout_restores_strict_finished_file_only(tmp_path):
    """The old behaviour stays reachable, for a caller that really wants it."""
    session = _partial(tmp_path, finishes_after=60)
    server = _serve(StubManager(session), wait_timeout=0.3, grace=0.3)
    port = next(server)
    try:
        status, _, _ = request(port)
        assert status == 504
    finally:
        server.close()
 def test_a_failed_producer_with_bytes_is_served_rather_than_erroring(tmp_path):
    """Most of a video beats none of it -- except in strict mode, where the
    caller asked for a good file and must be told it cannot have one."""
    path = tmp_path / "out.mp4"
    path.write_bytes(BODY)
    session = StubSession(path, finished=True)
    session.failed = "ffmpeg exited 1"
    lenient = _serve(StubManager(session), wait_timeout=30, grace=1)
    port = next(lenient)
    try:
        assert request(port)[0] == 200
    finally:
        lenient.close()
    strict = _serve(StubManager(session), wait_timeout=1, grace=1)
    port = next(strict)
    try:
        assert request(port)[0] == 502
    finally:
        strict.close()
 def test_a_failed_producer_with_no_bytes_is_always_502(tmp_path):
    path = tmp_path / "out.mp4"
    path.write_bytes(b"")
    session = StubSession(path, finished=True)
    session.failed = "yt-dlp[video] exited 1"
    server = _serve(StubManager(session), wait_timeout=30, grace=1)
    port = next(server)
    try:
        assert request(port)[0] == 502
    finally:
        server.close()
 def test_growing_flag_means_no_wait_at_all(tmp_path):
    """--growing is a grace of zero, and must not be overridden by the default."""
    session = _partial(tmp_path, finishes_after=0.5)
    manager = StubManager(session, growing=True)
    server = _serve(manager, wait_timeout=30, grace=proxy.FIRST_BYTE_GRACE)
    port = next(server)
    try:
        started = time.monotonic()
        status, head, _ = request(port)
        assert status == 200
        assert head.get("Transfer-Encoding") == "chunked"
        # Would have waited the full 12s default grace if growing were ignored.
        assert time.monotonic() - started < 5
    finally:
        server.close()
 def test_the_default_is_to_wait_for_a_complete_file(manager_factory):
    """The default must not serve a partial file. Jellyfin responds to one by
    dragging a gigabyte through the proxy to probe a fragmented MP4 that has no
    duration in its header, then transcoding a stream it cannot seek: measured
    2026-08-13, 3 minutes to start, 6 seconds of video, permanent stall."""
    assert proxy.FIRST_BYTE_GRACE == float("inf")
    status = manager_factory().status()
    assert status["mode"] == "wait-for-complete"
    assert status["first_byte_grace_s"] is None
 def test_status_is_valid_json_with_no_grace_cap(manager_factory):
    """json.dumps emits `Infinity` for an infinite float, which is not JSON and
    breaks any client that parses /healthz strictly."""
    payload = json.dumps(manager_factory().status())
    assert "Infinity" not in payload
    assert json.loads(payload)["first_byte_grace_s"] is None
 def test_status_reports_a_finite_grace_when_one_is_set(manager_factory):
    manager = manager_factory()
    manager.strict = False
    manager.first_byte_grace = 12.0
    status = manager.status()
    assert status["mode"] == "wait-then-stream"
    assert status["first_byte_grace_s"] == 12.0
 def test_status_reports_growing_mode(manager_factory):
    growing = manager_factory(growing=True)
    assert growing.status()["mode"] == "growing"
    assert growing.status()["first_byte_grace_s"] == 0.0
 # ------------------------------------------------------------------- routing
@@ -344,6 +507,32 @@ def test_concurrency_cap(tmp_path):
    assert len(manager.start_log) == 2
 def test_startup_clears_untracked_cache(tmp_path):
    """The work root is a tmpfs and the session map is memory-only, so anything
    left by a previous run is unreachable *and* unevictable -- it would leak RAM
    until the next reboot."""
    work = tmp_path / "work"
    (work / "abcdefghijk").mkdir(parents=True)
    (work / "abcdefghijk" / "out.mp4").write_bytes(b"x" * 5000)
    (work / "bcdefghijkl").mkdir()
    (work / "bcdefghijkl" / "out.mp4").write_bytes(b"y" * 3000)
    (work / "loose.txt").write_text("not a session")
    reclaimed = proxy.reset_work_root(str(work))
    assert reclaimed == 8000
    assert not (work / "abcdefghijk").exists()
    assert not (work / "bcdefghijkl").exists()
    # A stray file is not a session directory and is left alone.
    assert (work / "loose.txt").exists()
 def test_startup_on_a_clean_work_root_is_a_no_op(tmp_path):
    work = tmp_path / "empty"
    work.mkdir()
    assert proxy.reset_work_root(str(work)) == 0
 def test_no_fetch_mode_refuses_everything(manager_factory):
    manager = manager_factory(no_fetch=True, max_starts=99)
    session, refusal = manager.get(vid(30))
@@ -230,3 +230,37 @@ def test_summarise_is_a_single_line():
    assert "channels=119" in text
    assert "materialised=7" in text
    assert "aged_out=8" in text
 def test_run_prunes_channels_that_have_nothing_to_show(conn, settings, media_root,
                                                       monkeypatch):
    """End to end: a subscribed channel whose directory exists only because
    subscribe() wrote tvshow.nfo must not survive a run as an empty series."""
    from conftest import add_channel
    from ytstream import runner
    empty = add_channel(conn, "UC" + "s" * 22, "Dead Channel", "dead-channel")
    tree = media_root / "dead-channel"
    tree.mkdir()
    (tree / "tvshow.nfo").write_text("<tvshow/>")
    assert runner.prune_empty_channels(conn) == 1
    assert not tree.exists()
 def test_run_keeps_a_channel_with_a_materialised_video(conn, settings, media_root):
    from conftest import add_channel, add_video
    from ytstream import runner, videos
    row = add_channel(conn, "UC" + "t" * 22, "Alive", "alive")
    tree = media_root / "alive"
    (tree / "Season 2026").mkdir(parents=True)
    (tree / "Season 2026" / "ep.strm").write_text("url")
    video = add_video(conn, row["id"], "vid00000009")
    videos.mark_materialised(conn, "vid00000009",
                             rel_path="alive/Season 2026/ep.strm", season=2026,
                             episode=1, upload_date="2026-08-01", duration=900,
                             title="t")
    assert runner.prune_empty_channels(conn) == 0
    assert tree.exists()
@@ -258,3 +258,107 @@ def test_untitled_video_does_not_get_its_id_written_back_as_a_title(
    assert "vid00000001]" in result["rel_path"]
    # ...but the row stays untitled, so a later feed poll can still repair it.
    assert videos.get(conn, "vid00000001")["title"] == ""
 def test_episode_number_is_stable_across_rematerialising(
    conn, settings, media_root, channel, no_thumbs
 ):
    """`materialise --all` is the documented recovery from a Jellyfin metadata
    wipe. If it renumbered episodes, the recovery would create a second copy of
    every episode instead of repairing the first."""
    ids = []
    for index in range(3):
        row = add_video(conn, channel["id"], f"vid{index:08d}",
                        upload_date="2026-08-12")
        ids.append(strm.materialise(conn, settings, channel, row)["episode"])
    again = [strm.materialise(conn, settings, channel, videos.get(conn, f"vid{i:08d}"))
             ["episode"] for i in range(3)]
    assert again == ids == [8120, 8121, 8122]
 def test_rematerialising_leaves_no_orphans(conn, settings, media_root, channel,
                                           no_thumbs):
    for index in range(3):
        add_video(conn, channel["id"], f"vid{index:08d}", upload_date="2026-08-12")
    for index in range(3):
        strm.materialise(conn, settings, channel, videos.get(conn, f"vid{index:08d}"))
    for index in range(3):
        strm.materialise(conn, settings, channel, videos.get(conn, f"vid{index:08d}"))
    assert len(list(media_root.rglob("*.strm"))) == 3
    assert len([p for p in media_root.rglob("*.nfo") if p.name != "tvshow.nfo"]) == 3
 def test_a_renamed_episode_removes_its_old_files(conn, settings, media_root,
                                                 channel, no_thumbs):
    """The late-title path renames the file; the old one must not survive."""
    row = add_video(conn, channel["id"], "vid00000001", title="",
                    upload_date="2026-08-12")
    first = strm.materialise(conn, settings, channel, row)
    old = media_root / first["rel_path"]
    assert old.exists()
    with conn:
        conn.execute("UPDATE video SET title = ? WHERE video_id = ?",
                     ("Proper Title", "vid00000001"))
    second = strm.materialise(conn, settings, channel,
                              videos.get(conn, "vid00000001"))
    assert second["rel_path"] != first["rel_path"]
    assert not old.exists()
    assert (media_root / second["rel_path"]).exists()
    assert len(list(media_root.rglob("*.strm"))) == 1
 # ------------------------------------- pruning channels with nothing to show
 #
 # §5: a channel with nothing inside the window must not leave an empty series in
 # Jellyfin. runner creates directories lazily to honour that, but
 # channels.subscribe() mkdirs to write tvshow.nfo and a poster, so every
 # subscription got one regardless. Measured after approving all 119 real
 # subscriptions: 57 empty series.
 def test_prune_removes_a_channel_with_no_episodes(conn, media_root, channel):
    tree = media_root / "clabretro"
    tree.mkdir()
    (tree / "tvshow.nfo").write_text("<tvshow/>")
    (tree / "poster.jpg").write_bytes(b"x" * 100)
    assert strm.prune_if_no_episodes(channel) is True
    assert not tree.exists()
 def test_prune_leaves_a_channel_that_has_episodes(conn, settings, media_root,
                                                  channel, video, no_thumbs):
    strm.materialise(conn, settings, channel, video)
    assert strm.prune_if_no_episodes(channel) is False
    assert (media_root / "clabretro").exists()
 def test_prune_finds_episodes_in_any_season(conn, media_root, channel):
    """The .strm lives a directory down, so a shallow check would delete it."""
    deep = media_root / "clabretro" / "Season 2019"
    deep.mkdir(parents=True)
    (deep / "clabretro - S2019E1010 - Old [dQw4w9WgXcQ].strm").write_text("url")
    assert strm.prune_if_no_episodes(channel) is False
    assert deep.exists()
 def test_prune_is_a_no_op_when_there_is_no_directory(conn, channel):
    assert strm.prune_if_no_episodes(channel) is False
 def test_prune_refuses_a_blank_channel_directory(conn, media_root):
    from conftest import add_channel
    row = add_channel(conn, "UC" + "r" * 22, "Blank", "")
    keep = media_root / "keep"
    keep.mkdir()
    assert strm.prune_if_no_episodes(row) is False
    assert keep.exists()
@@ -207,3 +207,75 @@ def test_deleting_a_channel_cascades_to_its_videos(conn, channel):
    with conn:
        conn.execute("DELETE FROM channel WHERE id = ?", (channel["id"],))
    assert conn.execute("SELECT COUNT(*) FROM video").fetchone()[0] == 0
 # --------------------------------------------------------------- migrations
 def test_a_v1_database_migrates_to_v2(tmp_path):
    """The live database on susan was created at v1. A fresh install must end up
    identical to a migrated one, which is why the description column is a v2
    migration rather than an edit to the v1 script."""
    import sqlite3 as sq
    from ytstream import db
    path = tmp_path / "v1.db"
    raw = sq.connect(path)
    raw.executescript(db._SCHEMA_V1)
    raw.execute("PRAGMA user_version = 1")
    raw.commit()
    raw.close()
    conn = db.connect(path)
    try:
        assert conn.execute("PRAGMA user_version").fetchone()[0] == db.SCHEMA_VERSION
        columns = {row[1] for row in conn.execute("PRAGMA table_info(video)")}
        assert "description" in columns
    finally:
        conn.close()
 def test_migration_is_idempotent(tmp_path):
    from ytstream import db
    path = tmp_path / "twice.db"
    for _ in range(3):
        conn = db.connect(path)
        conn.close()
    conn = db.connect(path)
    try:
        columns = [row[1] for row in conn.execute("PRAGMA table_info(video)")]
        assert columns.count("description") == 1
    finally:
        conn.close()
 def test_fresh_and_migrated_schemas_match(tmp_path):
    """A fresh v2 install and a v1 database brought forward must agree, or the two
    populations diverge silently."""
    import sqlite3 as sq
    from ytstream import db
    fresh = db.connect(tmp_path / "fresh.db")
    fresh_cols = [tuple(row)[1:3] for row in fresh.execute("PRAGMA table_info(video)")]
    fresh.close()
    old = tmp_path / "old.db"
    raw = sq.connect(old)
    raw.executescript(db._SCHEMA_V1)
    raw.execute("PRAGMA user_version = 1")
    raw.commit()
    raw.close()
    migrated = db.connect(old)
    migrated_cols = [tuple(row)[1:3]
                     for row in migrated.execute("PRAGMA table_info(video)")]
    migrated.close()
    assert fresh_cols == migrated_cols
 def test_description_survives_a_round_trip(conn, channel):
    add_video(conn, channel["id"], "vid00000001", description="A synopsis")
    assert videos.get(conn, "vid00000001")["description"] == "A synopsis"
@@ -144,3 +144,75 @@ def test_csrf_token_rejects_tampering():
 def test_csrf_rejects_an_empty_token():
    secret = auth.new_secret()
    assert not auth.verify_csrf(secret, "s", "")
 # --------------------------------------------------- sources / approval queue
 def _source(**kw):
    base = {"key": "youtube:UCbrother", "label": "C Flux",
            "channel_id": "UCPcTWaLV8zwx4WP4QExHj4Q", "enabled": 1, "imported": 1,
            "last_sync_at": "2026-08-12T16:00:00+00:00", "last_sync_ok": 1,
            "consecutive_failures": 0, "last_error": None}
    base.update(kw)
    return base
 def _pending(n=3):
    return [{"id": i, "source": "youtube:UCbrother", "title": f"Channel {i}",
             "channel_id": f"UC{i:022d}", "seen_at": "2026-08-12T16:00:00+00:00"}
            for i in range(1, n + 1)]
 def test_pending_page_renders_the_queue():
    html = templates.pending_page(pending=_pending(3), sources=[_source()],
                                 csrf="tok").decode()
    assert "3 channel(s) waiting" in html
    assert "Channel 1" in html and "Channel 3" in html
    assert "Approve selected" in html
 def test_every_queue_row_shares_the_id_field_name():
    """The approve handler reads a repeated `id` field; if the template numbered
    them uniquely the multi-select would silently approve nothing."""
    html = templates.pending_page(pending=_pending(3), sources=[_source()],
                                 csrf="tok").decode()
    assert html.count('name="id"') == 3
 def test_pending_page_with_an_empty_queue():
    html = templates.pending_page(pending=[], sources=[_source()], csrf="tok").decode()
    assert "Nothing awaiting approval" in html
 def test_pending_page_with_no_sources():
    html = templates.pending_page(pending=[], sources=[], csrf="tok").decode()
    assert "No sources yet" in html
 def test_never_synced_source_is_labelled():
    html = templates.pending_page(pending=[], sources=[_source(last_sync_ok=None)],
                                 csrf="tok").decode()
    assert "never synced" in html
 def test_failing_source_shows_the_actionable_error_in_full():
    """The useful errors say exactly what to do; truncating them defeats the point."""
    message = ('subscriptions are private (subscriptionForbidden) — nothing '
               'changed. Fix: YouTube → Settings → Privacy → uncheck "Keep all '
               'my subscriptions private".')
    html = templates.pending_page(
        pending=[], sources=[_source(last_sync_ok=0, consecutive_failures=3,
                                     last_error=message)],
        csrf="tok").decode()
    assert "failing (3)" in html
    assert "Keep all" in html
 def test_hostile_pending_title_is_escaped():
    items = _pending(1)
    items[0]["title"] = '<img src=x onerror=alert(1)>'
    html = templates.pending_page(pending=items, sources=[_source()],
                                 csrf="tok").decode()
    assert "<img src=x" not in html
    assert "&lt;img" in html
@@ -295,7 +295,8 @@ class Api:
                yield {"video_id": video_id,
                       "published": published,
                       "published_at": exact,
-                       "title": title}, next_token
+                       "title": title,
                       "description": snippet.get("description") or ""}, next_token
                produced += 1
                if limit is not None and produced >= limit:
                    return
@@ -306,22 +307,33 @@ class Api:
    # ---------------------------------------------------------------- durations
-    def durations(self, video_ids: list[str]) -> dict[str, dict]:
+    def details(self, video_ids: list[str]) -> dict[str, dict]:
-        """{video_id: {duration, is_live}} for up to any number of ids.
+        """{video_id: {duration, is_live, title, description}} for any number of ids.
-        Batched 50 per call, so 441 videos costs 9 units. `is_live` comes from the
+        Batched 50 per call, so 441 videos costs 9 units. `snippet` rides along at
-        presence of liveStreamingDetails rather than from the duration, because
+        no extra cost and is the only way to fill in a title or description for a
-        live and upcoming items both report PT0S.
+        video older than the ~15 entries RSS reaches back — which for a 30-day
        window is a real fraction of every channel.
        `is_live` comes from the presence of liveStreamingDetails rather than from
        the duration, because live and upcoming items both report PT0S.
        """
        out: dict[str, dict] = {}
        for start in range(0, len(video_ids), PAGE_SIZE):
            batch = video_ids[start:start + PAGE_SIZE]
-            page = self._get("videos", part="contentDetails,liveStreamingDetails",
+            page = self._get(
-                             id=",".join(batch), maxResults=PAGE_SIZE)
+                "videos", part="snippet,contentDetails,liveStreamingDetails",
                id=",".join(batch), maxResults=PAGE_SIZE)
            for item in page.get("items") or []:
                details = item.get("contentDetails") or {}
                snippet = item.get("snippet") or {}
                out[item["id"]] = {
                    "duration": parse_duration(details.get("duration", "")),
                    "is_live": bool(item.get("liveStreamingDetails")),
                    "title": (snippet.get("title") or "").strip(),
                    "description": snippet.get("description") or "",
                }
        return out
    # Kept as the old name so nothing silently changes meaning mid-refactor.
    durations = details
@@ -360,6 +360,15 @@ def cmd_materialise(args) -> int:
        stats = runner.materialise_all(conn, settings, args.limit)
        print(f"materialised={stats['materialised']} shows={stats['shows']} "
              f"errors={stats['errors']}")
        if stats["materialised"]:
            client = jellyfin.from_settings(settings)
            client.refresh()
            if args.all:
                # A plain scan does not reliably notice a rewritten NFO — after one
                # such rewrite, 50 of 251 episodes kept their old metadata. This
                # forces the re-read, and does not probe.
                print("asking Jellyfin to re-read local metadata...")
                client.refresh_library_metadata(config.MEDIA_ROOT)
        return 1 if stats["errors"] else 0
    finally:
        conn.close()
@@ -375,6 +384,23 @@ def cmd_reap(args) -> int:
        conn.close()
 def cmd_refresh_metadata(args) -> int:
    """Force Jellyfin to re-read every NFO, without probing any media."""
    conn, settings = _open()
    try:
        client = jellyfin.from_settings(settings)
        if not client.configured:
            print("Jellyfin is not configured.", file=sys.stderr)
            return 1
        if not client.refresh_library_metadata(config.MEDIA_ROOT):
            print(f"No Jellyfin library covers {config.MEDIA_ROOT}.", file=sys.stderr)
            return 1
        print("Jellyfin is re-reading local metadata (this does not probe media).")
        return 0
    finally:
        conn.close()
 def cmd_run(args) -> int:
    try:
        with runner.exclusive_lock():
@@ -528,6 +554,10 @@ def build_parser() -> argparse.ArgumentParser:
    sub.add_parser("reap", help="delete videos past the retention window").set_defaults(
        func=cmd_reap)
    sub.add_parser("refresh-metadata",
                   help="make Jellyfin re-read the NFOs (never probes media)"
                   ).set_defaults(func=cmd_refresh_metadata)
    run_cmd = sub.add_parser("run", help="sync, poll, materialise, reap (what cron calls)")
    run_cmd.add_argument("--channel", type=int, help="restrict to one channel")
    run_cmd.set_defaults(func=cmd_run)
@@ -15,7 +15,7 @@ from pathlib import Path
 from . import config
-SCHEMA_VERSION = 1
+SCHEMA_VERSION = 2
 _SCHEMA_V1 = """
 CREATE TABLE IF NOT EXISTS channel (
@@ -121,6 +121,14 @@ CREATE TABLE IF NOT EXISTS setting (
 """
 # v2: the video description, used as the NFO <plot>. Both RSS
 # (media:group/media:description) and playlistItems.list (snippet.description) carry
 # it for free, and without it every episode has an empty synopsis in Jellyfin.
 _SCHEMA_V2 = """
 ALTER TABLE video ADD COLUMN description TEXT;
 """
 def connect(path: Path | None = None) -> sqlite3.Connection:
    """Open the database, applying migrations if needed."""
    path = Path(path) if path is not None else config.DB_PATH
@@ -144,6 +152,8 @@ def migrate(conn: sqlite3.Connection) -> int:
    with conn:
        if current < 1:
            conn.executescript(_SCHEMA_V1)
        if current < 2:
            conn.executescript(_SCHEMA_V2)
        # Future migrations append here, each guarded by `if current < N`.
        conn.execute(f"PRAGMA user_version = {SCHEMA_VERSION}")
    return SCHEMA_VERSION
@@ -100,6 +100,8 @@ def parse_entries(payload: bytes) -> list[dict]:
            {
                "video_id": video_id,
                "title": (entry.findtext("atom:title", "", NS) or "").strip(),
                "description": entry.findtext(
                    "media:group/media:description", "", NS) or "",
                "published": published_date,
                "published_at": published,
            }
@@ -152,6 +154,13 @@ def _record(
        # backfill saw it, for instance. If a later feed supplies the title, take
        # it, and if the episode is already on disk under its video id, remove the
        # files and re-queue so it is rewritten under the real name.
        if not (existing["description"] or "").strip() and entry.get("description"):
            with conn:
                conn.execute(
                    "UPDATE video SET description = ? WHERE video_id = ?",
                    (entry["description"], entry["video_id"]),
                )
        if not (existing["title"] or "").strip() and entry["title"]:
            with conn:
                conn.execute(
@@ -172,6 +181,7 @@ def _record(
        channel_pk=channel["id"],
        video_id=entry["video_id"],
        title=entry["title"],
        description=entry.get("description") or "",
        upload_date=entry["published"].isoformat(),
        published_at=entry.get("published_at"),
        state=state,
@@ -266,7 +276,7 @@ def enrich_durations(
    minimum = settings.get_int("min_duration_seconds")
    client = api.Api(settings.get_str("youtube_api_key"))
    try:
-        found = client.durations(video_ids)
+        found = client.details(video_ids)
    except api.ApiError as exc:
        # Durations are an enrichment, not a gate: a NULL duration costs a runtime
        # display, not a working library.
@@ -275,13 +285,36 @@ def enrich_durations(
    for video_id, info in found.items():
        videos.set_duration(conn, video_id, info["duration"])
        # Fill in title and description only when we do not already have them: the
        # feed is just as good a source and this must not overwrite it.
        existing = videos.get(conn, video_id)
        if existing is not None:
            for field in ("title", "description"):
                if not (existing[field] or "").strip() and info.get(field):
                    with conn:
                        conn.execute(
                            f"UPDATE video SET {field} = ? WHERE video_id = ?",
                            (info[field], video_id),
                        )
        stats["resolved"] += 1
        skip = None
        if info["is_live"]:
-            videos.set_state(conn, video_id, videos.SKIPPED_LIVE)
+            skip, key = videos.SKIPPED_LIVE, "live"
            stats["live"] += 1
        elif info["duration"] is not None and info["duration"] < minimum:
-            videos.set_state(conn, video_id, videos.SKIPPED_SHORT)
+            skip, key = videos.SKIPPED_SHORT, "shorts"
-            stats["shorts"] += 1
+
        if skip:
            # A video can reach this pass having already been materialised — a
            # premiere that turned into a livestream, or a duration lookup that
            # only succeeded on a later run. Setting the state without removing
            # the files would leave them orphaned: on disk for Jellyfin to show,
            # but with no row that admits to owning them.
            if existing is not None and existing["state"] == videos.MATERIALISED:
                strm.remove(existing)
                log.info("%s reclassified as %s after materialising; files removed",
                         video_id, skip)
            videos.set_state(conn, video_id, skip)
            stats[key] += 1
    return stats
@@ -335,6 +368,7 @@ def backfill_channel(
                    # backfill named after their video id, because RSS reaches
                    # back only ~23 days against a 30-day window.
                    title=entry.get("title") or "",
                    description=entry.get("description") or "",
                    upload_date=entry["published"].isoformat(),
                    published_at=entry["published_at"],
                    state=videos.LISTED,
@@ -390,6 +424,62 @@ def rescan_channel(
    return cursor.rowcount
 def top_up_to_min_keep(
    conn: sqlite3.Connection, settings: Settings, channel: sqlite3.Row
 ) -> int:
    """Promote the newest `skipped_old` rows until the channel has `min_keep_videos`
    videos that will end up on disk. Returns how many were promoted.
    §5 defines retention as **`max(retention_days, min_keep_videos newest)`**, and
    `reap.candidates()` implements the second half — but only for videos that were
    materialised in the first place. Discovery judges every video against the
    window alone and tombstones the rest as `skipped_old`, so a channel that has
    not uploaded inside the window materialises *nothing* and appears in Jellyfin
    as an empty series. That is the precise outcome §5 introduced this setting to
    prevent: "keeps its last 5 videos permanently visible instead of showing an
    empty shelf".
    Measured 2026-08-13 on the real 119 subscriptions: 441 videos materialised —
    exactly the in-window count Phase 0 predicted — but **57 channels had zero
    episodes**, and 55 of those were sitting on `skipped_old` rows. §5's own cost
    estimate was `441 + (52 x 5) ~= 700`, so the shortfall was in the code, not in
    the measurement.
    Only `materialised` and `listed` count towards the floor: `skipped_short` and
    `skipped_live` are excluded by policy and can never be episodes, so a channel
    whose five newest uploads are all Shorts correctly reaches further back for
    long-form ones.
    `aged_out` rows are deliberately not revived, for the reason `rescan_channel`
    gives. reap() protects the newest `min_keep_videos` on disk, so anything
    promoted here sits inside its protected set and will not be deleted straight
    back out — the two halves of the rule now agree.
    """
    keep = max(0, settings.get_int("min_keep_videos"))
    if not keep:
        return 0
    have = conn.execute(
        "SELECT count(*) FROM video WHERE channel_pk = ? AND state IN (?, ?)",
        (channel["id"], videos.MATERIALISED, videos.LISTED),
    ).fetchone()[0]
    shortfall = keep - have
    if shortfall <= 0:
        return 0
    with conn:
        cursor = conn.execute(
            "UPDATE video SET state = ? WHERE id IN ("
            "  SELECT id FROM video WHERE channel_pk = ? AND state = ?"
            "  ORDER BY upload_date DESC, id DESC LIMIT ?)",
            (videos.LISTED, channel["id"], videos.SKIPPED_OLD, shortfall),
        )
    if cursor.rowcount:
        log.info("%s: kept %d older video(s) to reach the %d-video floor",
                 channel["title"], cursor.rowcount, keep)
    return cursor.rowcount
 def poll_all(
    conn: sqlite3.Connection, settings: Settings, channel_pk: int | None = None
 ) -> dict:
@@ -400,7 +490,7 @@ def poll_all(
        rows = channels.all_channels(conn)
    totals = {"channels": 0, "queued": 0, "old": 0, "known": 0, "repaired": 0,
-              "titled": 0, "shorts": 0, "live": 0, "failed": 0}
+              "titled": 0, "shorts": 0, "live": 0, "kept": 0, "failed": 0}
    for channel in rows:
        totals["channels"] += 1
        if not channel["backfilled"]:
@@ -416,7 +506,17 @@ def poll_all(
        if "error" in stats:
            totals["failed"] += 1
-        for key in ("queued", "old", "known", "repaired", "titled", "shorts", "live"):
+        else:
            # After the poll, so it sees this run's discoveries and only reaches
            # back for older videos when the window genuinely left the channel
            # short. Skipped on a failed poll: a channel we could not read looks
            # empty, and topping it up from tombstones would be guessing.
            kept = top_up_to_min_keep(conn, settings, channel)
            if kept:
                stats["kept"] = kept
        for key in ("queued", "old", "known", "repaired", "titled", "shorts",
                    "live", "kept"):
            totals[key] += stats.get(key, 0)
        log.info("%s: %s", channel["title"], stats)
    return totals
@@ -138,6 +138,42 @@ class Jellyfin:
        except JellyfinError as exc:
            log.warning("jellyfin refresh failed: %s", exc)
    def reread_local_metadata(self, item_id: str, *, recursive: bool = True) -> None:
        """Re-read NFOs for an item without probing the media.
        A plain `/Library/Refresh` only re-reads an NFO when Jellyfin notices the
        file changed, and it does not always notice: after rewriting 251 NFOs, 50
        of them kept their old (empty) metadata. This is the fix, and it is safe —
        measured 2026-08-12, a recursive refresh of a 56-episode series in this
        mode made **zero** requests to the proxy.
        The safety is entirely in `replaceAllMetadata=false`. With it `true`,
        Jellyfin discards what it has and re-derives from the media, which for a
        .strm library means fetching every episode. Never set it.
        """
        try:
            self._request(
                "POST", f"/Items/{item_id}/Refresh",
                params={
                    "metadataRefreshMode": "Default",
                    "imageRefreshMode": "Default",
                    "replaceAllMetadata": "false",
                    "replaceAllImages": "false",
                    "recursive": "true" if recursive else "false",
                },
            )
        except JellyfinError as exc:
            log.warning("jellyfin metadata re-read failed: %s", exc)
    def refresh_library_metadata(self, path) -> bool:
        """Re-read every NFO under the library covering `path`. Best effort."""
        library = self.find_library(path)
        if not library or not library.get("ItemId"):
            log.warning("no Jellyfin library found for %s", path)
            return False
        self.reread_local_metadata(library["ItemId"])
        return True
 def from_settings(settings) -> Jellyfin:
    return Jellyfin(
@@ -80,6 +80,23 @@ def materialise_all(
    return stats
 def prune_empty_channels(conn: sqlite3.Connection) -> int:
    """Remove directories for subscribed channels that have no episodes to show.
    Runs after materialising and reaping, so it only sees the settled state. See
    `strm.prune_if_no_episodes` for why these directories exist at all.
    """
    pruned = 0
    for channel in conn.execute(
        "SELECT c.* FROM channel c WHERE NOT EXISTS ("
        "  SELECT 1 FROM video v WHERE v.channel_pk = c.id AND v.state = ?)",
        (videos.MATERIALISED,),
    ).fetchall():
        if strm.prune_if_no_episodes(channel):
            pruned += 1
    return pruned
 def run(
    conn: sqlite3.Connection, settings: Settings, channel_pk: int | None = None
 ) -> dict:
@@ -94,6 +111,7 @@ def run(
    result["poll"] = discovery.poll_all(conn, settings, channel_pk)
    result["materialise"] = materialise_all(conn, settings)
    result["reap"] = reap.run(conn, settings)
    result["pruned"] = prune_empty_channels(conn)
    if result["materialise"]["materialised"] or result["reap"]["aged_out"]:
        jellyfin.from_settings(settings).refresh()
@@ -124,6 +142,10 @@ def summarise(result: dict) -> str:
        ]
        if sync.get("refused"):
            parts.append(f"SYNC_REFUSED={sync['refused']}")
    if poll.get("kept"):
        parts.append(f"kept_for_floor={poll['kept']}")
    if result.get("pruned"):
        parts.append(f"pruned_empty={result['pruned']}")
    if made.get("errors"):
        parts.append(f"errors={made['errors']}")
    return " ".join(parts)
@@ -106,9 +106,18 @@ def materialise(
    bytes to the same paths.
    """
    upload_date = naming.parse_upload_date(video["upload_date"])
-    season, episode = videos.next_episode(
+
-        conn, channel["id"], upload_date, video["video_id"]
+    # An episode number, once assigned, is permanent. Re-deriving it would let it
-    )
+    # move: next_episode() looks at the MAX for that day excluding this row, so
    # re-materialising a day's videos in a different order renumbers them — and
    # Jellyfin treats a renumbered episode as a different episode. Reusing the
    # stored pair is what makes `materialise --all` safe to run.
    if video["season"] and video["episode"]:
        season, episode = video["season"], video["episode"]
    else:
        season, episode = videos.next_episode(
            conn, channel["id"], upload_date, video["video_id"]
        )
    # `title` is for the filename; `stored_title` is what goes back to the
    # database. They differ when we have no title yet: writing the video id back
    # would make the row look titled, permanently disabling the repair path in
@@ -120,6 +129,14 @@ def materialise(
        channel, season, episode, title, video["video_id"]
    )
    # If this video already lives somewhere else on disk, remove that copy before
    # writing the new one. A title arriving late changes the filename, and without
    # this the old `.strm`, `.nfo` and thumbnail stay behind — so Jellyfin shows the
    # episode twice and one of the two never gets updated again. Measured: running
    # `materialise --all` once left 102 orphaned NFOs against 251 episodes.
    if video["rel_path"] and video["rel_path"] != relative:
        remove(video)
    strm_path.parent.mkdir(parents=True, exist_ok=True)
    # No trailing newline: some Jellyfin versions have historically been fussy
    # about trailing whitespace in .strm files, and there is nothing to gain.
@@ -134,7 +151,7 @@ def materialise(
            show_title=channel["title"],
            season=season,
            episode=episode,
-            plot=None,
+            plot=video["description"],
            aired=upload_date.isoformat(),
            duration_seconds=video["duration"],
            video_id=video["video_id"],
@@ -190,6 +207,39 @@ def remove(video: sqlite3.Row) -> int:
    return removed
 def prune_if_no_episodes(channel: sqlite3.Row) -> bool:
    """Remove a subscribed channel's directory if it holds no episodes.
    §5 says a channel with nothing inside the window must not leave an empty
    series in Jellyfin, and `runner.materialise_pending` honours that by creating
    the directory lazily, from the first episode. But `channels.subscribe()` calls
    `_write_show_metadata()`, which mkdirs to write tvshow.nfo and the poster —
    so every subscription got a directory whether or not it had anything to show.
    Measured 2026-08-13 after approving all 119 subscriptions: **57 empty series**.
    `top_up_to_min_keep()` fixes 55 of those by keeping older videos. This handles
    the remainder — the channels with no long-form uploads at all, whose UULF
    playlist 404s (2 of the real 119). It runs after materialising, so "no
    episodes" means none were written this run either.
    Only ever removes a directory with no `.strm` in it, so an active channel is
    never touched. `subscribe()` runs once per channel, so a pruned directory does
    not come back every hour — it reappears only when the channel finally uploads
    something and `write_show()` recreates it.
    """
    directory = channel_dir(channel)
    if directory == config.MEDIA_ROOT or not str(channel["dir_name"]).strip():
        log.error("refusing to prune %s: unsafe channel directory", directory)
        return False
    if not directory.is_dir():
        return False
    if any(directory.rglob("*.strm")):
        return False
    shutil.rmtree(directory)
    log.info("pruned %s: subscribed but no episodes to show", directory)
    return True
 def remove_channel_tree(channel: sqlite3.Row) -> bool:
    """Delete a whole channel directory, on unsubscribe.
@@ -66,17 +66,19 @@ def insert(
    discovery_source: str,
    duration: int | None = None,
    published_at: str | None = None,
    description: str | None = None,
 ) -> None:
    with conn:
        conn.execute(
            "INSERT OR IGNORE INTO video "
-            "(video_id, channel_pk, title, upload_date, published_at, duration, "
+            "(video_id, channel_pk, title, description, upload_date, published_at, "
-            " state, discovery_source, discovered_at) "
+            " duration, state, discovery_source, discovered_at) "
-            "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
+            "VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
            (
                video_id,
                channel_pk,
                title,
                description,
                upload_date,
                published_at,
                duration,
@@ -16,7 +16,18 @@ import threading
 import urllib.parse
 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
-from .. import channels, config, db, discovery, jellyfin, util, videos, ytdlp
+from .. import (
    api,
    channels,
    config,
    db,
    discovery,
    jellyfin,
    subsync,
    util,
    videos,
    ytdlp,
 )
 from ..settings import EDITABLE, MASKED_KEYS, Settings, validate_all
 from . import auth, templates
@@ -77,15 +88,32 @@ class Handler(BaseHTTPRequestHandler):
        self.end_headers()
        self.wfile.write(body)
    def _parsed_form(self) -> dict[str, list[str]]:
        """Parse the body once and cache it.
        Cached because the request body can only be read from rfile once, and the
        approval queue needs both the single-value and multi-value views of it.
        """
        if getattr(self, "_form_cache", None) is None:
            length = int(self.headers.get("Content-Length") or 0)
            if length <= 0 or length > MAX_BODY:
                self._form_cache = {}
            else:
                raw = self.rfile.read(length).decode("utf-8", "replace")
                self._form_cache = urllib.parse.parse_qs(raw, keep_blank_values=True)
        return self._form_cache
    def _form(self) -> dict[str, str]:
-        length = int(self.headers.get("Content-Length") or 0)
+        """Last value wins, which is right for every single-value field."""
-        if length <= 0 or length > MAX_BODY:
+        return {key: values[-1] for key, values in self._parsed_form().items()}
-            return {}
+
-        raw = self.rfile.read(length).decode("utf-8", "replace")
+    def _form_list(self, key: str) -> list[str]:
-        return {
+        """Every value for a repeated field.
-            key: values[-1]
+
-            for key, values in urllib.parse.parse_qs(raw, keep_blank_values=True).items()
+        The approval queue is a form of checkboxes all named `id`. Reading it
-        }
+        through _form() would silently approve only the last box ticked.
        """
        return [value for value in self._parsed_form().get(key, []) if value]
    def _cookie_token(self) -> str:
        return auth.cookie_value(self.headers.get("Cookie") or "")
@@ -141,6 +169,10 @@ class Handler(BaseHTTPRequestHandler):
            if path == "/":
                return self._send(200, self._render_index(conn, settings, token))
            if path == "/pending":
                return self._send(
                    200, self._render_pending(conn, settings, token))
            return self._send(404, templates.page("Not found", "<h1>Not found</h1>"))
        finally:
            conn.close()
@@ -204,6 +236,14 @@ class Handler(BaseHTTPRequestHandler):
                return self._add_channel(conn, settings, token, form)
            if path == "/settings":
                return self._save_settings(conn, settings, token, form)
            if path == "/sources":
                return self._add_source(conn, settings, token, form)
            if path == "/sync":
                return self._sync_now(conn, settings)
            if path == "/pending/approve":
                return self._resolve_pending(conn, settings, form, "approved")
            if path == "/pending/reject":
                return self._resolve_pending(conn, settings, form, "rejected")
            parts = path.strip("/").split("/")
            if len(parts) == 3 and parts[0] == "channels" and parts[1].isdigit():
@@ -219,6 +259,59 @@ class Handler(BaseHTTPRequestHandler):
        finally:
            conn.close()
    # ------------------------------------------------- subscription sources
    def _render_pending(self, conn, settings: Settings, token: str) -> bytes:
        return templates.pending_page(
            pending=[dict(row) for row in subsync.pending(conn)],
            sources=[dict(row) for row in subsync.all_sources(conn)],
            csrf=auth.csrf_token(self._secret(settings), token),
        )
    def _add_source(self, conn, settings: Settings, token: str, form: dict) -> None:
        reference = (form.get("channel") or "").strip()
        if not reference:
            return self._redirect("/pending")
        client = api.Api(settings.get_str("youtube_api_key"))
        try:
            info = (client.channel(reference) if reference.startswith("UC")
                    else client.resolve_handle(reference))
        except api.ApiError as exc:
            log.error("could not resolve source %s: %s", reference, exc)
            return self._redirect("/pending")
        if not info:
            return self._redirect("/pending")
        subsync.add_source(conn, channel_id=info["channel_id"], label=info["title"])
        return self._redirect("/pending")
    def _sync_now(self, conn, settings: Settings) -> None:
        """Run a sync from the UI.
        Synchronous, and that is deliberate: it is one API call per 50
        subscriptions and the result is what the operator is about to look at.
        A background job would mean rendering a page that does not yet reflect
        the button that was just pressed.
        """
        try:
            subsync.sync_all(conn, settings)
        except Exception as exc:  # noqa: BLE001
            log.error("sync from the UI failed: %s", exc)
        return self._redirect("/pending")
    def _resolve_pending(self, conn, settings: Settings, form: dict,
                         resolution: str) -> None:
        ids = [int(value) for value in self._form_list("id") if value.isdigit()]
        if not ids:
            return self._redirect("/pending")
        if resolution == "approved":
            subsync.approve(conn, settings, ids)
        else:
            subsync.resolve(conn, ids, "rejected")
        return self._redirect("/pending")
    def _login(self, settings: Settings, form: dict) -> None:
        key = self._client_key()
        if self.server.throttle.locked(key):
@@ -251,3 +251,93 @@ def index_page(
 <footer>Downloads run hourly. Videos are deleted once they pass the retention
 window for their channel — this is a DVR, not an archive.</footer>"""
    return page("ytstream", body)
 # --------------------------------------------------------------------------
 # subscription sources and the approval queue
 def _source_panel(source: dict, csrf: str) -> str:
    if source["last_sync_ok"] is None:
        state = '<span class="badge">never synced</span>'
    elif source["last_sync_ok"]:
        state = '<span class="badge ok">ok</span>'
    else:
        state = (f'<span class="badge bad">failing '
                 f'({source["consecutive_failures"]})</span>')
    error = ""
    if source["last_error"]:
        # Shown in full rather than truncated: the useful ones say exactly what to
        # do ("subscriptions are private — uncheck Keep all my subscriptions
        # private"), and hiding that behind a log file defeats the point.
        error = f'<p class="muted">{_e(source["last_error"])}</p>'
    return f"""
 <div class="panel">
  <strong>{_e(source['label'])}</strong> {state}
  <p class="muted">{_e(source['channel_id'])} · last sync
     {_e(source['last_sync_at'] or 'never')}
     · first import {'done' if source['imported'] else 'pending'}</p>
  {error}
 </div>"""
 def pending_page(*, pending: list[dict], sources: list[dict], csrf: str) -> bytes:
    """The approval queue.
    A separate page rather than a section on the index because the first sync of a
    real account queued 119 channels, and that does not belong inline underneath
    the channel table.
    """
    source_html = "".join(_source_panel(source, csrf) for source in sources) or (
        '<p class="muted">No sources yet.</p>'
    )
    if pending:
        rows = "".join(f"""
 <tr>
  <td><input type="checkbox" name="id" value="{item['id']}" id="p{item['id']}"></td>
  <td><label for="p{item['id']}">{_e(item['title'])}</label></td>
  <td class="muted hide">{_e(item['channel_id'])}</td>
 </tr>""" for item in pending)
        queue = f"""
 <form method="post" id="queue">
  <input type="hidden" name="csrf" value="{_e(csrf)}">
  <p class="muted">{len(pending)} channel(s) waiting. Approving a channel
     backfills its recent videos and starts polling it; rejecting one means it is
     never offered again.</p>
  <table>
    <thead><tr><th></th><th>Channel</th><th class="hide">Channel id</th></tr></thead>
    <tbody>{rows}</tbody>
  </table>
  <div style="margin-top:.8rem">
    <button type="submit" formaction="/pending/approve">Approve selected</button>
    <button type="submit" formaction="/pending/reject" class="danger">
      Reject selected</button>
  </div>
 </form>"""
    else:
        queue = '<p class="muted">Nothing awaiting approval.</p>'
    body = f"""
 <h1>ytstream</h1>
 <nav class="muted"><a href="/">Channels</a> · <strong>Subscriptions</strong></nav>
 <h2>Mirrored accounts</h2>
 {source_html}
 <form method="post" action="/sources" class="row">
  <input type="hidden" name="csrf" value="{_e(csrf)}">
  <input type="text" name="channel" placeholder="@handle or UC... id"
         style="min-width:18rem">
  <button type="submit">Add account</button>
 </form>
 <form method="post" action="/sync" style="margin-top:.6rem">
  <input type="hidden" name="csrf" value="{_e(csrf)}">
  <button type="submit">Sync now</button>
 </form>
 <h2>Awaiting approval</h2>
 {queue}
 """
    return page("Subscriptions — ytstream", body)
Author	SHA1	Message	Date
Claude	04721a78ea	Measure parallel fetching: it does not help The ceiling is the internet connection at ~8 MB/s. Parallel chunks buy 10-30%, concurrency across different videos buys nothing, and --concurrent-fragments is inapplicable because these formats carry no fragments at all. yt-dlp is already doing the thing that matters: the formats advertise http_chunk_size=10485760 and that chunking is worth 13x, because a single long range request gets throttled to 0.60 MB/s. Also tested option 3 while here: NFO <streamdetails> is ignored for episodes -- RunTimeTicks stays null and MediaStreams empty -- so Jellyfin cannot be talked out of transcoding that way either. The useful number: download is 8 MB/s against a 0.39 MB/s playback bitrate, 20x headroom. Streaming while downloading was never bandwidth bound; it failed purely on Jellyfin's probe decision. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-13 12:08:33 +01:00
Claude	22d8828080	Never serve a partial file: it makes Jellyfin transcode The 12s first-byte grace was the wrong trade and a real play found it within the hour. A 2-hour upload took 3 minutes to start, played 6 seconds, and stalled. Jellyfin had run ffmpeg with -probesize 1G against the growing stream and then transcoded to HLS with libx264. The cause is the container. A fragmented MP4 with empty_moov has no duration in its header, so the only way to get one is to sum every fragment -- probing a growing file reads all of it. Jellyfin cannot establish duration, codec or bitrate, so it abandons direct play and transcodes a stream it also cannot seek. It was targeting 4.83 Mbps against a source measured at 3.29: re-encoding a stream that already fit, because it could not measure it. The same video once complete reports SupportsDirectPlay with the exact runtime and bitrate. So FIRST_BYTE_GRACE defaults to infinite again, with --wait-timeout raised to 600s for a 2-hour upload. A cold long video is slow to start, which is accepted: the fetch outlives the request so a retry is instant, and a retryable stall beats a transcode that wastes a gigabyte and cannot work. Both failure modes are recorded at the constant in the order measured so the 12s cap is not reintroduced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-13 11:32:12 +01:00
Claude	c12837cbb9	Record the completed decommission Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-13 11:10:24 +01:00
Claude	28d0e83130	Fix the Jellyfin rename call: it takes a name, not an id POST /Library/VirtualFolders/Name is the odd one out in that controller -- most of /Library/VirtualFolders/* takes an id, and passing one here returns a bare "HTTP 400: Error processing request." that says nothing about why. Verified against 10.11.4: name -> 204. Also records that renaming re-ids the library, because Jellyfin derives the ItemId from the name. ytstream is unaffected because find_library matches on path -- confirmed by a full refresh-metadata over 257 NFOs with 0 proxy requests straight after the rename. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-13 11:05:28 +01:00
Claude	89c3644d57	Script the Jellyfin library retirement Matches libraries by path, never by name, and reads the name to delete back from the API rather than assuming it: the endpoint takes a name, matches loosely on some versions, and "YouTube" is a prefix of "YouTube (stream)". Refuses to retire the old library unless ytstream's has episodes, so a broken replacement cannot leave the server with no YouTube library. Dry run by default. Also records that /opt/youtube-automate is verified fully pushed to its bare repo -- clean tree, specs.md tracked -- so it is safe to delete, while /var/lib/youtube-automate holds subs.db, which is state and not in the repo. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-13 11:00:47 +01:00
Claude	c7a80d200e	Hand cron over from youtube-automate, and script the rest Cron now calls `ytstream run` hourly and ytstream's update-ytdlp.sh weekly. Both inherit youtube-automate's healthchecks UUIDs and keep its schedules unchanged: a check may be configured with a cron expression rather than a simple period, so moving to the :23/04:50 slots the fragment proposed could have alerted on a job that ran fine. Inheriting also means the placeholder UUIDs never needed filling in. The old service turned out to track only 2 channels, and one of them -- The Pyramid Podcast -- was sitting unresolved in ytstream's approval queue. Decommissioning without checking would have silently dropped half of what the old service existed to follow. Approved and backfilled. decommission.sh does the two steps needing root (nginx repoint, disable the unit) and refuses until an admin password is set, because the UI fails closed and the hostname would otherwise serve a login nobody can pass. Deleting the 2.0 GB of old downloads and touching the Jellyfin libraries are left out on purpose. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-13 10:53:48 +01:00
Claude	b35b6a3689	Clear the stranded tmpfs cache at startup The session map is memory-only, so session directories surviving a restart can never be served and never be evicted -- and the work root is a tmpfs, so that is leaked RAM until reboot. The restart that ships the TTFB fix would have stranded 1.56 GB. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-13 10:34:08 +01:00
Claude	d3616e7532	Bound time-to-first-byte so a first play actually plays Playback failed in Jellyfin the day after deployment, for uncached videos only. The handler blocked on the full download-and-mux before sending anything at all -- not even response headers -- so a 46-minute upload sat silent for 79 seconds and the client gave up. The proxy counted it a success, which is why /healthz and doctor both looked fine. Yesterday's "DirectPlay verified" only ever ran against videos already pulled during testing, so the first-play path was never exercised. The output is already a fragmented MP4, so it is readable while being written; a finished file only buys a correct duration and working seeks. Cap the wait at --first-byte-grace (12s, explicit in the unit) and stream whatever has not muxed by then. Measured: 156s -> 12.0s TTFB on a 66-minute upload, with ranges on a complete file unchanged. The cost is a first play with no seek bar when the grace is missed. Every later play of that video is perfect. Also fixes a hang found while testing this: the streaming loop waited on "finished and good" rather than "finished", so a producer that died after writing some bytes held the connection for the full 45s stall timeout and then dropped it. The stale yt-dlp warning pointed at youtube-automate's venv, the tree we are decommissioning; it now names ytstream's. /healthz reports the serving mode and grace. Eight new proxy tests, 353 passing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-13 10:31:47 +01:00
Tom FluxandClaude Opus 5	d3bf8d6f19	Deploy it, and fix the six things installation found Both units are installed and running, 10 of 119 channels approved, 251 episodes live in Jellyfin with verified DirectPlay. 345 tests. Six problems surfaced that no test could have, and two of them were mine in the deploy scripts. deploy.sh had a circular dependency with bootstrap.sh: deploy started the units and told the operator to run bootstrap, but bootstrap refused to run until the state directory existed, which only deploy creates. The units started against a non-existent venv, failed 203/EXEC and restart-looped 17 and 21 times. deploy.sh now creates the directory, calls bootstrap itself through runuser so the venv is not left root-owned, and refuses to start units when the venv is still missing. Deno was absent, and `doctor` is the only reason we know. It is mandatory rather than nice-to-have — without a JS runtime yt-dlp cannot solve the n challenge, which youtube-automate measured on this machine as 22 formats instead of 29 plus throttling. Nothing else would have complained; playback would just have quietly degraded. bootstrap.sh now installs it and asserts yt-dlp reports it. Episodes had no synopsis at all, because materialise passed plot=None while both sources hand us descriptions for free. Now plumbed through from RSS (media:group/media:description) and from videos.list, which carries snippet.description in the call already being made for durations — so the ~40% of episodes older than RSS reaches get one too. That needed a schema v2 migration; v1 was left exactly as shipped so a fresh install and a migrated one are identical, and a test asserts it. `materialise --all` — the documented recovery from a Jellyfin metadata wipe — was itself creating duplicates. Episode numbers were re-derived each run, and next_episode() excludes the row being numbered, so re-materialising a day's videos in a different order renumbered them and orphaned the old files. One run left 102 orphaned NFOs against 251 episodes. An episode number is now permanent once assigned, and a video whose rel_path changes has its old files removed first. Running it twice is now a no-op. Two Jellyfin behaviours worth having in writing. It ignores <runtime> and <durationinseconds> for episodes while reading the rest of the NFO happily, so a .strm shows no duration until first played — not fixable without probing, which is the one thing this design exists to avoid. And a plain /Library/Refresh does not reliably re-read a rewritten NFO: after rewriting all 251, fifty kept their old empty metadata. The fix is metadataRefreshMode=Default with replaceAllMetadata=false, which took plots from 201 to 251 while the proxy served zero requests. §5's prohibition on replaceAllMetadata=true still stands — that one probes. Exposed as `ytstream refresh-metadata` and run automatically after `materialise --all`. The measurement §5 has been waiting for: a full Jellyfin scan of 251 .strm files took ~119 s, about 8 minutes per 1,000 episodes, and made zero media probes. That last number is the fact the whole design rests on, now confirmed at scale on the real library rather than on seven PoC files. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-12 17:21:35 +01:00
Tom FluxandClaude Opus 5	61cc1672ec	Admin routes for the subscription queue, and record the build in plan.md Phase 4's acceptance criterion is that my brother approves the first import himself, which needs a UI, so /pending now carries source management, sync-now, and multi-select approve/reject. Driven over real HTTP rather than only through the templates: unauthenticated requests redirect to login, all four new routes reject a missing or forged CSRF token, and the live account rendered 117 checkboxes. Approving three at once added three channels, which is the point of the fix underneath. _form() collapses repeated fields to the last value, which is correct for every single-value field but silently wrong for a form of checkboxes all named `id` — it would have approved only the last box ticked. Added _form_list(), with the parsed body cached because rfile can only be read once and the approval path needs both views of it. The approval page is deliberately its own page rather than a section on the index: the first sync of the real account queued 119 channels, and that does not belong inline under the channel table. Source errors are shown in full rather than truncated, because the useful ones say exactly what to do — "subscriptions are private, uncheck Keep all my subscriptions private" — and hiding that behind a log file defeats the purpose of surfacing it. plan.md §13 now reflects what is actually built rather than what was intended, and a new §17 records the three bugs the build turned up, including which of them a test caught and which two needed real data. 337 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-08-12 16:41:47 +01:00