Build ytstream: catalogue, retention, subscription mirror, proxy

Phases 1-4 of plan.md §13. Forked from youtube-automate as planned rather than
written from scratch: naming, NFO, auth, the admin UI, settings and the DB layer
came across largely unchanged, download.py is gone, and the pieces that only make
sense for a streaming library are new.

  api.py       YouTube Data API v3 client. The whole metadata path.
  strm.py      Materialising: a .strm, an .nfo and a thumbnail. Replaces the
               330-line download.py, because the job is writing a URL to a file.
  subsync.py   The subscription mirror, most of which is refusals.
  reap.py      Retention, rewritten around the 30-day window and min_keep_videos.
  discovery.py RSS polling plus an API-backed, resumable, bounded backfill.
  proxy/       The verified PoC, moved in with a systemd unit.

330 tests, all passing, no network and no yt-dlp in any of them. The suite leans
towards the failure paths, because that is where this design can actually hurt
someone: a 403 that looks like an unsubscribe, a video that ages out and comes
back, a title that never arrives. tests/test_proxy.py replaces the two standalone
scripts under proxy/ and now drives the real make_handler(mgr, ...) rather than the
PoC's make_handler(path, done), so routing and video-id validation are covered too.

Ran it end to end against the live API and it found three real bugs.

The first was mine and the tests caught it: strm.remove pruned empty directories
up to the media root, so a channel directory whose tvshow.nfo happened to be
missing would be deleted along with the season. It only looked safe because
tvshow.nfo normally stops the walk. The prune boundary is now the channel
directory explicitly.

The other two only showed up against real data, and they compounded. The backfill
inserted rows with no title and left the RSS poll to fill them in — but RSS returns
15 entries, which for Pitch Side spans 23 days against a 30-day window, so five of
twenty episodes were named after their video ids. Worse, strm.materialise wrote
that fallback back to the database as the title, which made the row look titled and
permanently disabled the repair path. Both fixed: playlistItems.list now requests
snippet as well as contentDetails, which costs the same single quota unit and
carries the title alongside the exact publish date, and the fallback is used for the
filename without being persisted. A title that does arrive late now also removes the
badly-named files and re-queues, so the episode is rewritten rather than keeping its
video-id name forever. Verified against the live API: all twenty Pitch Side episodes
now carry real titles.

Measured on the real account: 119 subscriptions queued for approval and none added
on the first sync, then a two-channel run backfilled and materialised 26 episodes in
under seven seconds.

Two deliberate departures from plan.md, both recorded there:

min_keep_videos defaults to 5 rather than being left as an open question. Without
it 52 of 117 measured channels are empty Jellyfin series that flicker in and out as
their single video crosses the retention line, and the plan already recommended it.

The Jellyfin refresh is a bare /Library/Refresh with a comment explaining why it
must stay that way. A normal scan makes zero media probes; FullRefresh does probe,
and at 400 episodes that is 400 cold starts.

Not yet done: no systemd units are installed (needs root — deploy/deploy.sh), the
admin UI has no routes for sources or the approval queue yet, and nothing has been
pointed at the real media root.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

tests/test_runner.py

@@ -0,0 +1,232 @@
+"""Orchestration: ordering, the lock, and what turns the cron check red."""
+
+from __future__ import annotations
+
+import pytest
+
+from ytstream import config, discovery, jellyfin, reap, runner, subsync, videos
+
+from conftest import add_video
+
+
+@pytest.fixture()
+def quiet_jellyfin(monkeypatch):
+    """Jellyfin refresh is best effort; record calls instead of making them."""
+    calls = []
+    monkeypatch.setattr(jellyfin.Jellyfin, "refresh", lambda self: calls.append(1))
+    return calls
+
+
+@pytest.fixture()
+def no_thumbs(monkeypatch):
+    from ytstream import strm
+
+    monkeypatch.setattr(strm, "fetch_thumbnail", lambda *a, **k: False)
+
+
+# ----------------------------------------------------------------------- lock
+
+
+def test_lock_is_exclusive(tmp_path):
+    path = tmp_path / "run.lock"
+    with runner.exclusive_lock(path):
+        with pytest.raises(runner.AlreadyRunning):
+            with runner.exclusive_lock(path):
+                pass  # pragma: no cover
+
+
+def test_lock_is_released_after_use(tmp_path):
+    path = tmp_path / "run.lock"
+    with runner.exclusive_lock(path):
+        pass
+    with runner.exclusive_lock(path):
+        pass
+
+
+def test_lock_is_released_even_when_the_body_raises(tmp_path):
+    path = tmp_path / "run.lock"
+    with pytest.raises(ValueError):
+        with runner.exclusive_lock(path):
+            raise ValueError("boom")
+    with runner.exclusive_lock(path):
+        pass
+
+
+# --------------------------------------------------------------- materialising
+
+
+def test_materialise_all_writes_everything_listed(conn, settings, media_root,
+                                                  channel, no_thumbs):
+    for index in range(3):
+        add_video(conn, channel["id"], f"vid{index:08d}")
+
+    stats = runner.materialise_all(conn, settings)
+
+    assert stats["materialised"] == 3
+    assert stats["errors"] == 0
+    assert videos.queue_depth(conn) == 0
+
+
+def test_tvshow_nfo_is_written_once_per_channel(conn, settings, media_root,
+                                                channel, no_thumbs):
+    for index in range(3):
+        add_video(conn, channel["id"], f"vid{index:08d}")
+
+    stats = runner.materialise_all(conn, settings)
+
+    assert stats["shows"] == 1
+    assert (media_root / "clabretro" / "tvshow.nfo").exists()
+
+
+def test_channel_with_nothing_in_the_window_creates_no_directory(
+    conn, settings, media_root, channel, no_thumbs
+):
+    """52 of 117 measured channels are in this state, and an empty series in
+    Jellyfin looks like a bug rather than a quiet channel."""
+    add_video(conn, channel["id"], "vid00000001", state=videos.SKIPPED_OLD)
+
+    runner.materialise_all(conn, settings)
+
+    assert not (media_root / "clabretro").exists()
+
+
+def test_materialise_all_honours_the_limit(conn, settings, media_root, channel,
+                                           no_thumbs):
+    for index in range(5):
+        add_video(conn, channel["id"], f"vid{index:08d}")
+    stats = runner.materialise_all(conn, settings, limit=2)
+    assert stats["materialised"] == 2
+
+
+def test_materialise_skips_a_video_whose_channel_vanished(conn, settings,
+                                                         media_root, channel,
+                                                         no_thumbs):
+    add_video(conn, channel["id"], "vid00000001")
+    # Simulate the channel being unsubscribed between claiming and writing, with
+    # foreign keys off so the row survives to be found.
+    conn.execute("PRAGMA foreign_keys = OFF")
+    with conn:
+        conn.execute("DELETE FROM channel WHERE id = ?", (channel["id"],))
+
+    stats = runner.materialise_all(conn, settings)
+
+    assert stats["materialised"] == 0
+    assert stats["errors"] == 0
+
+
+# ------------------------------------------------------------------ full cycle
+
+
+def test_run_order_is_sync_poll_materialise_reap(conn, settings, media_root,
+                                                 channel, monkeypatch,
+                                                 quiet_jellyfin):
+    order = []
+    monkeypatch.setattr(subsync, "sync_all",
+                        lambda *a: order.append("sync") or {"added": 0, "queued": 0,
+                                                            "removed": 0, "refused": 0,
+                                                            "sources": 0,
+                                                            "pending_removal": 0})
+    monkeypatch.setattr(discovery, "poll_all",
+                        lambda *a: order.append("poll") or {"queued": 0})
+    monkeypatch.setattr(runner, "materialise_all",
+                        lambda *a, **k: order.append("materialise") or
+                        {"materialised": 0, "shows": 0, "errors": 0})
+    monkeypatch.setattr(reap, "run",
+                        lambda *a: order.append("reap") or {"aged_out": 0})
+
+    runner.run(conn, settings)
+
+    assert order == ["sync", "poll", "materialise", "reap"]
+
+
+def test_single_channel_run_skips_the_sync(conn, settings, channel, monkeypatch,
+                                           quiet_jellyfin):
+    """A targeted run is an operator action, not a mirror pass."""
+    monkeypatch.setattr(subsync, "sync_all",
+                        lambda *a: pytest.fail("sync should not run"))
+    monkeypatch.setattr(discovery, "poll_all", lambda *a: {"queued": 0})
+    monkeypatch.setattr(reap, "run", lambda *a: {"aged_out": 0})
+
+    result = runner.run(conn, settings, channel_pk=channel["id"])
+
+    assert "sync" not in result
+
+
+def test_run_records_last_run_at(conn, settings, channel, monkeypatch,
+                                 quiet_jellyfin):
+    monkeypatch.setattr(subsync, "sync_all", lambda *a: {"added": 0, "queued": 0,
+                                                         "removed": 0, "refused": 0,
+                                                         "sources": 0,
+                                                         "pending_removal": 0})
+    monkeypatch.setattr(discovery, "poll_all", lambda *a: {"queued": 0})
+    monkeypatch.setattr(reap, "run", lambda *a: {"aged_out": 0})
+
+    runner.run(conn, settings)
+
+    assert settings.raw("last_run_at")
+
+
+def test_jellyfin_is_refreshed_only_when_the_tree_changed(
+    conn, settings, media_root, channel, monkeypatch, quiet_jellyfin, no_thumbs
+):
+    monkeypatch.setattr(subsync, "sync_all", lambda *a: {"added": 0, "queued": 0,
+                                                         "removed": 0, "refused": 0,
+                                                         "sources": 0,
+                                                         "pending_removal": 0})
+    monkeypatch.setattr(discovery, "poll_all", lambda *a: {"queued": 0})
+
+    runner.run(conn, settings)
+    assert quiet_jellyfin == []
+
+    add_video(conn, channel["id"], "vid00000001")
+    runner.run(conn, settings)
+    assert len(quiet_jellyfin) == 1
+
+
+# ------------------------------------------------------------------ exit codes
+
+
+def test_refused_sync_turns_the_check_red():
+    """A silently-broken mirror is the worst outcome available: nothing looks
+    wrong until someone asks why a channel never appeared."""
+    assert runner.exit_code({"sync": {"refused": 1}}) == 1
+
+
+def test_materialise_errors_turn_the_check_red():
+    assert runner.exit_code({"materialise": {"errors": 2}}) == 1
+
+
+def test_poll_failures_alone_do_not_turn_the_check_red():
+    """Two of 119 measured channels fail permanently — terminated or private."""
+    assert runner.exit_code({"poll": {"failed": 2}}) == 0
+
+
+def test_clean_run_is_zero():
+    assert runner.exit_code({"sync": {"refused": 0}, "poll": {"failed": 0},
+                             "materialise": {"errors": 0}}) == 0
+
+
+# ------------------------------------------------------------------- summarise
+
+
+def test_summarise_mentions_a_refusal():
+    text = runner.summarise({
+        "sync": {"added": 0, "queued": 0, "removed": 0, "refused": 1},
+        "poll": {"channels": 5, "queued": 0},
+        "materialise": {"materialised": 0},
+        "reap": {"aged_out": 0},
+    })
+    assert "SYNC_REFUSED" in text
+
+
+def test_summarise_is_a_single_line():
+    text = runner.summarise({
+        "sync": {"added": 1, "queued": 2, "removed": 3, "refused": 0},
+        "poll": {"channels": 119, "queued": 4, "shorts": 5, "live": 6, "failed": 2},
+        "materialise": {"materialised": 7, "errors": 0},
+        "reap": {"aged_out": 8},
+    })
+    assert "\n" not in text
+    assert "channels=119" in text
+    assert "materialised=7" in text
+    assert "aged_out=8" in text