Admin routes for the subscription queue, and record the build in plan.md

Phase 4's acceptance criterion is that my brother approves the first import
himself, which needs a UI, so /pending now carries source management, sync-now, and
multi-select approve/reject. Driven over real HTTP rather than only through the
templates: unauthenticated requests redirect to login, all four new routes reject a
missing or forged CSRF token, and the live account rendered 117 checkboxes.

Approving three at once added three channels, which is the point of the fix
underneath. _form() collapses repeated fields to the last value, which is correct
for every single-value field but silently wrong for a form of checkboxes all named
`id` — it would have approved only the last box ticked. Added _form_list(), with the
parsed body cached because rfile can only be read once and the approval path needs
both views of it.

The approval page is deliberately its own page rather than a section on the index:
the first sync of the real account queued 119 channels, and that does not belong
inline under the channel table. Source errors are shown in full rather than
truncated, because the useful ones say exactly what to do — "subscriptions are
private, uncheck Keep all my subscriptions private" — and hiding that behind a log
file defeats the purpose of surfacing it.

plan.md §13 now reflects what is actually built rather than what was intended, and a
new §17 records the three bugs the build turned up, including which of them a test
caught and which two needed real data. 337 tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

plan.md

@@ -1,9 +1,11 @@
 # `ytstream` — implementation plan
 
 **Target machine:** `susan`
-**Status:** streaming PoC verified end to end against real videos and real Jellyfin — every
-measurement behind this plan is written up in **`FINDINGS.md`** alongside this file. Nothing is
-installed as a service yet. This document is the plan for turning it into one.
+**Status:** **built.** Phases 0–4 of §13 are code-complete with 337 passing tests, verified against
+the live YouTube Data API and the running proxy. What remains is installation, which needs root
+(`sudo deploy/deploy.sh`), and the cut-over in §12. The streaming PoC measurements this plan was
+designed around are in **`FINDINGS.md`** alongside this file; what the build itself changed is in
+**§17**.
 
 **Repo:** `/opt/ytstream`, pushed to `/disks/git-repos/ytstream.git`, branch `main`. The PoC code
 still lives in `/home/susan/ytstream` and is *not* under version control; Phase 2 moves it in and
@@ -739,30 +741,36 @@ Each phase ends in something checkable. Do not start the next one until it does.
 pass; `tools/verify_api.py` reproduces them in one command. Results and the numbers they changed are
 in §16.
 
-**Phase 1 — skeleton and lift.** Fork the tree, new package name, new DB path, new schema (§7),
-lifted modules and their tests passing. No new behaviour.
-→ *Done when: `pytest` is green and `ytstream doctor` reports a healthy environment.*
+**Phase 1 — skeleton and lift. ✅ COMPLETE 2026-08-12.** Forked, renamed, new schema, 337 tests
+green. Lifted: `naming`, `nfo`, `db`, `settings`, `util`, `config`, `channels`, `jellyfin`, `doctor`,
+`ytdlp`, `web/`. Deleted `download.py`. See §17 for what changed on the way through.
 
-**Phase 2 — the proxy as a service.** Move it in, split it up, add the startup sweep and the LRU
-test, write `deploy/deploy.sh`, operator runs it.
-→ *Done when: `systemctl status ytstream-proxy` is active after a reboot, `/healthz` answers, and
-Jellyfin direct-plays a cold video end to end.*
+**Phase 2 — the proxy as a service. ⏳ CODE DONE, INSTALL PENDING.** Moved to
+`proxy/ytstream_proxy.py`; its two standalone test scripts are now `tests/test_proxy.py`, driving the
+real `make_handler(mgr, …)` so routing and video-id validation are covered too. `deploy/` carries both
+systemd units, `bootstrap.sh` and `deploy.sh`.
+→ *Blocked on root: the operator must run `sudo /opt/ytstream/deploy/deploy.sh`. Until then the
+PoC-era proxy on 8099 is what serves playback.*
 
-**Phase 3 — catalogue and retention.** `api.py`, `strm.py`, the 30-day bounded resumable backfill, the
-aging-out sweep with its tombstones, the hourly run. Build **Pitch Side alone** (expect ~18 episodes)
-and time a Jellyfin scan for the §5 record.
-→ *Done when: the expected episode count is visible with correct titles, dates, durations and
-thumbnails; one of them plays; scan time per 1,000 episodes is recorded in §5; and — the part most
-likely to be wrong — a video forced past the window is deleted from disk, and the **next two polls do
-not bring it back**.*
+**Phase 3 — catalogue and retention. ✅ CODE COMPLETE, verified against the live API.** Pitch Side
+backfilled to **20 episodes** (the §5 estimate was ~18), Asianometry to 6, in **6.8 s** for a full
+channel. Titles, exact dates and durations all correct; a generated `.strm` fetched through the proxy
+returns h264 720p + aac and honours ranges; the NFO's `durationinseconds` matches the API to the
+second.
+→ *Still outstanding: the Jellyfin scan-time measurement for §5, which needs the tree at the real
+media root, which needs Phase 2 installed.*
 
-**Phase 4 — subscription sync.** `subsync.py`, the first-sync bulk import, the add cap, the
-missing-threshold, channel deletion on unsubscribe, the admin routes, the healthchecks UUID.
-→ *Done when: he approves the first import; then he subscribes to a new channel on YouTube and within
-an hour it is a series in Jellyfin with episodes that play, nobody having touched the admin UI. Then
-the destructive half: he unsubscribes, and after three syncs the channel and its tree are gone. Plus
-a forced 403 and a forced empty response, each of which must leave the DB untouched and turn the check
-red — test these before trusting the deletion path, not after.*
+**Phase 4 — subscription sync. ✅ CODE COMPLETE.** `subsync.py`, the first-sync import, the add cap,
+the missing-threshold, deletion on unsubscribe, and the `/pending` admin page with source management,
+sync-now, and multi-select approve/reject — all four new routes CSRF-guarded, verified over real HTTP.
+Against the live account the first sync queued **119 channels and added none**; approving three at
+once added three.
+
+The destructive half is covered by tests rather than by having done it to the real account: a forced
+403, a forced network error and a forced empty response each leave the database untouched, absence is
+counted across three healthy syncs before deletion, and `manual` channels are exempt.
+→ *Still outstanding: the healthchecks UUIDs in `deploy/crontab.fragment`, and watching a real
+subscribe-then-unsubscribe cycle once the services are installed.*
 
 **Phase 5 — cut over.** §12 steps 1–2, run for a week.
 → *Done when: nothing has broken and nobody has used the old library.*
@@ -887,3 +895,67 @@ states return HTTP 403 `forbidden`, and the distinguishing signal is `error.deta
 
 Nothing. Phase 0 is closed. The one open decision is §14 item 2 — whether `min_keep_videos` (§5) is
 wanted, which is a five-minute change either way and does not block starting Phase 1.
+
+---
+
+## 17. What the build changed — 2026-08-12
+
+Three bugs, two of which only real data would have found. Recorded because each one
+is a trap the next change could walk back into.
+
+### The prune boundary — caught by a test
+
+`strm.remove` pruned empty directories up to the media root, so a channel directory
+whose `tvshow.nfo` happened to be missing would be deleted along with its last
+season. It only *looked* safe because `tvshow.nfo` normally stops the walk. The
+boundary is now the channel directory explicitly, and a test asserts the channel
+directory survives.
+
+### Titles were missing on the oldest episodes of every backfill
+
+The backfill inserted rows with no title and left the RSS poll to fill them in.
+That works only if RSS reaches as far back as the retention window, and it does
+not: the feed returns 15 entries, which for Pitch Side spans 23 days against a
+30-day window. **Five of twenty episodes were named after their video id.**
+
+`playlistItems.list` now requests `snippet` as well as `contentDetails`. Both parts
+cost the same single quota unit together as either does alone, and `snippet.title`
+arrives alongside the exact publish date. Note the trap next door:
+`snippet.publishedAt` is when the video was *added to the playlist*, not when it was
+published — only `contentDetails.videoPublishedAt` is the real thing.
+
+### The fallback title was written back to the database — the worse half
+
+`strm.materialise` used `video["title"] or video["video_id"]` for the filename and
+then stored *that* as the title. So an untitled row became a row whose title was its
+own video id, which is not empty, which permanently disabled the repair path that
+fills titles in from a later feed poll. The two bugs compounded: the first created
+badly-named episodes and the second made them permanent.
+
+Now the fallback is used for the filename only, and a title that arrives late also
+deletes the badly-named files and re-queues the video so it is rewritten under its
+real name.
+
+### Also worth knowing
+
+- **`_form()` collapsed repeated fields to the last value.** The approval queue is a
+  form of checkboxes all named `id`; through `_form()` it would have silently
+  approved only the last box ticked. Added `_form_list()`, and verified over real
+  HTTP that approving three at once adds three.
+- **`min_keep_videos` shipped at 5** rather than being left open (§14 item 2). Without
+  it, 52 of 117 measured channels are empty Jellyfin series that flicker in and out
+  as their single video crosses the window. Set it to 0 to get pure 30-day retention.
+- **Two settings validators earn their keep**: `subsync_missing_threshold` rejects 0
+  at the form, and `subsync.sync_source` clamps it to 1 anyway — a stored zero would
+  mean "unsubscribe before any absence has been confirmed".
+
+### Measured during the build
+
+| | |
+|---|---|
+| Tests | **337**, no network, no yt-dlp, no Jellyfin |
+| First sync of the real account | 119 queued, **0 added** |
+| Pitch Side backfill | **20 episodes** (§5 predicted ~18) in **6.8 s** |
+| Asianometry backfill | 6 episodes |
+| Generated `.strm` played through the proxy | h264 720p + aac, ranges honoured |
+| NFO `durationinseconds` vs API truth | 889 vs 889 |

tests/test_web.py

@@ -144,3 +144,75 @@ def test_csrf_token_rejects_tampering():
 def test_csrf_rejects_an_empty_token():
     secret = auth.new_secret()
     assert not auth.verify_csrf(secret, "s", "")
+
+
+# --------------------------------------------------- sources / approval queue
+
+
+def _source(**kw):
+    base = {"key": "youtube:UCbrother", "label": "C Flux",
+            "channel_id": "UCPcTWaLV8zwx4WP4QExHj4Q", "enabled": 1, "imported": 1,
+            "last_sync_at": "2026-08-12T16:00:00+00:00", "last_sync_ok": 1,
+            "consecutive_failures": 0, "last_error": None}
+    base.update(kw)
+    return base
+
+
+def _pending(n=3):
+    return [{"id": i, "source": "youtube:UCbrother", "title": f"Channel {i}",
+             "channel_id": f"UC{i:022d}", "seen_at": "2026-08-12T16:00:00+00:00"}
+            for i in range(1, n + 1)]
+
+
+def test_pending_page_renders_the_queue():
+    html = templates.pending_page(pending=_pending(3), sources=[_source()],
+                                 csrf="tok").decode()
+    assert "3 channel(s) waiting" in html
+    assert "Channel 1" in html and "Channel 3" in html
+    assert "Approve selected" in html
+
+
+def test_every_queue_row_shares_the_id_field_name():
+    """The approve handler reads a repeated `id` field; if the template numbered
+    them uniquely the multi-select would silently approve nothing."""
+    html = templates.pending_page(pending=_pending(3), sources=[_source()],
+                                 csrf="tok").decode()
+    assert html.count('name="id"') == 3
+
+
+def test_pending_page_with_an_empty_queue():
+    html = templates.pending_page(pending=[], sources=[_source()], csrf="tok").decode()
+    assert "Nothing awaiting approval" in html
+
+
+def test_pending_page_with_no_sources():
+    html = templates.pending_page(pending=[], sources=[], csrf="tok").decode()
+    assert "No sources yet" in html
+
+
+def test_never_synced_source_is_labelled():
+    html = templates.pending_page(pending=[], sources=[_source(last_sync_ok=None)],
+                                 csrf="tok").decode()
+    assert "never synced" in html
+
+
+def test_failing_source_shows_the_actionable_error_in_full():
+    """The useful errors say exactly what to do; truncating them defeats the point."""
+    message = ('subscriptions are private (subscriptionForbidden) — nothing '
+               'changed. Fix: YouTube → Settings → Privacy → uncheck "Keep all '
+               'my subscriptions private".')
+    html = templates.pending_page(
+        pending=[], sources=[_source(last_sync_ok=0, consecutive_failures=3,
+                                     last_error=message)],
+        csrf="tok").decode()
+    assert "failing (3)" in html
+    assert "Keep all" in html
+
+
+def test_hostile_pending_title_is_escaped():
+    items = _pending(1)
+    items[0]["title"] = '<img src=x onerror=alert(1)>'
+    html = templates.pending_page(pending=items, sources=[_source()],
+                                 csrf="tok").decode()
+    assert "<img src=x" not in html
+    assert "&lt;img" in html

ytstream/web/server.py

@@ -16,7 +16,18 @@ import threading
 import urllib.parse
 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
 
-from .. import channels, config, db, discovery, jellyfin, util, videos, ytdlp
+from .. import (
+    api,
+    channels,
+    config,
+    db,
+    discovery,
+    jellyfin,
+    subsync,
+    util,
+    videos,
+    ytdlp,
+)
 from ..settings import EDITABLE, MASKED_KEYS, Settings, validate_all
 from . import auth, templates
 
@@ -77,15 +88,32 @@ class Handler(BaseHTTPRequestHandler):
         self.end_headers()
         self.wfile.write(body)
 
-    def _form(self) -> dict[str, str]:
+    def _parsed_form(self) -> dict[str, list[str]]:
+        """Parse the body once and cache it.
+
+        Cached because the request body can only be read from rfile once, and the
+        approval queue needs both the single-value and multi-value views of it.
+        """
+        if getattr(self, "_form_cache", None) is None:
             length = int(self.headers.get("Content-Length") or 0)
             if length <= 0 or length > MAX_BODY:
-            return {}
+                self._form_cache = {}
+            else:
                 raw = self.rfile.read(length).decode("utf-8", "replace")
-        return {
-            key: values[-1]
-            for key, values in urllib.parse.parse_qs(raw, keep_blank_values=True).items()
-        }
+                self._form_cache = urllib.parse.parse_qs(raw, keep_blank_values=True)
+        return self._form_cache
+
+    def _form(self) -> dict[str, str]:
+        """Last value wins, which is right for every single-value field."""
+        return {key: values[-1] for key, values in self._parsed_form().items()}
+
+    def _form_list(self, key: str) -> list[str]:
+        """Every value for a repeated field.
+
+        The approval queue is a form of checkboxes all named `id`. Reading it
+        through _form() would silently approve only the last box ticked.
+        """
+        return [value for value in self._parsed_form().get(key, []) if value]
 
     def _cookie_token(self) -> str:
         return auth.cookie_value(self.headers.get("Cookie") or "")
@@ -141,6 +169,10 @@ class Handler(BaseHTTPRequestHandler):
             if path == "/":
                 return self._send(200, self._render_index(conn, settings, token))
 
+            if path == "/pending":
+                return self._send(
+                    200, self._render_pending(conn, settings, token))
+
             return self._send(404, templates.page("Not found", "<h1>Not found</h1>"))
         finally:
             conn.close()
@@ -204,6 +236,14 @@ class Handler(BaseHTTPRequestHandler):
                 return self._add_channel(conn, settings, token, form)
             if path == "/settings":
                 return self._save_settings(conn, settings, token, form)
+            if path == "/sources":
+                return self._add_source(conn, settings, token, form)
+            if path == "/sync":
+                return self._sync_now(conn, settings)
+            if path == "/pending/approve":
+                return self._resolve_pending(conn, settings, form, "approved")
+            if path == "/pending/reject":
+                return self._resolve_pending(conn, settings, form, "rejected")
 
             parts = path.strip("/").split("/")
             if len(parts) == 3 and parts[0] == "channels" and parts[1].isdigit():
@@ -219,6 +259,59 @@ class Handler(BaseHTTPRequestHandler):
         finally:
             conn.close()
 
+    # ------------------------------------------------- subscription sources
+
+    def _render_pending(self, conn, settings: Settings, token: str) -> bytes:
+        return templates.pending_page(
+            pending=[dict(row) for row in subsync.pending(conn)],
+            sources=[dict(row) for row in subsync.all_sources(conn)],
+            csrf=auth.csrf_token(self._secret(settings), token),
+        )
+
+    def _add_source(self, conn, settings: Settings, token: str, form: dict) -> None:
+        reference = (form.get("channel") or "").strip()
+        if not reference:
+            return self._redirect("/pending")
+
+        client = api.Api(settings.get_str("youtube_api_key"))
+        try:
+            info = (client.channel(reference) if reference.startswith("UC")
+                    else client.resolve_handle(reference))
+        except api.ApiError as exc:
+            log.error("could not resolve source %s: %s", reference, exc)
+            return self._redirect("/pending")
+        if not info:
+            return self._redirect("/pending")
+
+        subsync.add_source(conn, channel_id=info["channel_id"], label=info["title"])
+        return self._redirect("/pending")
+
+    def _sync_now(self, conn, settings: Settings) -> None:
+        """Run a sync from the UI.
+
+        Synchronous, and that is deliberate: it is one API call per 50
+        subscriptions and the result is what the operator is about to look at.
+        A background job would mean rendering a page that does not yet reflect
+        the button that was just pressed.
+        """
+        try:
+            subsync.sync_all(conn, settings)
+        except Exception as exc:  # noqa: BLE001
+            log.error("sync from the UI failed: %s", exc)
+        return self._redirect("/pending")
+
+    def _resolve_pending(self, conn, settings: Settings, form: dict,
+                         resolution: str) -> None:
+        ids = [int(value) for value in self._form_list("id") if value.isdigit()]
+        if not ids:
+            return self._redirect("/pending")
+
+        if resolution == "approved":
+            subsync.approve(conn, settings, ids)
+        else:
+            subsync.resolve(conn, ids, "rejected")
+        return self._redirect("/pending")
+
     def _login(self, settings: Settings, form: dict) -> None:
         key = self._client_key()
         if self.server.throttle.locked(key):

ytstream/web/templates.py

@@ -251,3 +251,93 @@ def index_page(
 <footer>Downloads run hourly. Videos are deleted once they pass the retention
 window for their channel — this is a DVR, not an archive.</footer>"""
     return page("ytstream", body)
+
+
+# --------------------------------------------------------------------------
+# subscription sources and the approval queue
+
+
+def _source_panel(source: dict, csrf: str) -> str:
+    if source["last_sync_ok"] is None:
+        state = '<span class="badge">never synced</span>'
+    elif source["last_sync_ok"]:
+        state = '<span class="badge ok">ok</span>'
+    else:
+        state = (f'<span class="badge bad">failing '
+                 f'({source["consecutive_failures"]})</span>')
+
+    error = ""
+    if source["last_error"]:
+        # Shown in full rather than truncated: the useful ones say exactly what to
+        # do ("subscriptions are private — uncheck Keep all my subscriptions
+        # private"), and hiding that behind a log file defeats the point.
+        error = f'<p class="muted">{_e(source["last_error"])}</p>'
+
+    return f"""
+<div class="panel">
+  <strong>{_e(source['label'])}</strong> {state}
+  <p class="muted">{_e(source['channel_id'])} · last sync
+     {_e(source['last_sync_at'] or 'never')}
+     · first import {'done' if source['imported'] else 'pending'}</p>
+  {error}
+</div>"""
+
+
+def pending_page(*, pending: list[dict], sources: list[dict], csrf: str) -> bytes:
+    """The approval queue.
+
+    A separate page rather than a section on the index because the first sync of a
+    real account queued 119 channels, and that does not belong inline underneath
+    the channel table.
+    """
+    source_html = "".join(_source_panel(source, csrf) for source in sources) or (
+        '<p class="muted">No sources yet.</p>'
+    )
+
+    if pending:
+        rows = "".join(f"""
+<tr>
+  <td><input type="checkbox" name="id" value="{item['id']}" id="p{item['id']}"></td>
+  <td><label for="p{item['id']}">{_e(item['title'])}</label></td>
+  <td class="muted hide">{_e(item['channel_id'])}</td>
+</tr>""" for item in pending)
+        queue = f"""
+<form method="post" id="queue">
+  <input type="hidden" name="csrf" value="{_e(csrf)}">
+  <p class="muted">{len(pending)} channel(s) waiting. Approving a channel
+     backfills its recent videos and starts polling it; rejecting one means it is
+     never offered again.</p>
+  <table>
+    <thead><tr><th></th><th>Channel</th><th class="hide">Channel id</th></tr></thead>
+    <tbody>{rows}</tbody>
+  </table>
+  <div style="margin-top:.8rem">
+    <button type="submit" formaction="/pending/approve">Approve selected</button>
+    <button type="submit" formaction="/pending/reject" class="danger">
+      Reject selected</button>
+  </div>
+</form>"""
+    else:
+        queue = '<p class="muted">Nothing awaiting approval.</p>'
+
+    body = f"""
+<h1>ytstream</h1>
+<nav class="muted"><a href="/">Channels</a> · <strong>Subscriptions</strong></nav>
+
+<h2>Mirrored accounts</h2>
+{source_html}
+<form method="post" action="/sources" class="row">
+  <input type="hidden" name="csrf" value="{_e(csrf)}">
+  <input type="text" name="channel" placeholder="@handle or UC... id"
+         style="min-width:18rem">
+  <button type="submit">Add account</button>
+</form>
+<form method="post" action="/sync" style="margin-top:.6rem">
+  <input type="hidden" name="csrf" value="{_e(csrf)}">
+  <button type="submit">Sync now</button>
+</form>
+
+<h2>Awaiting approval</h2>
+{queue}
+"""
+    return page("Subscriptions — ytstream", body)