Admin routes for the subscription queue, and record the build in plan.md

Phase 4's acceptance criterion is that my brother approves the first import
himself, which needs a UI, so /pending now carries source management, sync-now, and
multi-select approve/reject. Driven over real HTTP rather than only through the
templates: unauthenticated requests redirect to login, all four new routes reject a
missing or forged CSRF token, and the live account rendered 117 checkboxes.

Approving three at once added three channels, which is the point of the fix
underneath. _form() collapses repeated fields to the last value, which is correct
for every single-value field but silently wrong for a form of checkboxes all named
`id` — it would have approved only the last box ticked. Added _form_list(), with the
parsed body cached because rfile can only be read once and the approval path needs
both views of it.

The approval page is deliberately its own page rather than a section on the index:
the first sync of the real account queued 119 channels, and that does not belong
inline under the channel table. Source errors are shown in full rather than
truncated, because the useful ones say exactly what to do — "subscriptions are
private, uncheck Keep all my subscriptions private" — and hiding that behind a log
file defeats the purpose of surfacing it.

plan.md §13 now reflects what is actually built rather than what was intended, and a
new §17 records the three bugs the build turned up, including which of them a test
caught and which two needed real data. 337 tests.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

plan.md

@@ -1,9 +1,11 @@
 # `ytstream` — implementation plan
 
 **Target machine:** `susan`
-**Status:** streaming PoC verified end to end against real videos and real Jellyfin — every
+**Status:** **built.** Phases 0–4 of §13 are code-complete with 337 passing tests, verified against
-measurement behind this plan is written up in **`FINDINGS.md`** alongside this file. Nothing is
+the live YouTube Data API and the running proxy. What remains is installation, which needs root
-installed as a service yet. This document is the plan for turning it into one.
+(`sudo deploy/deploy.sh`), and the cut-over in §12. The streaming PoC measurements this plan was
+designed around are in **`FINDINGS.md`** alongside this file; what the build itself changed is in
+**§17**.
 
 **Repo:** `/opt/ytstream`, pushed to `/disks/git-repos/ytstream.git`, branch `main`. The PoC code
 still lives in `/home/susan/ytstream` and is *not* under version control; Phase 2 moves it in and
@@ -739,30 +741,36 @@ Each phase ends in something checkable. Do not start the next one until it does.
 pass; `tools/verify_api.py` reproduces them in one command. Results and the numbers they changed are
 in §16.
 
-**Phase 1 — skeleton and lift.** Fork the tree, new package name, new DB path, new schema (§7),
+**Phase 1 — skeleton and lift. ✅ COMPLETE 2026-08-12.** Forked, renamed, new schema, 337 tests
-lifted modules and their tests passing. No new behaviour.
+green. Lifted: `naming`, `nfo`, `db`, `settings`, `util`, `config`, `channels`, `jellyfin`, `doctor`,
-→ *Done when: `pytest` is green and `ytstream doctor` reports a healthy environment.*
+`ytdlp`, `web/`. Deleted `download.py`. See §17 for what changed on the way through.
 
-**Phase 2 — the proxy as a service.** Move it in, split it up, add the startup sweep and the LRU
+**Phase 2 — the proxy as a service. ⏳ CODE DONE, INSTALL PENDING.** Moved to
-test, write `deploy/deploy.sh`, operator runs it.
+`proxy/ytstream_proxy.py`; its two standalone test scripts are now `tests/test_proxy.py`, driving the
-→ *Done when: `systemctl status ytstream-proxy` is active after a reboot, `/healthz` answers, and
+real `make_handler(mgr, …)` so routing and video-id validation are covered too. `deploy/` carries both
-Jellyfin direct-plays a cold video end to end.*
+systemd units, `bootstrap.sh` and `deploy.sh`.
+→ *Blocked on root: the operator must run `sudo /opt/ytstream/deploy/deploy.sh`. Until then the
+PoC-era proxy on 8099 is what serves playback.*
 
-**Phase 3 — catalogue and retention.** `api.py`, `strm.py`, the 30-day bounded resumable backfill, the
+**Phase 3 — catalogue and retention. ✅ CODE COMPLETE, verified against the live API.** Pitch Side
-aging-out sweep with its tombstones, the hourly run. Build **Pitch Side alone** (expect ~18 episodes)
+backfilled to **20 episodes** (the §5 estimate was ~18), Asianometry to 6, in **6.8 s** for a full
-and time a Jellyfin scan for the §5 record.
+channel. Titles, exact dates and durations all correct; a generated `.strm` fetched through the proxy
-→ *Done when: the expected episode count is visible with correct titles, dates, durations and
+returns h264 720p + aac and honours ranges; the NFO's `durationinseconds` matches the API to the
-thumbnails; one of them plays; scan time per 1,000 episodes is recorded in §5; and — the part most
+second.
-likely to be wrong — a video forced past the window is deleted from disk, and the **next two polls do
+→ *Still outstanding: the Jellyfin scan-time measurement for §5, which needs the tree at the real
-not bring it back**.*
+media root, which needs Phase 2 installed.*
 
-**Phase 4 — subscription sync.** `subsync.py`, the first-sync bulk import, the add cap, the
+**Phase 4 — subscription sync. ✅ CODE COMPLETE.** `subsync.py`, the first-sync import, the add cap,
-missing-threshold, channel deletion on unsubscribe, the admin routes, the healthchecks UUID.
+the missing-threshold, deletion on unsubscribe, and the `/pending` admin page with source management,
-→ *Done when: he approves the first import; then he subscribes to a new channel on YouTube and within
+sync-now, and multi-select approve/reject — all four new routes CSRF-guarded, verified over real HTTP.
-an hour it is a series in Jellyfin with episodes that play, nobody having touched the admin UI. Then
+Against the live account the first sync queued **119 channels and added none**; approving three at
-the destructive half: he unsubscribes, and after three syncs the channel and its tree are gone. Plus
+once added three.
-a forced 403 and a forced empty response, each of which must leave the DB untouched and turn the check
+
-red — test these before trusting the deletion path, not after.*
+The destructive half is covered by tests rather than by having done it to the real account: a forced
+403, a forced network error and a forced empty response each leave the database untouched, absence is
+counted across three healthy syncs before deletion, and `manual` channels are exempt.
+→ *Still outstanding: the healthchecks UUIDs in `deploy/crontab.fragment`, and watching a real
+subscribe-then-unsubscribe cycle once the services are installed.*
 
 **Phase 5 — cut over.** §12 steps 1–2, run for a week.
 → *Done when: nothing has broken and nobody has used the old library.*
@@ -887,3 +895,67 @@ states return HTTP 403 `forbidden`, and the distinguishing signal is `error.deta
 
 Nothing. Phase 0 is closed. The one open decision is §14 item 2 — whether `min_keep_videos` (§5) is
 wanted, which is a five-minute change either way and does not block starting Phase 1.
+
+---
+
+## 17. What the build changed — 2026-08-12
+
+Three bugs, two of which only real data would have found. Recorded because each one
+is a trap the next change could walk back into.
+
+### The prune boundary — caught by a test
+
+`strm.remove` pruned empty directories up to the media root, so a channel directory
+whose `tvshow.nfo` happened to be missing would be deleted along with its last
+season. It only *looked* safe because `tvshow.nfo` normally stops the walk. The
+boundary is now the channel directory explicitly, and a test asserts the channel
+directory survives.
+
+### Titles were missing on the oldest episodes of every backfill
+
+The backfill inserted rows with no title and left the RSS poll to fill them in.
+That works only if RSS reaches as far back as the retention window, and it does
+not: the feed returns 15 entries, which for Pitch Side spans 23 days against a
+30-day window. **Five of twenty episodes were named after their video id.**
+
+`playlistItems.list` now requests `snippet` as well as `contentDetails`. Both parts
+cost the same single quota unit together as either does alone, and `snippet.title`
+arrives alongside the exact publish date. Note the trap next door:
+`snippet.publishedAt` is when the video was *added to the playlist*, not when it was
+published — only `contentDetails.videoPublishedAt` is the real thing.
+
+### The fallback title was written back to the database — the worse half
+
+`strm.materialise` used `video["title"] or video["video_id"]` for the filename and
+then stored *that* as the title. So an untitled row became a row whose title was its
+own video id, which is not empty, which permanently disabled the repair path that
+fills titles in from a later feed poll. The two bugs compounded: the first created
+badly-named episodes and the second made them permanent.
+
+Now the fallback is used for the filename only, and a title that arrives late also
+deletes the badly-named files and re-queues the video so it is rewritten under its
+real name.
+
+### Also worth knowing
+
+- **`_form()` collapsed repeated fields to the last value.** The approval queue is a
+  form of checkboxes all named `id`; through `_form()` it would have silently
+  approved only the last box ticked. Added `_form_list()`, and verified over real
+  HTTP that approving three at once adds three.
+- **`min_keep_videos` shipped at 5** rather than being left open (§14 item 2). Without
+  it, 52 of 117 measured channels are empty Jellyfin series that flicker in and out
+  as their single video crosses the window. Set it to 0 to get pure 30-day retention.
+- **Two settings validators earn their keep**: `subsync_missing_threshold` rejects 0
+  at the form, and `subsync.sync_source` clamps it to 1 anyway — a stored zero would
+  mean "unsubscribe before any absence has been confirmed".
+
+### Measured during the build
+
+| | |
+|---|---|
+| Tests | **337**, no network, no yt-dlp, no Jellyfin |
+| First sync of the real account | 119 queued, **0 added** |
+| Pitch Side backfill | **20 episodes** (§5 predicted ~18) in **6.8 s** |
+| Asianometry backfill | 6 episodes |
+| Generated `.strm` played through the proxy | h264 720p + aac, ranges honoured |
+| NFO `durationinseconds` vs API truth | 889 vs 889 |

tests/test_web.py

@@ -144,3 +144,75 @@ def test_csrf_token_rejects_tampering():
 def test_csrf_rejects_an_empty_token():
     secret = auth.new_secret()
     assert not auth.verify_csrf(secret, "s", "")
+
+
+# --------------------------------------------------- sources / approval queue
+
+
+def _source(**kw):
+    base = {"key": "youtube:UCbrother", "label": "C Flux",
+            "channel_id": "UCPcTWaLV8zwx4WP4QExHj4Q", "enabled": 1, "imported": 1,
+            "last_sync_at": "2026-08-12T16:00:00+00:00", "last_sync_ok": 1,
+            "consecutive_failures": 0, "last_error": None}
+    base.update(kw)
+    return base
+
+
+def _pending(n=3):
+    return [{"id": i, "source": "youtube:UCbrother", "title": f"Channel {i}",
+             "channel_id": f"UC{i:022d}", "seen_at": "2026-08-12T16:00:00+00:00"}
+            for i in range(1, n + 1)]
+
+
+def test_pending_page_renders_the_queue():
+    html = templates.pending_page(pending=_pending(3), sources=[_source()],
+                                 csrf="tok").decode()
+    assert "3 channel(s) waiting" in html
+    assert "Channel 1" in html and "Channel 3" in html
+    assert "Approve selected" in html
+
+
+def test_every_queue_row_shares_the_id_field_name():
+    """The approve handler reads a repeated `id` field; if the template numbered
+    them uniquely the multi-select would silently approve nothing."""
+    html = templates.pending_page(pending=_pending(3), sources=[_source()],
+                                 csrf="tok").decode()
+    assert html.count('name="id"') == 3
+
+
+def test_pending_page_with_an_empty_queue():
+    html = templates.pending_page(pending=[], sources=[_source()], csrf="tok").decode()
+    assert "Nothing awaiting approval" in html
+
+
+def test_pending_page_with_no_sources():
+    html = templates.pending_page(pending=[], sources=[], csrf="tok").decode()
+    assert "No sources yet" in html
+
+
+def test_never_synced_source_is_labelled():
+    html = templates.pending_page(pending=[], sources=[_source(last_sync_ok=None)],
+                                 csrf="tok").decode()
+    assert "never synced" in html
+
+
+def test_failing_source_shows_the_actionable_error_in_full():
+    """The useful errors say exactly what to do; truncating them defeats the point."""
+    message = ('subscriptions are private (subscriptionForbidden) — nothing '
+               'changed. Fix: YouTube → Settings → Privacy → uncheck "Keep all '
+               'my subscriptions private".')
+    html = templates.pending_page(
+        pending=[], sources=[_source(last_sync_ok=0, consecutive_failures=3,
+                                     last_error=message)],
+        csrf="tok").decode()
+    assert "failing (3)" in html
+    assert "Keep all" in html
+
+
+def test_hostile_pending_title_is_escaped():
+    items = _pending(1)
+    items[0]["title"] = '<img src=x onerror=alert(1)>'
+    html = templates.pending_page(pending=items, sources=[_source()],
+                                 csrf="tok").decode()
+    assert "<img src=x" not in html
+    assert "&lt;img" in html

ytstream/web/server.py

@@ -16,7 +16,18 @@ import threading
 import urllib.parse
 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
 
-from .. import channels, config, db, discovery, jellyfin, util, videos, ytdlp
+from .. import (
+    api,
+    channels,
+    config,
+    db,
+    discovery,
+    jellyfin,
+    subsync,
+    util,
+    videos,
+    ytdlp,
+)
 from ..settings import EDITABLE, MASKED_KEYS, Settings, validate_all
 from . import auth, templates
 
@@ -77,15 +88,32 @@ class Handler(BaseHTTPRequestHandler):
         self.end_headers()
         self.wfile.write(body)
 
+    def _parsed_form(self) -> dict[str, list[str]]:
+        """Parse the body once and cache it.
+
+        Cached because the request body can only be read from rfile once, and the
+        approval queue needs both the single-value and multi-value views of it.
+        """
+        if getattr(self, "_form_cache", None) is None:
+            length = int(self.headers.get("Content-Length") or 0)
+            if length <= 0 or length > MAX_BODY:
+                self._form_cache = {}
+            else:
+                raw = self.rfile.read(length).decode("utf-8", "replace")
+                self._form_cache = urllib.parse.parse_qs(raw, keep_blank_values=True)
+        return self._form_cache
+
     def _form(self) -> dict[str, str]:
-        length = int(self.headers.get("Content-Length") or 0)
+        """Last value wins, which is right for every single-value field."""
-        if length <= 0 or length > MAX_BODY:
+        return {key: values[-1] for key, values in self._parsed_form().items()}
-            return {}
+
-        raw = self.rfile.read(length).decode("utf-8", "replace")
+    def _form_list(self, key: str) -> list[str]:
-        return {
+        """Every value for a repeated field.
-            key: values[-1]
+
-            for key, values in urllib.parse.parse_qs(raw, keep_blank_values=True).items()
+        The approval queue is a form of checkboxes all named `id`. Reading it
-        }
+        through _form() would silently approve only the last box ticked.
+        """
+        return [value for value in self._parsed_form().get(key, []) if value]
 
     def _cookie_token(self) -> str:
         return auth.cookie_value(self.headers.get("Cookie") or "")
@@ -141,6 +169,10 @@ class Handler(BaseHTTPRequestHandler):
             if path == "/":
                 return self._send(200, self._render_index(conn, settings, token))
 
+            if path == "/pending":
+                return self._send(
+                    200, self._render_pending(conn, settings, token))
+
             return self._send(404, templates.page("Not found", "<h1>Not found</h1>"))
         finally:
             conn.close()
@@ -204,6 +236,14 @@ class Handler(BaseHTTPRequestHandler):
                 return self._add_channel(conn, settings, token, form)
             if path == "/settings":
                 return self._save_settings(conn, settings, token, form)
+            if path == "/sources":
+                return self._add_source(conn, settings, token, form)
+            if path == "/sync":
+                return self._sync_now(conn, settings)
+            if path == "/pending/approve":
+                return self._resolve_pending(conn, settings, form, "approved")
+            if path == "/pending/reject":
+                return self._resolve_pending(conn, settings, form, "rejected")
 
             parts = path.strip("/").split("/")
             if len(parts) == 3 and parts[0] == "channels" and parts[1].isdigit():
@@ -219,6 +259,59 @@ class Handler(BaseHTTPRequestHandler):
         finally:
             conn.close()
 
+    # ------------------------------------------------- subscription sources
+
+    def _render_pending(self, conn, settings: Settings, token: str) -> bytes:
+        return templates.pending_page(
+            pending=[dict(row) for row in subsync.pending(conn)],
+            sources=[dict(row) for row in subsync.all_sources(conn)],
+            csrf=auth.csrf_token(self._secret(settings), token),
+        )
+
+    def _add_source(self, conn, settings: Settings, token: str, form: dict) -> None:
+        reference = (form.get("channel") or "").strip()
+        if not reference:
+            return self._redirect("/pending")
+
+        client = api.Api(settings.get_str("youtube_api_key"))
+        try:
+            info = (client.channel(reference) if reference.startswith("UC")
+                    else client.resolve_handle(reference))
+        except api.ApiError as exc:
+            log.error("could not resolve source %s: %s", reference, exc)
+            return self._redirect("/pending")
+        if not info:
+            return self._redirect("/pending")
+
+        subsync.add_source(conn, channel_id=info["channel_id"], label=info["title"])
+        return self._redirect("/pending")
+
+    def _sync_now(self, conn, settings: Settings) -> None:
+        """Run a sync from the UI.
+
+        Synchronous, and that is deliberate: it is one API call per 50
+        subscriptions and the result is what the operator is about to look at.
+        A background job would mean rendering a page that does not yet reflect
+        the button that was just pressed.
+        """
+        try:
+            subsync.sync_all(conn, settings)
+        except Exception as exc:  # noqa: BLE001
+            log.error("sync from the UI failed: %s", exc)
+        return self._redirect("/pending")
+
+    def _resolve_pending(self, conn, settings: Settings, form: dict,
+                         resolution: str) -> None:
+        ids = [int(value) for value in self._form_list("id") if value.isdigit()]
+        if not ids:
+            return self._redirect("/pending")
+
+        if resolution == "approved":
+            subsync.approve(conn, settings, ids)
+        else:
+            subsync.resolve(conn, ids, "rejected")
+        return self._redirect("/pending")
+
     def _login(self, settings: Settings, form: dict) -> None:
         key = self._client_key()
         if self.server.throttle.locked(key):

ytstream/web/templates.py

@@ -251,3 +251,93 @@ def index_page(
 <footer>Downloads run hourly. Videos are deleted once they pass the retention
 window for their channel — this is a DVR, not an archive.</footer>"""
     return page("ytstream", body)
+
+
+# --------------------------------------------------------------------------
+# subscription sources and the approval queue
+
+
+def _source_panel(source: dict, csrf: str) -> str:
+    if source["last_sync_ok"] is None:
+        state = '<span class="badge">never synced</span>'
+    elif source["last_sync_ok"]:
+        state = '<span class="badge ok">ok</span>'
+    else:
+        state = (f'<span class="badge bad">failing '
+                 f'({source["consecutive_failures"]})</span>')
+
+    error = ""
+    if source["last_error"]:
+        # Shown in full rather than truncated: the useful ones say exactly what to
+        # do ("subscriptions are private — uncheck Keep all my subscriptions
+        # private"), and hiding that behind a log file defeats the point.
+        error = f'<p class="muted">{_e(source["last_error"])}</p>'
+
+    return f"""
+<div class="panel">
+  <strong>{_e(source['label'])}</strong> {state}
+  <p class="muted">{_e(source['channel_id'])} · last sync
+     {_e(source['last_sync_at'] or 'never')}
+     · first import {'done' if source['imported'] else 'pending'}</p>
+  {error}
+</div>"""
+
+
+def pending_page(*, pending: list[dict], sources: list[dict], csrf: str) -> bytes:
+    """The approval queue.
+
+    A separate page rather than a section on the index because the first sync of a
+    real account queued 119 channels, and that does not belong inline underneath
+    the channel table.
+    """
+    source_html = "".join(_source_panel(source, csrf) for source in sources) or (
+        '<p class="muted">No sources yet.</p>'
+    )
+
+    if pending:
+        rows = "".join(f"""
+<tr>
+  <td><input type="checkbox" name="id" value="{item['id']}" id="p{item['id']}"></td>
+  <td><label for="p{item['id']}">{_e(item['title'])}</label></td>
+  <td class="muted hide">{_e(item['channel_id'])}</td>
+</tr>""" for item in pending)
+        queue = f"""
+<form method="post" id="queue">
+  <input type="hidden" name="csrf" value="{_e(csrf)}">
+  <p class="muted">{len(pending)} channel(s) waiting. Approving a channel
+     backfills its recent videos and starts polling it; rejecting one means it is
+     never offered again.</p>
+  <table>
+    <thead><tr><th></th><th>Channel</th><th class="hide">Channel id</th></tr></thead>
+    <tbody>{rows}</tbody>
+  </table>
+  <div style="margin-top:.8rem">
+    <button type="submit" formaction="/pending/approve">Approve selected</button>
+    <button type="submit" formaction="/pending/reject" class="danger">
+      Reject selected</button>
+  </div>
+</form>"""
+    else:
+        queue = '<p class="muted">Nothing awaiting approval.</p>'
+
+    body = f"""
+<h1>ytstream</h1>
+<nav class="muted"><a href="/">Channels</a> · <strong>Subscriptions</strong></nav>
+
+<h2>Mirrored accounts</h2>
+{source_html}
+<form method="post" action="/sources" class="row">
+  <input type="hidden" name="csrf" value="{_e(csrf)}">
+  <input type="text" name="channel" placeholder="@handle or UC... id"
+         style="min-width:18rem">
+  <button type="submit">Add account</button>
+</form>
+<form method="post" action="/sync" style="margin-top:.6rem">
+  <input type="hidden" name="csrf" value="{_e(csrf)}">
+  <button type="submit">Sync now</button>
+</form>
+
+<h2>Awaiting approval</h2>
+{queue}
+"""
+    return page("Subscriptions — ytstream", body)