Files
Tom FluxandClaude Opus 5 d3bf8d6f19 Deploy it, and fix the six things installation found
Both units are installed and running, 10 of 119 channels approved, 251 episodes
live in Jellyfin with verified DirectPlay. 345 tests. Six problems surfaced that no
test could have, and two of them were mine in the deploy scripts.

deploy.sh had a circular dependency with bootstrap.sh: deploy started the units and
told the operator to run bootstrap, but bootstrap refused to run until the state
directory existed, which only deploy creates. The units started against a
non-existent venv, failed 203/EXEC and restart-looped 17 and 21 times. deploy.sh now
creates the directory, calls bootstrap itself through runuser so the venv is not left
root-owned, and refuses to start units when the venv is still missing.

Deno was absent, and `doctor` is the only reason we know. It is mandatory rather than
nice-to-have — without a JS runtime yt-dlp cannot solve the n challenge, which
youtube-automate measured on this machine as 22 formats instead of 29 plus
throttling. Nothing else would have complained; playback would just have quietly
degraded. bootstrap.sh now installs it and asserts yt-dlp reports it.

Episodes had no synopsis at all, because materialise passed plot=None while both
sources hand us descriptions for free. Now plumbed through from RSS
(media:group/media:description) and from videos.list, which carries
snippet.description in the call already being made for durations — so the ~40% of
episodes older than RSS reaches get one too. That needed a schema v2 migration; v1
was left exactly as shipped so a fresh install and a migrated one are identical, and
a test asserts it.

`materialise --all` — the documented recovery from a Jellyfin metadata wipe — was
itself creating duplicates. Episode numbers were re-derived each run, and
next_episode() excludes the row being numbered, so re-materialising a day's videos in
a different order renumbered them and orphaned the old files. One run left 102
orphaned NFOs against 251 episodes. An episode number is now permanent once assigned,
and a video whose rel_path changes has its old files removed first. Running it twice
is now a no-op.

Two Jellyfin behaviours worth having in writing. It ignores <runtime> and
<durationinseconds> for episodes while reading the rest of the NFO happily, so a
.strm shows no duration until first played — not fixable without probing, which is
the one thing this design exists to avoid. And a plain /Library/Refresh does not
reliably re-read a rewritten NFO: after rewriting all 251, fifty kept their old empty
metadata. The fix is metadataRefreshMode=Default with replaceAllMetadata=false, which
took plots from 201 to 251 while the proxy served zero requests. §5's prohibition on
replaceAllMetadata=true still stands — that one probes. Exposed as
`ytstream refresh-metadata` and run automatically after `materialise --all`.

The measurement §5 has been waiting for: a full Jellyfin scan of 251 .strm files
took ~119 s, about 8 minutes per 1,000 episodes, and made zero media probes. That
last number is the fact the whole design rests on, now confirmed at scale on the real
library rather than on seven PoC files.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-12 17:21:35 +01:00

207 lines
6.7 KiB
Python

"""Test fixtures.
Every path the application uses is redirected into a tmpdir. No test touches the
network, a real yt-dlp, a real Jellyfin, or the real media tree — the YouTube API
client is always a stub (see `FakeApi`), because the point of a test suite here is
to pin down behaviour that only shows up on the failure paths: a 403, an empty
response, a video that ages out and must not come back.
"""
from __future__ import annotations
import os
import tempfile
from pathlib import Path
# config resolves its paths at import time, so the environment has to be set
# before anything from the package is imported.
_SANDBOX = Path(tempfile.mkdtemp(prefix="yts-tests-"))
os.environ.setdefault("YTS_STATE_DIR", str(_SANDBOX / "state"))
os.environ.setdefault("YTS_MEDIA_ROOT", str(_SANDBOX / "media"))
os.environ.setdefault("YTS_DB_PATH", str(_SANDBOX / "state" / "ytstream.db"))
os.environ.setdefault("YTS_LOCK_PATH", str(_SANDBOX / "state" / "run.lock"))
os.environ.setdefault("YTS_VENV_BIN", str(_SANDBOX / "venv" / "bin"))
import pytest # noqa: E402
from ytstream import api, config, db, util, videos # noqa: E402
from ytstream.settings import Settings # noqa: E402
FIXTURES = Path(__file__).parent / "fixtures"
# A real-looking channel id: UC + 22 chars.
CHANNEL_ID = "UCW7jUEpYT_t0Gsf632d6_wQ"
@pytest.fixture()
def media_root(tmp_path, monkeypatch):
"""Point the media root at a per-test tmpdir."""
root = tmp_path / "media"
root.mkdir(parents=True)
monkeypatch.setattr(config, "MEDIA_ROOT", root)
return root
@pytest.fixture()
def conn(tmp_path):
connection = db.connect(tmp_path / "ytstream.db")
yield connection
connection.close()
@pytest.fixture()
def settings(conn):
settings = Settings(conn)
# Tests that reach the API go through FakeApi, but the code refuses to call
# out at all without a key, so give it one that is never used for real.
settings.set("youtube_api_key", "test-key")
return settings
@pytest.fixture()
def channel(conn):
"""One subscribed channel, returned as a row."""
return add_channel(conn, CHANNEL_ID, "clabretro", "clabretro")
@pytest.fixture()
def no_network(monkeypatch):
"""Fail loudly if anything tries to open a socket.
Belt and braces: a test that accidentally hits the network would pass locally
and fail in a different week for reasons nobody could reproduce.
"""
import urllib.request
def forbidden(*args, **kwargs):
raise AssertionError("test attempted a network call")
monkeypatch.setattr(urllib.request, "urlopen", forbidden)
# --------------------------------------------------------------------- helpers
def add_channel(conn, channel_id: str, title: str, dir_name: str, **kwargs):
fields = {"source": "youtube", "backfilled": 1, "uploads_playlist": "UULF"}
fields.update(kwargs)
columns = ", ".join(fields)
marks = ", ".join("?" * len(fields))
with conn:
conn.execute(
f"INSERT INTO channel (channel_id, handle, title, description, dir_name, "
f"added_at, {columns}) VALUES (?, ?, ?, ?, ?, ?, {marks})",
(channel_id, f"@{dir_name}", title, "A channel", dir_name,
util.utcnow_iso(), *fields.values()),
)
return conn.execute(
"SELECT * FROM channel WHERE channel_id = ?", (channel_id,)
).fetchone()
def add_video(conn, channel_pk, video_id, **kwargs):
"""Insert a video row with sensible defaults."""
defaults = {
"title": f"Video {video_id}",
"upload_date": "2026-08-01",
"state": videos.LISTED,
"discovery_source": videos.SOURCE_UULF,
"duration": 900,
}
defaults.update(kwargs)
videos.insert(conn, channel_pk=channel_pk, video_id=video_id, **defaults)
return videos.get(conn, video_id)
def feed_bytes(name: str) -> bytes:
return (FIXTURES / name).read_bytes()
def make_feed(entries: list[dict], *, playlist_published: str = "2019-01-01T00:00:00+00:00") -> bytes:
"""Build an Atom feed shaped like YouTube's.
Includes the feed-level <published> that is NOT an entry, because scraping
timestamps instead of walking atom:entry picks it up and yields nonsense
upload rates — a mistake made once while measuring.
"""
items = "".join(
f"""
<entry>
<id>yt:video:{e['video_id']}</id>
<yt:videoId>{e['video_id']}</yt:videoId>
<title>{e.get('title', 'Untitled')}</title>
<published>{e['published']}</published>
</entry>"""
for e in entries
)
return f"""<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns:yt="http://www.youtube.com/xml/schemas/2015"
xmlns:media="http://search.yahoo.com/mrss/"
xmlns="http://www.w3.org/2005/Atom">
<title>Videos</title>
<published>{playlist_published}</published>{items}
</feed>""".encode()
class FakeApi:
"""Stand-in for `api.Api` with scriptable responses.
Counts calls so tests can assert the batching actually batches — the whole
quota argument in the plan rests on 50 ids per call, and a regression to one
call per video would be silent and expensive.
"""
def __init__(self, *, subs=None, uploads=None, durations=None, channel=None,
raises=None):
self._subs = subs
self._uploads = uploads or []
self._durations = durations or {}
self._channel = channel
self._raises = raises
self.calls = 0
self.duration_calls = 0
self.subscription_calls = 0
def subscriptions(self, channel_id):
self.subscription_calls += 1
self.calls += 1
if isinstance(self._raises, Exception):
raise self._raises
return list(self._subs or [])
def uploads(self, channel_id, *, kind="UULF", since=None, limit=None,
page_token=None):
self.calls += 1
produced = 0
for entry, token in self._uploads:
if since is not None and entry["published"] < since:
return
yield entry, token
produced += 1
if limit is not None and produced >= limit:
return
def details(self, video_ids):
self.duration_calls += 1
self.calls += 1
return {vid: self._durations[vid] for vid in video_ids
if vid in self._durations}
# `durations` was the old name; kept so a stale caller fails loudly in tests
# rather than silently skipping enrichment.
durations = details
def channel(self, channel_id):
self.calls += 1
return self._channel
def resolve_handle(self, handle):
self.calls += 1
return self._channel
def patch_api(monkeypatch, module, fake: FakeApi):
"""Make `module.api.Api(...)` return `fake` regardless of arguments."""
monkeypatch.setattr(module.api, "Api", lambda *a, **kw: fake)
return fake