Build ytstream: catalogue, retention, subscription mirror, proxy
Phases 1-4 of plan.md §13. Forked from youtube-automate as planned rather than
written from scratch: naming, NFO, auth, the admin UI, settings and the DB layer
came across largely unchanged, download.py is gone, and the pieces that only make
sense for a streaming library are new.
api.py YouTube Data API v3 client. The whole metadata path.
strm.py Materialising: a .strm, an .nfo and a thumbnail. Replaces the
330-line download.py, because the job is writing a URL to a file.
subsync.py The subscription mirror, most of which is refusals.
reap.py Retention, rewritten around the 30-day window and min_keep_videos.
discovery.py RSS polling plus an API-backed, resumable, bounded backfill.
proxy/ The verified PoC, moved in with a systemd unit.
330 tests, all passing, no network and no yt-dlp in any of them. The suite leans
towards the failure paths, because that is where this design can actually hurt
someone: a 403 that looks like an unsubscribe, a video that ages out and comes
back, a title that never arrives. tests/test_proxy.py replaces the two standalone
scripts under proxy/ and now drives the real make_handler(mgr, ...) rather than the
PoC's make_handler(path, done), so routing and video-id validation are covered too.
Ran it end to end against the live API and it found three real bugs.
The first was mine and the tests caught it: strm.remove pruned empty directories
up to the media root, so a channel directory whose tvshow.nfo happened to be
missing would be deleted along with the season. It only looked safe because
tvshow.nfo normally stops the walk. The prune boundary is now the channel
directory explicitly.
The other two only showed up against real data, and they compounded. The backfill
inserted rows with no title and left the RSS poll to fill them in — but RSS returns
15 entries, which for Pitch Side spans 23 days against a 30-day window, so five of
twenty episodes were named after their video ids. Worse, strm.materialise wrote
that fallback back to the database as the title, which made the row look titled and
permanently disabled the repair path. Both fixed: playlistItems.list now requests
snippet as well as contentDetails, which costs the same single quota unit and
carries the title alongside the exact publish date, and the fallback is used for the
filename without being persisted. A title that does arrive late now also removes the
badly-named files and re-queues, so the episode is rewritten rather than keeping its
video-id name forever. Verified against the live API: all twenty Pitch Side episodes
now carry real titles.
Measured on the real account: 119 subscriptions queued for approval and none added
on the first sync, then a two-channel run backfilled and materialised 26 episodes in
under seven seconds.
Two deliberate departures from plan.md, both recorded there:
min_keep_videos defaults to 5 rather than being left as an open question. Without
it 52 of 117 measured channels are empty Jellyfin series that flicker in and out as
their single video crosses the retention line, and the plan already recommended it.
The Jellyfin refresh is a bare /Library/Refresh with a comment explaining why it
must stay that way. A normal scan makes zero media probes; FullRefresh does probe,
and at 400 episodes that is 400 cold starts.
Not yet done: no systemd units are installed (needs root — deploy/deploy.sh), the
admin UI has no routes for sources or the approval queue yet, and nothing has been
pointed at the real media root.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Tom Flux
co-authored by
Claude Opus 5
Claude Opus 5
parent
f640c064c6
commit
155f05773d
@@ -0,0 +1,202 @@
|
||||
"""Test fixtures.
|
||||
|
||||
Every path the application uses is redirected into a tmpdir. No test touches the
|
||||
network, a real yt-dlp, a real Jellyfin, or the real media tree — the YouTube API
|
||||
client is always a stub (see `FakeApi`), because the point of a test suite here is
|
||||
to pin down behaviour that only shows up on the failure paths: a 403, an empty
|
||||
response, a video that ages out and must not come back.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
# config resolves its paths at import time, so the environment has to be set
|
||||
# before anything from the package is imported.
|
||||
_SANDBOX = Path(tempfile.mkdtemp(prefix="yts-tests-"))
|
||||
os.environ.setdefault("YTS_STATE_DIR", str(_SANDBOX / "state"))
|
||||
os.environ.setdefault("YTS_MEDIA_ROOT", str(_SANDBOX / "media"))
|
||||
os.environ.setdefault("YTS_DB_PATH", str(_SANDBOX / "state" / "ytstream.db"))
|
||||
os.environ.setdefault("YTS_LOCK_PATH", str(_SANDBOX / "state" / "run.lock"))
|
||||
os.environ.setdefault("YTS_VENV_BIN", str(_SANDBOX / "venv" / "bin"))
|
||||
|
||||
import pytest # noqa: E402
|
||||
|
||||
from ytstream import api, config, db, util, videos # noqa: E402
|
||||
from ytstream.settings import Settings # noqa: E402
|
||||
|
||||
FIXTURES = Path(__file__).parent / "fixtures"
|
||||
|
||||
# A real-looking channel id: UC + 22 chars.
|
||||
CHANNEL_ID = "UCW7jUEpYT_t0Gsf632d6_wQ"
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def media_root(tmp_path, monkeypatch):
|
||||
"""Point the media root at a per-test tmpdir."""
|
||||
root = tmp_path / "media"
|
||||
root.mkdir(parents=True)
|
||||
monkeypatch.setattr(config, "MEDIA_ROOT", root)
|
||||
return root
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def conn(tmp_path):
|
||||
connection = db.connect(tmp_path / "ytstream.db")
|
||||
yield connection
|
||||
connection.close()
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def settings(conn):
|
||||
settings = Settings(conn)
|
||||
# Tests that reach the API go through FakeApi, but the code refuses to call
|
||||
# out at all without a key, so give it one that is never used for real.
|
||||
settings.set("youtube_api_key", "test-key")
|
||||
return settings
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def channel(conn):
|
||||
"""One subscribed channel, returned as a row."""
|
||||
return add_channel(conn, CHANNEL_ID, "clabretro", "clabretro")
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def no_network(monkeypatch):
|
||||
"""Fail loudly if anything tries to open a socket.
|
||||
|
||||
Belt and braces: a test that accidentally hits the network would pass locally
|
||||
and fail in a different week for reasons nobody could reproduce.
|
||||
"""
|
||||
import urllib.request
|
||||
|
||||
def forbidden(*args, **kwargs):
|
||||
raise AssertionError("test attempted a network call")
|
||||
|
||||
monkeypatch.setattr(urllib.request, "urlopen", forbidden)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------- helpers
|
||||
|
||||
|
||||
def add_channel(conn, channel_id: str, title: str, dir_name: str, **kwargs):
|
||||
fields = {"source": "youtube", "backfilled": 1, "uploads_playlist": "UULF"}
|
||||
fields.update(kwargs)
|
||||
columns = ", ".join(fields)
|
||||
marks = ", ".join("?" * len(fields))
|
||||
with conn:
|
||||
conn.execute(
|
||||
f"INSERT INTO channel (channel_id, handle, title, description, dir_name, "
|
||||
f"added_at, {columns}) VALUES (?, ?, ?, ?, ?, ?, {marks})",
|
||||
(channel_id, f"@{dir_name}", title, "A channel", dir_name,
|
||||
util.utcnow_iso(), *fields.values()),
|
||||
)
|
||||
return conn.execute(
|
||||
"SELECT * FROM channel WHERE channel_id = ?", (channel_id,)
|
||||
).fetchone()
|
||||
|
||||
|
||||
def add_video(conn, channel_pk, video_id, **kwargs):
|
||||
"""Insert a video row with sensible defaults."""
|
||||
defaults = {
|
||||
"title": f"Video {video_id}",
|
||||
"upload_date": "2026-08-01",
|
||||
"state": videos.LISTED,
|
||||
"discovery_source": videos.SOURCE_UULF,
|
||||
"duration": 900,
|
||||
}
|
||||
defaults.update(kwargs)
|
||||
videos.insert(conn, channel_pk=channel_pk, video_id=video_id, **defaults)
|
||||
return videos.get(conn, video_id)
|
||||
|
||||
|
||||
def feed_bytes(name: str) -> bytes:
|
||||
return (FIXTURES / name).read_bytes()
|
||||
|
||||
|
||||
def make_feed(entries: list[dict], *, playlist_published: str = "2019-01-01T00:00:00+00:00") -> bytes:
|
||||
"""Build an Atom feed shaped like YouTube's.
|
||||
|
||||
Includes the feed-level <published> that is NOT an entry, because scraping
|
||||
timestamps instead of walking atom:entry picks it up and yields nonsense
|
||||
upload rates — a mistake made once while measuring.
|
||||
"""
|
||||
items = "".join(
|
||||
f"""
|
||||
<entry>
|
||||
<id>yt:video:{e['video_id']}</id>
|
||||
<yt:videoId>{e['video_id']}</yt:videoId>
|
||||
<title>{e.get('title', 'Untitled')}</title>
|
||||
<published>{e['published']}</published>
|
||||
</entry>"""
|
||||
for e in entries
|
||||
)
|
||||
return f"""<?xml version="1.0" encoding="UTF-8"?>
|
||||
<feed xmlns:yt="http://www.youtube.com/xml/schemas/2015"
|
||||
xmlns:media="http://search.yahoo.com/mrss/"
|
||||
xmlns="http://www.w3.org/2005/Atom">
|
||||
<title>Videos</title>
|
||||
<published>{playlist_published}</published>{items}
|
||||
</feed>""".encode()
|
||||
|
||||
|
||||
class FakeApi:
|
||||
"""Stand-in for `api.Api` with scriptable responses.
|
||||
|
||||
Counts calls so tests can assert the batching actually batches — the whole
|
||||
quota argument in the plan rests on 50 ids per call, and a regression to one
|
||||
call per video would be silent and expensive.
|
||||
"""
|
||||
|
||||
def __init__(self, *, subs=None, uploads=None, durations=None, channel=None,
|
||||
raises=None):
|
||||
self._subs = subs
|
||||
self._uploads = uploads or []
|
||||
self._durations = durations or {}
|
||||
self._channel = channel
|
||||
self._raises = raises
|
||||
self.calls = 0
|
||||
self.duration_calls = 0
|
||||
self.subscription_calls = 0
|
||||
|
||||
def subscriptions(self, channel_id):
|
||||
self.subscription_calls += 1
|
||||
self.calls += 1
|
||||
if isinstance(self._raises, Exception):
|
||||
raise self._raises
|
||||
return list(self._subs or [])
|
||||
|
||||
def uploads(self, channel_id, *, kind="UULF", since=None, limit=None,
|
||||
page_token=None):
|
||||
self.calls += 1
|
||||
produced = 0
|
||||
for entry, token in self._uploads:
|
||||
if since is not None and entry["published"] < since:
|
||||
return
|
||||
yield entry, token
|
||||
produced += 1
|
||||
if limit is not None and produced >= limit:
|
||||
return
|
||||
|
||||
def durations(self, video_ids):
|
||||
self.duration_calls += 1
|
||||
self.calls += 1
|
||||
return {vid: self._durations[vid] for vid in video_ids
|
||||
if vid in self._durations}
|
||||
|
||||
def channel(self, channel_id):
|
||||
self.calls += 1
|
||||
return self._channel
|
||||
|
||||
def resolve_handle(self, handle):
|
||||
self.calls += 1
|
||||
return self._channel
|
||||
|
||||
|
||||
def patch_api(monkeypatch, module, fake: FakeApi):
|
||||
"""Make `module.api.Api(...)` return `fake` regardless of arguments."""
|
||||
monkeypatch.setattr(module.api, "Api", lambda *a, **kw: fake)
|
||||
return fake
|
||||
Vendored
+37
@@ -0,0 +1,37 @@
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<feed xmlns:yt="http://www.youtube.com/xml/schemas/2015"
|
||||
xmlns:media="http://search.yahoo.com/mrss/"
|
||||
xmlns="http://www.w3.org/2005/Atom">
|
||||
<id>yt:playlist:UULFW7jUEpYT_t0Gsf632d6_wQ</id>
|
||||
<title>Uploads from clabretro</title>
|
||||
<entry>
|
||||
<id>yt:video:08Ajr5fP52I</id>
|
||||
<yt:videoId>08Ajr5fP52I</yt:videoId>
|
||||
<yt:channelId>UCW7jUEpYT_t0Gsf632d6_wQ</yt:channelId>
|
||||
<title>Learning to Design 3D Prints</title>
|
||||
<published>2026-08-07T15:00:11+00:00</published>
|
||||
<media:group>
|
||||
<media:description>Tinkercad & a cheap printer. Part 1/3 <of a series>.</media:description>
|
||||
</media:group>
|
||||
</entry>
|
||||
<entry>
|
||||
<id>yt:video:8k8nAQq0s_s</id>
|
||||
<yt:videoId>8k8nAQq0s_s</yt:videoId>
|
||||
<yt:channelId>UCW7jUEpYT_t0Gsf632d6_wQ</yt:channelId>
|
||||
<title>Trying to use a Nortel PBX: part two</title>
|
||||
<published>2026-08-02T14:30:00+00:00</published>
|
||||
<media:group>
|
||||
<media:description>Telephony experiments.</media:description>
|
||||
</media:group>
|
||||
</entry>
|
||||
<entry>
|
||||
<id>yt:video:vcYYcQyecNQ</id>
|
||||
<yt:videoId>vcYYcQyecNQ</yt:videoId>
|
||||
<yt:channelId>UCW7jUEpYT_t0Gsf632d6_wQ</yt:channelId>
|
||||
<title>IBM Director on an xSeries 346 from 2004</title>
|
||||
<published>2026-06-17T12:00:00+00:00</published>
|
||||
<media:group>
|
||||
<media:description>Old enterprise management software.</media:description>
|
||||
</media:group>
|
||||
</entry>
|
||||
</feed>
|
||||
@@ -0,0 +1,329 @@
|
||||
"""The YouTube Data API client, with no YouTube involved.
|
||||
|
||||
Two things here earn their keep beyond ordinary coverage:
|
||||
|
||||
* **Error classification.** Three distinct operator mistakes all arrive as HTTP
|
||||
403 `forbidden`, and the only reliable signal is `error.details[].reason`. That
|
||||
cost three attempts to diagnose by hand once; it should cost nobody a second one.
|
||||
* **`totalResults` is not the list length.** It reported 127 against 119 actually
|
||||
returned, because terminated and private channels still count as subscriptions.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import io
|
||||
import json
|
||||
import urllib.error
|
||||
|
||||
import pytest
|
||||
|
||||
from ytstream import api
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- durations
|
||||
|
||||
|
||||
@pytest.mark.parametrize("text,expected", [
|
||||
("PT1H2M3S", 3723),
|
||||
("PT21M", 1260),
|
||||
("PT45S", 45),
|
||||
("P1DT2H", 93600),
|
||||
("P1W", 604800),
|
||||
("PT0S", 0), # live and upcoming videos report this
|
||||
("", None),
|
||||
("garbage", None),
|
||||
("1H2M", None),
|
||||
])
|
||||
def test_parse_duration(text, expected):
|
||||
assert api.parse_duration(text) == expected
|
||||
|
||||
|
||||
def test_parse_published_keeps_the_exact_string():
|
||||
"""The date drives naming; the exact stamp is kept because approximate_date
|
||||
is wrong by up to two days and episode numbers cannot be re-derived later."""
|
||||
day, exact = api.parse_published("2026-08-11T16:32:10Z")
|
||||
assert day.isoformat() == "2026-08-11"
|
||||
assert exact == "2026-08-11T16:32:10Z"
|
||||
|
||||
|
||||
def test_parse_published_normalises_to_utc():
|
||||
day, _ = api.parse_published("2026-08-11T23:30:00-08:00")
|
||||
assert day.isoformat() == "2026-08-12"
|
||||
|
||||
|
||||
def test_parse_published_tolerates_rubbish():
|
||||
assert api.parse_published("") == (None, None)
|
||||
assert api.parse_published("not a date")[0] is None
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- playlists
|
||||
|
||||
|
||||
def test_uploads_playlist_id():
|
||||
assert api.uploads_playlist_id("UCjCJ2LaOIsPzOoXUTMDI3wg") == "UULFjCJ2LaOIsPzOoXUTMDI3wg"
|
||||
assert api.uploads_playlist_id("UCjCJ2LaOIsPzOoXUTMDI3wg", "UU") == "UUjCJ2LaOIsPzOoXUTMDI3wg"
|
||||
|
||||
|
||||
def test_uploads_playlist_id_rejects_non_channel_ids():
|
||||
with pytest.raises(ValueError):
|
||||
api.uploads_playlist_id("PLnotachannel")
|
||||
|
||||
|
||||
# -------------------------------------------------------- error classification
|
||||
|
||||
|
||||
def _http_error(code, body):
|
||||
return urllib.error.HTTPError(
|
||||
"https://example/", code, "Forbidden", {},
|
||||
io.BytesIO(json.dumps(body).encode()),
|
||||
)
|
||||
|
||||
|
||||
def _classify(code, body):
|
||||
return api.Api("k")._classify(_http_error(code, body))
|
||||
|
||||
|
||||
def test_service_disabled_is_not_configured():
|
||||
"""The API is not enabled on the project. Carries an activation URL."""
|
||||
error = _classify(403, {"error": {
|
||||
"code": 403,
|
||||
"message": "YouTube Data API v3 has not been used in project 510818173753 "
|
||||
"before or it is disabled.",
|
||||
"errors": [{"reason": "accessNotConfigured"}],
|
||||
"details": [{"reason": "SERVICE_DISABLED",
|
||||
"metadata": {"activationUrl": "https://console/enable"}}],
|
||||
}})
|
||||
assert isinstance(error, api.NotConfigured)
|
||||
|
||||
|
||||
def test_key_service_blocked_is_not_configured():
|
||||
"""A different mistake entirely — the key's own API restrictions — and it
|
||||
reports only `forbidden` in errors[]."""
|
||||
error = _classify(403, {"error": {
|
||||
"code": 403,
|
||||
"message": "Requests to this API youtube method "
|
||||
"youtube.api.v3.V3DataVideoService.List are blocked.",
|
||||
"errors": [{"reason": "forbidden"}],
|
||||
"details": [{"reason": "API_KEY_SERVICE_BLOCKED"}],
|
||||
}})
|
||||
assert isinstance(error, api.NotConfigured)
|
||||
|
||||
|
||||
def test_subscription_forbidden_is_its_own_type():
|
||||
"""Must never be mistaken for an empty subscription list."""
|
||||
error = _classify(403, {"error": {
|
||||
"code": 403,
|
||||
"message": "The requester is not allowed to access the requested subscriptions.",
|
||||
"errors": [{"reason": "subscriptionForbidden"}],
|
||||
}})
|
||||
assert isinstance(error, api.SubscriptionsPrivate)
|
||||
assert not isinstance(error, api.NotConfigured)
|
||||
|
||||
|
||||
def test_transient_server_error_is_plain_api_error():
|
||||
error = _classify(500, {"error": {
|
||||
"code": 500, "message": "Backend Error",
|
||||
"errors": [{"reason": "backendError"}],
|
||||
}})
|
||||
assert type(error) is api.ApiError
|
||||
|
||||
|
||||
def test_quota_exceeded_is_plain_api_error():
|
||||
"""Retrying tomorrow helps, so it must not be classed as misconfiguration."""
|
||||
error = _classify(403, {"error": {
|
||||
"code": 403, "message": "The request cannot be completed because you have "
|
||||
"exceeded your quota.",
|
||||
"errors": [{"reason": "quotaExceeded"}],
|
||||
}})
|
||||
assert type(error) is api.ApiError
|
||||
|
||||
|
||||
def test_unparseable_body_still_yields_an_error():
|
||||
exc = urllib.error.HTTPError(
|
||||
"https://example/", 502, "Bad Gateway", {}, io.BytesIO(b"<html>nope</html>")
|
||||
)
|
||||
error = api.Api("k")._classify(exc)
|
||||
assert isinstance(error, api.ApiError)
|
||||
assert error.status == 502
|
||||
|
||||
|
||||
def test_missing_key_raises_before_any_request(monkeypatch):
|
||||
import urllib.request
|
||||
|
||||
monkeypatch.setattr(urllib.request, "urlopen", lambda *a, **k:
|
||||
(_ for _ in ()).throw(AssertionError("should not be called")))
|
||||
with pytest.raises(api.NotConfigured):
|
||||
api.Api("").durations(["dQw4w9WgXcQ"])
|
||||
|
||||
|
||||
# -------------------------------------------------------------- request shapes
|
||||
|
||||
|
||||
class Recorder:
|
||||
"""Captures the URLs requested and replays canned pages."""
|
||||
|
||||
def __init__(self, pages):
|
||||
self.pages = list(pages)
|
||||
self.urls = []
|
||||
|
||||
def __call__(self, request, timeout=None):
|
||||
self.urls.append(request.full_url)
|
||||
payload = json.dumps(self.pages.pop(0)).encode()
|
||||
|
||||
class Response(io.BytesIO):
|
||||
def __enter__(self_inner):
|
||||
return self_inner
|
||||
|
||||
def __exit__(self_inner, *exc):
|
||||
return False
|
||||
|
||||
return Response(payload)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def recorder(monkeypatch):
|
||||
def install(pages):
|
||||
rec = Recorder(pages)
|
||||
monkeypatch.setattr(api.urllib.request, "urlopen", rec)
|
||||
return rec
|
||||
return install
|
||||
|
||||
|
||||
def test_subscriptions_pages_to_exhaustion(recorder):
|
||||
rec = recorder([
|
||||
{"pageInfo": {"totalResults": 127},
|
||||
"items": [{"snippet": {"title": f"C{i}",
|
||||
"resourceId": {"channelId": f"UC{i:022d}"}}}
|
||||
for i in range(50)],
|
||||
"nextPageToken": "T2"},
|
||||
{"pageInfo": {"totalResults": 127},
|
||||
"items": [{"snippet": {"title": "Last",
|
||||
"resourceId": {"channelId": "UC" + "z" * 22}}}]},
|
||||
])
|
||||
|
||||
found = api.Api("k").subscriptions("UCbrother")
|
||||
|
||||
assert len(found) == 51
|
||||
assert len(rec.urls) == 2
|
||||
assert "pageToken=T2" in rec.urls[1]
|
||||
|
||||
|
||||
def test_subscriptions_returns_the_fetched_list_not_total_results(recorder):
|
||||
"""Measured: totalResults said 127, pagination returned 119."""
|
||||
recorder([{"pageInfo": {"totalResults": 127},
|
||||
"items": [{"snippet": {"title": "One",
|
||||
"resourceId": {"channelId": "UC" + "a" * 22}}}]}])
|
||||
|
||||
found = api.Api("k").subscriptions("UCbrother")
|
||||
|
||||
assert len(found) == 1
|
||||
|
||||
|
||||
def test_subscriptions_deduplicates(recorder):
|
||||
recorder([{"items": [
|
||||
{"snippet": {"title": "Dup", "resourceId": {"channelId": "UC" + "a" * 22}}},
|
||||
{"snippet": {"title": "Dup", "resourceId": {"channelId": "UC" + "a" * 22}}},
|
||||
]}])
|
||||
assert len(api.Api("k").subscriptions("UCbrother")) == 1
|
||||
|
||||
|
||||
def test_subscriptions_skips_entries_without_a_channel_id(recorder):
|
||||
recorder([{"items": [
|
||||
{"snippet": {"title": "Broken", "resourceId": {}}},
|
||||
{"snippet": {"title": "Fine", "resourceId": {"channelId": "UC" + "a" * 22}}},
|
||||
]}])
|
||||
assert [entry["title"] for entry in api.Api("k").subscriptions("UCb")] == ["Fine"]
|
||||
|
||||
|
||||
def test_durations_batches_fifty_ids_per_call(recorder):
|
||||
"""The whole quota argument rests on this. One call per video would be a
|
||||
silent 50x regression."""
|
||||
ids = [f"vid{i:08d}" for i in range(120)]
|
||||
rec = recorder([
|
||||
{"items": [{"id": vid, "contentDetails": {"duration": "PT10M"}}
|
||||
for vid in ids[start:start + 50]]}
|
||||
for start in (0, 50, 100)
|
||||
])
|
||||
|
||||
found = api.Api("k").durations(ids)
|
||||
|
||||
assert len(found) == 120
|
||||
assert len(rec.urls) == 3
|
||||
|
||||
|
||||
def test_durations_flags_livestreams(recorder):
|
||||
recorder([{"items": [
|
||||
{"id": "live0000001", "contentDetails": {"duration": "PT0S"},
|
||||
"liveStreamingDetails": {"actualStartTime": "2026-08-01T00:00:00Z"}},
|
||||
{"id": "vod00000001", "contentDetails": {"duration": "PT30M"}},
|
||||
]}])
|
||||
|
||||
found = api.Api("k").durations(["live0000001", "vod00000001"])
|
||||
|
||||
assert found["live0000001"]["is_live"] is True
|
||||
assert found["vod00000001"]["is_live"] is False
|
||||
|
||||
|
||||
def test_uploads_stops_at_the_since_date(recorder):
|
||||
recorder([{"items": [
|
||||
{"contentDetails": {"videoId": "new00000001",
|
||||
"videoPublishedAt": "2026-08-10T00:00:00Z"}},
|
||||
{"contentDetails": {"videoId": "old00000001",
|
||||
"videoPublishedAt": "2026-01-01T00:00:00Z"}},
|
||||
{"contentDetails": {"videoId": "old00000002",
|
||||
"videoPublishedAt": "2025-01-01T00:00:00Z"}},
|
||||
], "nextPageToken": "T2"}])
|
||||
|
||||
from datetime import date
|
||||
|
||||
got = list(api.Api("k").uploads("UC" + "a" * 22, since=date(2026, 8, 1)))
|
||||
|
||||
assert [entry["video_id"] for entry, _ in got] == ["new00000001"]
|
||||
|
||||
|
||||
def test_uploads_honours_the_limit(recorder):
|
||||
recorder([{"items": [
|
||||
{"contentDetails": {"videoId": f"vid{i:08d}",
|
||||
"videoPublishedAt": "2026-08-10T00:00:00Z"}}
|
||||
for i in range(10)
|
||||
]}])
|
||||
|
||||
got = list(api.Api("k").uploads("UC" + "a" * 22, limit=3))
|
||||
|
||||
assert len(got) == 3
|
||||
|
||||
|
||||
def test_uploads_skips_entries_with_no_publish_date(recorder):
|
||||
"""A private or deleted video keeps its playlist slot but loses its date.
|
||||
Skipping is right; stopping would truncate the backfill at that point."""
|
||||
recorder([{"items": [
|
||||
{"contentDetails": {"videoId": "priv0000001"}},
|
||||
{"contentDetails": {"videoId": "good0000001",
|
||||
"videoPublishedAt": "2026-08-10T00:00:00Z"}},
|
||||
]}])
|
||||
|
||||
got = list(api.Api("k").uploads("UC" + "a" * 22))
|
||||
|
||||
assert [entry["video_id"] for entry, _ in got] == ["good0000001"]
|
||||
|
||||
|
||||
def test_uploads_yields_the_next_page_token_for_resumability(recorder):
|
||||
recorder([{"items": [
|
||||
{"contentDetails": {"videoId": "vid00000001",
|
||||
"videoPublishedAt": "2026-08-10T00:00:00Z"}}
|
||||
], "nextPageToken": "CARRY_ON"}])
|
||||
|
||||
# Take only the first entry: the generator would otherwise follow the token
|
||||
# on to a page the recorder has not been given, which is correct paging
|
||||
# behaviour and covered elsewhere.
|
||||
_, token = next(api.Api("k").uploads("UC" + "a" * 22))
|
||||
|
||||
assert token == "CARRY_ON"
|
||||
|
||||
|
||||
def test_call_counter_tracks_quota(recorder):
|
||||
recorder([{"items": []}, {"items": []}])
|
||||
client = api.Api("k")
|
||||
client.durations(["a"])
|
||||
client.durations(["b"])
|
||||
assert client.calls == 2
|
||||
@@ -0,0 +1,203 @@
|
||||
"""Password hashing, session cookies, CSRF tokens and login throttling."""
|
||||
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
from ytstream.web import auth
|
||||
|
||||
|
||||
class TestPasswords:
|
||||
def test_round_trip(self):
|
||||
stored = auth.hash_password("correct horse battery staple")
|
||||
assert auth.verify_password(stored, "correct horse battery staple")
|
||||
|
||||
def test_wrong_password_rejected(self):
|
||||
stored = auth.hash_password("secret")
|
||||
assert not auth.verify_password(stored, "Secret")
|
||||
assert not auth.verify_password(stored, "")
|
||||
|
||||
def test_salt_makes_hashes_unique(self):
|
||||
assert auth.hash_password("same") != auth.hash_password("same")
|
||||
|
||||
def test_hash_is_not_the_plaintext(self):
|
||||
assert "secret" not in auth.hash_password("secret")
|
||||
|
||||
def test_empty_stored_hash_rejects_everything(self):
|
||||
assert not auth.verify_password("", "anything")
|
||||
|
||||
def test_malformed_stored_hash_does_not_raise(self):
|
||||
for junk in ("nonsense", "scrypt$bad", "a$b$c$d$e$f", "scrypt$x$y$z$q$r"):
|
||||
assert auth.verify_password(junk, "anything") is False
|
||||
|
||||
def test_unicode_password(self):
|
||||
stored = auth.hash_password("pässwörd🎬")
|
||||
assert auth.verify_password(stored, "pässwörd🎬")
|
||||
|
||||
|
||||
class TestSessions:
|
||||
def test_issue_and_verify(self):
|
||||
secret = auth.new_secret()
|
||||
token = auth.issue_session(secret)
|
||||
assert auth.verify_session(secret, token)
|
||||
|
||||
def test_a_different_secret_rejects(self):
|
||||
token = auth.issue_session(auth.new_secret())
|
||||
assert not auth.verify_session(auth.new_secret(), token)
|
||||
|
||||
def test_tampered_payload_rejected(self):
|
||||
secret = auth.new_secret()
|
||||
token = auth.issue_session(secret)
|
||||
payload, signature = token.split(".", 1)
|
||||
assert not auth.verify_session(secret, f"{payload}x.{signature}")
|
||||
|
||||
def test_tampered_signature_rejected(self):
|
||||
secret = auth.new_secret()
|
||||
payload, _ = auth.issue_session(secret).split(".", 1)
|
||||
assert not auth.verify_session(secret, f"{payload}.deadbeef")
|
||||
|
||||
def test_garbage_rejected(self):
|
||||
secret = auth.new_secret()
|
||||
for junk in ("", "no-dot", "a.b.c", "...."):
|
||||
assert auth.verify_session(secret, junk) is False
|
||||
|
||||
def test_expires_after_a_year(self):
|
||||
secret = auth.new_secret()
|
||||
issued = time.time() - auth.SESSION_MAX_AGE - 10
|
||||
token = auth.issue_session(secret, issued_at=issued)
|
||||
assert not auth.verify_session(secret, token)
|
||||
|
||||
def test_still_valid_just_inside_a_year(self):
|
||||
secret = auth.new_secret()
|
||||
issued = time.time() - auth.SESSION_MAX_AGE + 60
|
||||
token = auth.issue_session(secret, issued_at=issued)
|
||||
assert auth.verify_session(secret, token)
|
||||
|
||||
def test_a_token_from_the_future_is_rejected(self):
|
||||
secret = auth.new_secret()
|
||||
token = auth.issue_session(secret, issued_at=time.time() + 3600)
|
||||
assert not auth.verify_session(secret, token)
|
||||
|
||||
|
||||
class TestCookie:
|
||||
def test_carries_the_hardening_flags(self):
|
||||
header = auth.cookie_header("abc")
|
||||
for flag in ("HttpOnly", "Secure", "SameSite=Lax", "Path=/"):
|
||||
assert flag in header
|
||||
assert f"Max-Age={auth.SESSION_MAX_AGE}" in header
|
||||
|
||||
def test_secure_can_be_omitted_for_local_http_testing(self):
|
||||
assert "Secure" not in auth.cookie_header("abc", secure=False)
|
||||
|
||||
def test_clear_cookie_expires_immediately(self):
|
||||
assert "Max-Age=0" in auth.clear_cookie_header()
|
||||
|
||||
|
||||
class TestCookieParsing:
|
||||
"""Regression tests for a real failure: http.cookies.SimpleCookie silently
|
||||
drops everything after a value it dislikes, which made valid sessions
|
||||
invisible and bounced users back to the login page with no error."""
|
||||
|
||||
def test_finds_our_cookie_alone(self):
|
||||
assert auth.cookie_value("yta_session=abc") == "abc"
|
||||
|
||||
def test_finds_it_after_a_neighbour(self):
|
||||
assert auth.cookie_value("sessionid=xyz; yta_session=abc") == "abc"
|
||||
|
||||
def test_finds_it_before_a_neighbour(self):
|
||||
assert auth.cookie_value("yta_session=abc; sessionid=xyz") == "abc"
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"neighbour",
|
||||
[
|
||||
'prefs={"a":1}', # JSON value — what actually broke it
|
||||
"junk=[1,2,3]",
|
||||
"weird=a b c",
|
||||
"empty=",
|
||||
"novalue",
|
||||
"quoted=\"has spaces\"",
|
||||
"path=/a/b/c",
|
||||
"colons=a:b:c",
|
||||
"comma=a,b",
|
||||
],
|
||||
)
|
||||
def test_survives_hostile_neighbours(self, neighbour):
|
||||
assert auth.cookie_value(f"{neighbour}; yta_session=abc") == "abc"
|
||||
assert auth.cookie_value(f"yta_session=abc; {neighbour}") == "abc"
|
||||
|
||||
def test_strips_surrounding_quotes(self):
|
||||
assert auth.cookie_value('yta_session="abc"') == "abc"
|
||||
|
||||
def test_tolerates_whitespace(self):
|
||||
assert auth.cookie_value(" yta_session = abc ") == "abc"
|
||||
|
||||
def test_absent_cookie_returns_empty(self):
|
||||
assert auth.cookie_value("sessionid=xyz") == ""
|
||||
|
||||
def test_empty_header_returns_empty(self):
|
||||
assert auth.cookie_value("") == ""
|
||||
assert auth.cookie_value(None) == ""
|
||||
|
||||
def test_does_not_match_a_name_that_merely_contains_ours(self):
|
||||
assert auth.cookie_value("not_yta_session=nope") == ""
|
||||
|
||||
def test_real_token_round_trips_through_the_header(self):
|
||||
secret = auth.new_secret()
|
||||
token = auth.issue_session(secret)
|
||||
header = f'prefs={{"theme":"dark"}}; yta_session={token}; other=1'
|
||||
assert auth.verify_session(secret, auth.cookie_value(header))
|
||||
|
||||
|
||||
class TestCsrf:
|
||||
def test_token_verifies(self):
|
||||
secret, session = auth.new_secret(), auth.issue_session(auth.new_secret())
|
||||
token = auth.csrf_token(secret, session)
|
||||
assert auth.verify_csrf(secret, session, token)
|
||||
|
||||
def test_token_is_bound_to_the_session(self):
|
||||
secret = auth.new_secret()
|
||||
one = auth.issue_session(secret, issued_at=1000)
|
||||
two = auth.issue_session(secret, issued_at=2000)
|
||||
assert not auth.verify_csrf(secret, two, auth.csrf_token(secret, one))
|
||||
|
||||
def test_empty_token_rejected(self):
|
||||
secret, session = auth.new_secret(), "sess"
|
||||
assert not auth.verify_csrf(secret, session, "")
|
||||
|
||||
def test_wrong_token_rejected(self):
|
||||
secret, session = auth.new_secret(), "sess"
|
||||
assert not auth.verify_csrf(secret, session, "deadbeef")
|
||||
|
||||
|
||||
class TestThrottle:
|
||||
def test_allows_up_to_the_limit(self):
|
||||
throttle = auth.LoginThrottle(max_failures=3, lockout=60)
|
||||
for _ in range(2):
|
||||
throttle.record_failure("1.2.3.4", now=1000)
|
||||
assert not throttle.locked("1.2.3.4", now=1000)
|
||||
|
||||
def test_locks_after_the_limit(self):
|
||||
throttle = auth.LoginThrottle(max_failures=3, lockout=60)
|
||||
for _ in range(3):
|
||||
throttle.record_failure("1.2.3.4", now=1000)
|
||||
assert throttle.locked("1.2.3.4", now=1000)
|
||||
|
||||
def test_lock_expires(self):
|
||||
throttle = auth.LoginThrottle(max_failures=3, lockout=60)
|
||||
for _ in range(3):
|
||||
throttle.record_failure("1.2.3.4", now=1000)
|
||||
assert not throttle.locked("1.2.3.4", now=1061)
|
||||
|
||||
def test_success_clears_the_counter(self):
|
||||
throttle = auth.LoginThrottle(max_failures=3, lockout=60)
|
||||
for _ in range(2):
|
||||
throttle.record_failure("1.2.3.4", now=1000)
|
||||
throttle.record_success("1.2.3.4")
|
||||
assert not throttle.locked("1.2.3.4", now=1000)
|
||||
|
||||
def test_addresses_are_tracked_separately(self):
|
||||
throttle = auth.LoginThrottle(max_failures=2, lockout=60)
|
||||
for _ in range(2):
|
||||
throttle.record_failure("1.1.1.1", now=1000)
|
||||
assert throttle.locked("1.1.1.1", now=1000)
|
||||
assert not throttle.locked("2.2.2.2", now=1000)
|
||||
@@ -0,0 +1,151 @@
|
||||
"""Channel subscribe/unsubscribe, and the API-only sync path."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from ytstream import api, channels, config
|
||||
|
||||
from conftest import CHANNEL_ID, FakeApi, add_channel, add_video
|
||||
|
||||
|
||||
def test_normalise_url_accepts_every_reference_form():
|
||||
assert channels.normalise_url(CHANNEL_ID).endswith("/channel/" + CHANNEL_ID)
|
||||
assert channels.normalise_url("@clabretro") == "https://www.youtube.com/@clabretro"
|
||||
assert channels.normalise_url("clabretro") == "https://www.youtube.com/@clabretro"
|
||||
assert channels.normalise_url("https://youtube.com/x") == "https://youtube.com/x"
|
||||
assert channels.normalise_url("youtube.com/x") == "https://youtube.com/x"
|
||||
|
||||
|
||||
def test_normalise_url_rejects_nonsense():
|
||||
with pytest.raises(channels.ResolutionError):
|
||||
channels.normalise_url("")
|
||||
with pytest.raises(channels.ResolutionError):
|
||||
channels.normalise_url("not a handle!!")
|
||||
|
||||
|
||||
def test_uulf_playlist_id():
|
||||
assert channels.uulf_playlist_id(CHANNEL_ID) == "UULF" + CHANNEL_ID[2:]
|
||||
|
||||
|
||||
# ---------------------------------------------------------- subscribe_from_sync
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def api_channel(monkeypatch):
|
||||
"""Patch channels' lazily-imported api module."""
|
||||
fake = FakeApi(channel={"channel_id": "UC" + "a" * 22, "title": "Alpha Channel",
|
||||
"description": "About alpha", "handle": "@alpha",
|
||||
"avatar_url": None})
|
||||
monkeypatch.setattr(api, "Api", lambda *a, **kw: fake)
|
||||
return fake
|
||||
|
||||
|
||||
def test_subscribe_from_sync_never_calls_yt_dlp(conn, settings, media_root,
|
||||
api_channel, monkeypatch):
|
||||
"""119 channels' worth of yt-dlp resolution is the request burst this design
|
||||
exists to avoid."""
|
||||
monkeypatch.setattr(channels.ytdlp, "run_json",
|
||||
lambda *a, **k: pytest.fail("yt-dlp must not be called"))
|
||||
|
||||
row = channels.subscribe_from_sync(conn, settings, "UC" + "a" * 22, "Alpha")
|
||||
|
||||
assert row["title"] == "Alpha Channel"
|
||||
assert row["source"] == "youtube"
|
||||
|
||||
|
||||
def test_subscribe_from_sync_uses_the_api_description(conn, settings, media_root,
|
||||
api_channel):
|
||||
row = channels.subscribe_from_sync(conn, settings, "UC" + "a" * 22, "Alpha")
|
||||
assert row["description"] == "About alpha"
|
||||
|
||||
|
||||
def test_subscribe_from_sync_is_idempotent(conn, settings, media_root, api_channel):
|
||||
first = channels.subscribe_from_sync(conn, settings, "UC" + "a" * 22, "Alpha")
|
||||
second = channels.subscribe_from_sync(conn, settings, "UC" + "a" * 22, "Alpha")
|
||||
assert first["id"] == second["id"]
|
||||
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 1
|
||||
|
||||
|
||||
def test_subscribe_from_sync_survives_a_metadata_failure(conn, settings, media_root,
|
||||
monkeypatch):
|
||||
"""A channel we cannot describe is still a channel we can mirror."""
|
||||
class Failing(FakeApi):
|
||||
def channel(self, channel_id):
|
||||
raise api.ApiError(500, "backendError", "boom")
|
||||
|
||||
monkeypatch.setattr(api, "Api", lambda *a, **kw: Failing())
|
||||
|
||||
row = channels.subscribe_from_sync(conn, settings, "UC" + "a" * 22, "Fallback Name")
|
||||
|
||||
assert row["title"] == "Fallback Name"
|
||||
|
||||
|
||||
def test_subscribe_from_sync_makes_no_directory(conn, settings, media_root,
|
||||
monkeypatch):
|
||||
"""The tree appears with the first episode, so a channel with nothing inside
|
||||
the window leaves no empty series behind."""
|
||||
class NoArt(FakeApi):
|
||||
def channel(self, channel_id):
|
||||
return {"channel_id": channel_id, "title": "Quiet", "description": "",
|
||||
"handle": None, "avatar_url": None}
|
||||
|
||||
monkeypatch.setattr(api, "Api", lambda *a, **kw: NoArt())
|
||||
|
||||
row = channels.subscribe_from_sync(conn, settings, "UC" + "a" * 22, "Quiet")
|
||||
|
||||
assert not (media_root / row["dir_name"]).exists()
|
||||
|
||||
|
||||
def test_directory_names_are_deduplicated(conn, settings, media_root, monkeypatch):
|
||||
"""Two channels can legitimately share a title; dir_name is UNIQUE."""
|
||||
class Same(FakeApi):
|
||||
def channel(self, channel_id):
|
||||
return {"channel_id": channel_id, "title": "Same Name",
|
||||
"description": "", "handle": None, "avatar_url": None}
|
||||
|
||||
monkeypatch.setattr(api, "Api", lambda *a, **kw: Same())
|
||||
|
||||
first = channels.subscribe_from_sync(conn, settings, "UC" + "a" * 22, "Same Name")
|
||||
second = channels.subscribe_from_sync(conn, settings, "UC" + "b" * 22, "Same Name")
|
||||
|
||||
assert first["dir_name"] == "Same Name"
|
||||
assert second["dir_name"] == "Same Name (2)"
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ unsubscribe
|
||||
|
||||
|
||||
def test_unsubscribe_removes_the_tree_and_the_rows(conn, media_root, channel):
|
||||
tree = media_root / "clabretro"
|
||||
(tree / "Season 2026").mkdir(parents=True)
|
||||
(tree / "tvshow.nfo").write_text("<tvshow/>")
|
||||
add_video(conn, channel["id"], "vid00000001")
|
||||
|
||||
title = channels.unsubscribe(conn, channel["id"])
|
||||
|
||||
assert title == "clabretro"
|
||||
assert not tree.exists()
|
||||
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 0
|
||||
assert conn.execute("SELECT COUNT(*) FROM video").fetchone()[0] == 0
|
||||
|
||||
|
||||
def test_unsubscribe_rejects_an_unknown_id(conn):
|
||||
with pytest.raises(LookupError):
|
||||
channels.unsubscribe(conn, 999)
|
||||
|
||||
|
||||
def test_lookup_helpers(conn, channel):
|
||||
assert channels.get(conn, channel["id"])["title"] == "clabretro"
|
||||
assert channels.get_by_channel_id(conn, CHANNEL_ID)["id"] == channel["id"]
|
||||
assert channels.get(conn, 999) is None
|
||||
assert channels.get_by_channel_id(conn, "UCnope") is None
|
||||
|
||||
|
||||
def test_all_channels_is_sorted_case_insensitively(conn):
|
||||
add_channel(conn, "UC" + "a" * 22, "zebra", "zebra")
|
||||
add_channel(conn, "UC" + "b" * 22, "Apple", "Apple")
|
||||
|
||||
titles = [row["title"] for row in channels.all_channels(conn)]
|
||||
|
||||
assert titles == ["Apple", "zebra"]
|
||||
@@ -0,0 +1,391 @@
|
||||
"""Discovery: feed parsing, the API backfill, and duration enrichment."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import date, timedelta
|
||||
|
||||
import pytest
|
||||
|
||||
from ytstream import api, discovery, util, videos
|
||||
|
||||
from conftest import FakeApi, add_video, make_feed, patch_api
|
||||
|
||||
|
||||
def _entry(video_id, published, title="A video"):
|
||||
return {"video_id": video_id, "published": published, "title": title}
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def offline(monkeypatch):
|
||||
"""No feed fetches and no API calls unless a test asks for them."""
|
||||
monkeypatch.setattr(discovery, "fetch_feed",
|
||||
lambda *a, **k: pytest.fail("unexpected feed fetch"))
|
||||
patch_api(monkeypatch, discovery, FakeApi())
|
||||
|
||||
|
||||
# --------------------------------------------------------------- feed parsing
|
||||
|
||||
|
||||
def test_parse_entries_reads_only_atom_entries():
|
||||
"""The feed-level <published> is the playlist's creation date, sometimes years
|
||||
old. Treating it as a video produced a nonsense 0.01/day upload rate once."""
|
||||
payload = make_feed(
|
||||
[_entry("vid00000001", "2026-08-11T10:00:00+00:00")],
|
||||
playlist_published="2019-03-01T00:00:00+00:00",
|
||||
)
|
||||
|
||||
entries = discovery.parse_entries(payload)
|
||||
|
||||
assert len(entries) == 1
|
||||
assert entries[0]["published"] == date(2026, 8, 11)
|
||||
|
||||
|
||||
def test_parse_entries_keeps_the_exact_timestamp():
|
||||
payload = make_feed([_entry("vid00000001", "2026-08-11T16:32:10+00:00")])
|
||||
assert discovery.parse_entries(payload)[0]["published_at"].startswith(
|
||||
"2026-08-11T16:32:10"
|
||||
)
|
||||
|
||||
|
||||
def test_parse_entries_skips_undated_entries():
|
||||
payload = make_feed([_entry("vid00000001", "not-a-date"),
|
||||
_entry("vid00000002", "2026-08-11T10:00:00+00:00")])
|
||||
assert [e["video_id"] for e in discovery.parse_entries(payload)] == ["vid00000002"]
|
||||
|
||||
|
||||
def test_parse_entries_rejects_garbage():
|
||||
with pytest.raises(discovery.FeedUnavailable):
|
||||
discovery.parse_entries(b"<not xml")
|
||||
|
||||
|
||||
def test_empty_feed_is_not_an_error():
|
||||
assert discovery.parse_entries(make_feed([])) == []
|
||||
|
||||
|
||||
def test_feed_urls_use_the_long_form_playlist():
|
||||
url = discovery.uulf_feed_url("UCjCJ2LaOIsPzOoXUTMDI3wg")
|
||||
assert "playlist_id=UULFjCJ2LaOIsPzOoXUTMDI3wg" in url
|
||||
assert "channel_id=UCjCJ2LaOIsPzOoXUTMDI3wg" in discovery.uc_feed_url(
|
||||
"UCjCJ2LaOIsPzOoXUTMDI3wg"
|
||||
)
|
||||
|
||||
|
||||
# -------------------------------------------------------------------- polling
|
||||
|
||||
|
||||
def _install_feed(monkeypatch, entries, *, uulf=True):
|
||||
payload = make_feed(entries)
|
||||
|
||||
def fetch(url, timeout=30.0):
|
||||
if "playlist_id=UULF" in url:
|
||||
return payload if uulf else None
|
||||
return payload if not uulf else None
|
||||
|
||||
monkeypatch.setattr(discovery, "fetch_feed", fetch)
|
||||
|
||||
|
||||
def test_poll_queues_videos_inside_the_window(conn, settings, channel, monkeypatch):
|
||||
today = util.today()
|
||||
_install_feed(monkeypatch, [_entry("vid00000001", f"{today}T10:00:00+00:00")])
|
||||
patch_api(monkeypatch, discovery,
|
||||
FakeApi(durations={"vid00000001": {"duration": 900, "is_live": False}}))
|
||||
|
||||
stats = discovery.poll_channel(conn, settings, channel)
|
||||
|
||||
assert stats["queued"] == 1
|
||||
assert stats["source"] == videos.SOURCE_UULF
|
||||
row = videos.get(conn, "vid00000001")
|
||||
assert row["state"] == videos.LISTED
|
||||
assert row["duration"] == 900
|
||||
|
||||
|
||||
def test_poll_marks_older_videos_skipped_old(conn, settings, channel, monkeypatch):
|
||||
old = util.today() - timedelta(days=90)
|
||||
_install_feed(monkeypatch, [_entry("vid00000001", f"{old}T10:00:00+00:00")])
|
||||
patch_api(monkeypatch, discovery, FakeApi())
|
||||
|
||||
stats = discovery.poll_channel(conn, settings, channel)
|
||||
|
||||
assert stats["old"] == 1
|
||||
assert videos.get(conn, "vid00000001")["state"] == videos.SKIPPED_OLD
|
||||
|
||||
|
||||
def test_poll_falls_back_to_the_channel_feed(conn, settings, channel, monkeypatch):
|
||||
today = util.today()
|
||||
_install_feed(monkeypatch, [_entry("vid00000001", f"{today}T10:00:00+00:00")],
|
||||
uulf=False)
|
||||
patch_api(monkeypatch, discovery, FakeApi())
|
||||
|
||||
stats = discovery.poll_channel(conn, settings, channel)
|
||||
|
||||
assert stats["source"] == videos.SOURCE_UC
|
||||
assert videos.get(conn, "vid00000001")["discovery_source"] == videos.SOURCE_UC
|
||||
|
||||
|
||||
def test_poll_records_feed_failure_without_raising(conn, settings, channel, monkeypatch):
|
||||
def boom(url, timeout=30.0):
|
||||
raise discovery.FeedUnavailable("HTTP 404")
|
||||
|
||||
monkeypatch.setattr(discovery, "fetch_feed", boom)
|
||||
|
||||
stats = discovery.poll_channel(conn, settings, channel)
|
||||
|
||||
assert "error" in stats
|
||||
row = conn.execute("SELECT * FROM channel WHERE id = ?", (channel["id"],)).fetchone()
|
||||
assert row["last_poll_ok"] == 0
|
||||
assert row["consecutive_poll_failures"] == 1
|
||||
|
||||
|
||||
def test_two_of_119_channels_failing_is_survivable(conn, settings, channel, monkeypatch):
|
||||
"""Measured: terminated channels stay in the subscription list and 404 here."""
|
||||
monkeypatch.setattr(discovery, "fetch_feed",
|
||||
lambda *a, **k: (_ for _ in ()).throw(
|
||||
discovery.FeedUnavailable("HTTP 404")))
|
||||
totals = discovery.poll_all(conn, settings)
|
||||
assert totals["failed"] == 1
|
||||
assert totals["channels"] == 1
|
||||
|
||||
|
||||
def test_poll_fills_in_a_missing_title(conn, settings, channel, monkeypatch):
|
||||
"""Backfill inserts rows with no title; the feed is where titles come from."""
|
||||
today = util.today()
|
||||
add_video(conn, channel["id"], "vid00000001", title="",
|
||||
upload_date=today.isoformat())
|
||||
_install_feed(monkeypatch,
|
||||
[_entry("vid00000001", f"{today}T10:00:00+00:00", "Real Title")])
|
||||
patch_api(monkeypatch, discovery, FakeApi())
|
||||
|
||||
stats = discovery.poll_channel(conn, settings, channel)
|
||||
|
||||
assert stats["titled"] == 1
|
||||
assert videos.get(conn, "vid00000001")["title"] == "Real Title"
|
||||
|
||||
|
||||
def test_uulf_repairs_a_video_the_fallback_called_short(
|
||||
conn, settings, channel, monkeypatch
|
||||
):
|
||||
today = util.today()
|
||||
add_video(conn, channel["id"], "vid00000001", state=videos.SKIPPED_SHORT,
|
||||
discovery_source=videos.SOURCE_UC, upload_date=today.isoformat())
|
||||
_install_feed(monkeypatch, [_entry("vid00000001", f"{today}T10:00:00+00:00")])
|
||||
patch_api(monkeypatch, discovery, FakeApi())
|
||||
|
||||
stats = discovery.poll_channel(conn, settings, channel)
|
||||
|
||||
assert stats["repaired"] == 1
|
||||
row = videos.get(conn, "vid00000001")
|
||||
assert row["state"] == videos.LISTED
|
||||
assert row["discovery_source"] == videos.SOURCE_UULF
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- enrichment
|
||||
|
||||
|
||||
def test_enrich_filters_shorts(conn, settings, channel, monkeypatch):
|
||||
"""Measured: 38 of 50 consecutive uploads on a real channel were <=120s."""
|
||||
add_video(conn, channel["id"], "short000001", duration=None)
|
||||
patch_api(monkeypatch, discovery,
|
||||
FakeApi(durations={"short000001": {"duration": 45, "is_live": False}}))
|
||||
|
||||
stats = discovery.enrich_durations(conn, settings, ["short000001"])
|
||||
|
||||
assert stats["shorts"] == 1
|
||||
assert videos.get(conn, "short000001")["state"] == videos.SKIPPED_SHORT
|
||||
|
||||
|
||||
def test_enrich_filters_livestreams_regardless_of_duration(
|
||||
conn, settings, channel, monkeypatch
|
||||
):
|
||||
"""Live and upcoming both report PT0S, so duration cannot be the signal."""
|
||||
add_video(conn, channel["id"], "live0000001", duration=None)
|
||||
patch_api(monkeypatch, discovery,
|
||||
FakeApi(durations={"live0000001": {"duration": 0, "is_live": True}}))
|
||||
|
||||
stats = discovery.enrich_durations(conn, settings, ["live0000001"])
|
||||
|
||||
assert stats["live"] == 1
|
||||
assert videos.get(conn, "live0000001")["state"] == videos.SKIPPED_LIVE
|
||||
|
||||
|
||||
def test_enrich_keeps_long_videos(conn, settings, channel, monkeypatch):
|
||||
add_video(conn, channel["id"], "long0000001", duration=None)
|
||||
patch_api(monkeypatch, discovery,
|
||||
FakeApi(durations={"long0000001": {"duration": 2790, "is_live": False}}))
|
||||
|
||||
discovery.enrich_durations(conn, settings, ["long0000001"])
|
||||
|
||||
row = videos.get(conn, "long0000001")
|
||||
assert row["state"] == videos.LISTED
|
||||
assert row["duration"] == 2790
|
||||
|
||||
|
||||
def test_enrich_survives_an_api_failure(conn, settings, channel, monkeypatch):
|
||||
"""A NULL duration costs a runtime display, not a working library."""
|
||||
add_video(conn, channel["id"], "vid00000001", duration=None)
|
||||
|
||||
class Failing(FakeApi):
|
||||
def durations(self, ids):
|
||||
raise api.ApiError(500, "backendError", "boom")
|
||||
|
||||
patch_api(monkeypatch, discovery, Failing())
|
||||
|
||||
stats = discovery.enrich_durations(conn, settings, ["vid00000001"])
|
||||
|
||||
assert stats["resolved"] == 0
|
||||
assert videos.get(conn, "vid00000001")["state"] == videos.LISTED
|
||||
|
||||
|
||||
def test_enrich_with_no_ids_makes_no_call(conn, settings, monkeypatch):
|
||||
fake = patch_api(monkeypatch, discovery, FakeApi())
|
||||
discovery.enrich_durations(conn, settings, [])
|
||||
assert fake.calls == 0
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- backfill
|
||||
|
||||
|
||||
def test_backfill_queues_the_window(conn, settings, channel, monkeypatch):
|
||||
today = util.today()
|
||||
uploads = [
|
||||
({"video_id": f"vid{i:08d}", "published": today - timedelta(days=i),
|
||||
"published_at": f"{today - timedelta(days=i)}T00:00:00Z"}, None)
|
||||
for i in range(3)
|
||||
]
|
||||
patch_api(monkeypatch, discovery, FakeApi(
|
||||
uploads=uploads,
|
||||
durations={f"vid{i:08d}": {"duration": 900, "is_live": False}
|
||||
for i in range(3)}))
|
||||
|
||||
stats = discovery.backfill_channel(conn, settings, channel)
|
||||
|
||||
assert stats["queued"] == 3
|
||||
assert videos.get(conn, "vid00000000")["state"] == videos.LISTED
|
||||
assert conn.execute("SELECT backfilled FROM channel WHERE id = ?",
|
||||
(channel["id"],)).fetchone()[0] == 1
|
||||
|
||||
|
||||
def test_backfill_stores_the_exact_publish_time(conn, settings, channel, monkeypatch):
|
||||
today = util.today()
|
||||
patch_api(monkeypatch, discovery, FakeApi(uploads=[
|
||||
({"video_id": "vid00000001", "published": today,
|
||||
"published_at": "2026-08-11T16:32:10Z"}, None)]))
|
||||
|
||||
discovery.backfill_channel(conn, settings, channel)
|
||||
|
||||
assert videos.get(conn, "vid00000001")["published_at"] == "2026-08-11T16:32:10Z"
|
||||
|
||||
|
||||
def test_backfill_respects_the_video_cap(conn, settings, channel, monkeypatch):
|
||||
settings.set("backfill_max_videos", "2")
|
||||
today = util.today()
|
||||
patch_api(monkeypatch, discovery, FakeApi(uploads=[
|
||||
({"video_id": f"vid{i:08d}", "published": today, "published_at": None}, None)
|
||||
for i in range(10)]))
|
||||
|
||||
stats = discovery.backfill_channel(conn, settings, channel)
|
||||
|
||||
assert stats["queued"] == 2
|
||||
|
||||
|
||||
def test_backfill_clears_the_cursor_when_complete(conn, settings, channel, monkeypatch):
|
||||
patch_api(monkeypatch, discovery, FakeApi(uploads=[
|
||||
({"video_id": "vid00000001", "published": util.today(),
|
||||
"published_at": None}, "TOKEN")]))
|
||||
|
||||
discovery.backfill_channel(conn, settings, channel)
|
||||
|
||||
assert conn.execute("SELECT backfill_cursor FROM channel WHERE id = ?",
|
||||
(channel["id"],)).fetchone()[0] is None
|
||||
|
||||
|
||||
def test_backfill_leaves_the_flag_unset_when_the_api_is_unusable(
|
||||
conn, settings, channel, monkeypatch
|
||||
):
|
||||
"""So it retries once a key is configured, rather than silently never running."""
|
||||
with conn:
|
||||
conn.execute("UPDATE channel SET backfilled = 0 WHERE id = ?", (channel["id"],))
|
||||
|
||||
class Failing(FakeApi):
|
||||
def uploads(self, *a, **kw):
|
||||
raise api.NotConfigured(403, "forbidden", "blocked")
|
||||
yield # pragma: no cover
|
||||
|
||||
patch_api(monkeypatch, discovery, Failing())
|
||||
|
||||
stats = discovery.backfill_channel(conn, settings, channel)
|
||||
|
||||
assert "error" in stats
|
||||
assert conn.execute("SELECT backfilled FROM channel WHERE id = ?",
|
||||
(channel["id"],)).fetchone()[0] == 0
|
||||
|
||||
|
||||
def test_backfill_does_not_duplicate_known_videos(conn, settings, channel, monkeypatch):
|
||||
add_video(conn, channel["id"], "vid00000001")
|
||||
patch_api(monkeypatch, discovery, FakeApi(uploads=[
|
||||
({"video_id": "vid00000001", "published": util.today(),
|
||||
"published_at": None}, None)]))
|
||||
|
||||
stats = discovery.backfill_channel(conn, settings, channel)
|
||||
|
||||
assert stats["queued"] == 0
|
||||
assert conn.execute("SELECT COUNT(*) FROM video").fetchone()[0] == 1
|
||||
|
||||
|
||||
# ------------------------------------------------------- titles (regression)
|
||||
|
||||
|
||||
def test_backfill_takes_the_title_from_the_api(conn, settings, channel, monkeypatch):
|
||||
"""Not an optimisation. RSS returns 15 entries, which for a channel posting
|
||||
under one long-form video a day reaches back only ~23 days against a 30-day
|
||||
window — so the oldest ~5 of every 20-episode backfill was being named after
|
||||
its video id."""
|
||||
patch_api(monkeypatch, discovery, FakeApi(uploads=[
|
||||
({"video_id": "vid00000001", "published": util.today(),
|
||||
"published_at": None, "title": "A Real Title"}, None)]))
|
||||
|
||||
discovery.backfill_channel(conn, settings, channel)
|
||||
|
||||
assert videos.get(conn, "vid00000001")["title"] == "A Real Title"
|
||||
|
||||
|
||||
def test_backfill_tolerates_a_missing_title(conn, settings, channel, monkeypatch):
|
||||
patch_api(monkeypatch, discovery, FakeApi(uploads=[
|
||||
({"video_id": "vid00000001", "published": util.today(),
|
||||
"published_at": None}, None)]))
|
||||
|
||||
discovery.backfill_channel(conn, settings, channel)
|
||||
|
||||
assert videos.get(conn, "vid00000001")["title"] == ""
|
||||
|
||||
|
||||
def test_a_late_title_renames_an_already_materialised_episode(
|
||||
conn, settings, media_root, channel, monkeypatch
|
||||
):
|
||||
"""The repair has to move the file, not just the row — otherwise the episode
|
||||
keeps its video-id filename forever."""
|
||||
from ytstream import strm
|
||||
|
||||
monkeypatch.setattr(strm, "fetch_thumbnail", lambda *a, **k: False)
|
||||
today = util.today()
|
||||
add_video(conn, channel["id"], "vid00000001", title="",
|
||||
upload_date=today.isoformat())
|
||||
first = strm.materialise(conn, settings, channel,
|
||||
videos.get(conn, "vid00000001"))
|
||||
assert "vid00000001]" in first["rel_path"]
|
||||
old_path = media_root / first["rel_path"]
|
||||
assert old_path.exists()
|
||||
|
||||
outcome = discovery._record(
|
||||
conn, channel,
|
||||
{"video_id": "vid00000001", "title": "Proper Name", "published": today,
|
||||
"published_at": None},
|
||||
videos.SOURCE_UULF, today - timedelta(days=30),
|
||||
)
|
||||
|
||||
assert outcome == "titled"
|
||||
assert videos.get(conn, "vid00000001")["state"] == videos.LISTED
|
||||
assert not old_path.exists()
|
||||
|
||||
second = strm.materialise(conn, settings, channel,
|
||||
videos.get(conn, "vid00000001"))
|
||||
assert "Proper Name" in second["rel_path"]
|
||||
@@ -0,0 +1,128 @@
|
||||
from datetime import date
|
||||
|
||||
import pytest
|
||||
|
||||
from ytstream import naming
|
||||
|
||||
|
||||
class TestSanitise:
|
||||
@pytest.mark.parametrize(
|
||||
"raw, expected",
|
||||
[
|
||||
("Hermitcraft S11#11: Expanding Business", "Hermitcraft S11#11 Expanding Business"),
|
||||
("A/B", "A B"),
|
||||
('Say "hello" <now>', "Say hello now"),
|
||||
("path\\to\\thing", "path to thing"),
|
||||
("what? really * | yes", "what really yes"),
|
||||
(" .leading and trailing. ", "leading and trailing"),
|
||||
("line one\nline two", "line one line two"),
|
||||
("tabs\tand\r\nnewlines", "tabs and newlines"),
|
||||
],
|
||||
)
|
||||
def test_removes_illegal_characters(self, raw, expected):
|
||||
assert naming.sanitize_component(raw) == expected
|
||||
|
||||
def test_empty_input_is_empty(self):
|
||||
assert naming.sanitize_component("") == ""
|
||||
assert naming.sanitize_component(None) == ""
|
||||
|
||||
def test_title_that_is_only_illegal_characters_collapses_to_empty(self):
|
||||
assert naming.sanitize_component("///???") == ""
|
||||
|
||||
def test_truncates_to_120_characters(self):
|
||||
long = "word " * 60
|
||||
result = naming.sanitize_component(long)
|
||||
assert len(result) <= naming.MAX_TITLE_LEN
|
||||
|
||||
def test_truncation_prefers_a_word_boundary(self):
|
||||
text = "alpha bravo charlie delta echo foxtrot golf hotel india juliet " * 3
|
||||
result = naming.sanitize_component(text)
|
||||
assert not result.endswith(" ")
|
||||
# Should not cut mid-word when a boundary is available late enough.
|
||||
assert result == result.rstrip()
|
||||
assert " " in result
|
||||
|
||||
def test_hard_cuts_when_no_late_word_boundary_exists(self):
|
||||
text = "a" + "b" * 400
|
||||
result = naming.sanitize_component(text)
|
||||
assert len(result) == naming.MAX_TITLE_LEN
|
||||
|
||||
|
||||
class TestEpisodeNumbering:
|
||||
@pytest.mark.parametrize(
|
||||
"day, ordinal, expected",
|
||||
[
|
||||
(date(2026, 8, 11), 0, 8110),
|
||||
(date(2026, 8, 11), 1, 8111),
|
||||
(date(2026, 8, 12), 0, 8120),
|
||||
(date(2026, 1, 1), 0, 1010),
|
||||
(date(2026, 12, 31), 0, 12310),
|
||||
(date(2026, 12, 31), 9, 12319),
|
||||
],
|
||||
)
|
||||
def test_episode_number(self, day, ordinal, expected):
|
||||
assert naming.episode_number(day, ordinal) == expected
|
||||
|
||||
def test_ordinal_is_clamped_at_nine(self):
|
||||
assert naming.episode_number(date(2026, 8, 11), 12) == 8119
|
||||
|
||||
def test_negative_ordinal_clamps_to_zero(self):
|
||||
assert naming.episode_number(date(2026, 8, 11), -3) == 8110
|
||||
|
||||
def test_numbers_sort_chronologically_across_the_year(self):
|
||||
days = [date(2026, 1, 1), date(2026, 6, 15), date(2026, 8, 11), date(2026, 12, 31)]
|
||||
numbers = [naming.episode_number(day, 0) for day in days]
|
||||
assert numbers == sorted(numbers)
|
||||
|
||||
def test_episode_range_covers_ten_slots(self):
|
||||
low, high = naming.episode_range(date(2026, 8, 11))
|
||||
assert (low, high) == (8110, 8119)
|
||||
|
||||
def test_season_is_the_upload_year(self):
|
||||
assert naming.season_for(date(2026, 8, 11)) == 2026
|
||||
|
||||
|
||||
class TestParseUploadDate:
|
||||
def test_accepts_ytdlp_compact_form(self):
|
||||
assert naming.parse_upload_date("20260811") == date(2026, 8, 11)
|
||||
|
||||
def test_accepts_iso_form(self):
|
||||
assert naming.parse_upload_date("2026-08-11") == date(2026, 8, 11)
|
||||
|
||||
def test_accepts_iso_timestamp(self):
|
||||
assert naming.parse_upload_date("2026-08-11T12:00:00+00:00") == date(2026, 8, 11)
|
||||
|
||||
def test_passes_through_a_date(self):
|
||||
assert naming.parse_upload_date(date(2026, 8, 11)) == date(2026, 8, 11)
|
||||
|
||||
def test_rejects_nonsense(self):
|
||||
with pytest.raises(ValueError):
|
||||
naming.parse_upload_date("not a date")
|
||||
|
||||
|
||||
class TestBasename:
|
||||
def test_includes_video_id_for_uniqueness(self):
|
||||
stem = naming.basename("clabretro", 2026, 8110, "A Title", "dQw4w9WgXcQ")
|
||||
assert stem == "clabretro - S2026E8110 - A Title [dQw4w9WgXcQ]"
|
||||
|
||||
def test_episode_is_unpadded_to_match_the_nfo(self):
|
||||
stem = naming.basename("c", 2026, 8110, "t", "id")
|
||||
assert "S2026E8110" in stem
|
||||
assert "E08110" not in stem
|
||||
|
||||
def test_falls_back_to_video_id_when_the_title_sanitises_away(self):
|
||||
stem = naming.basename("c", 2026, 8110, "///", "dQw4w9WgXcQ")
|
||||
assert stem.endswith("dQw4w9WgXcQ [dQw4w9WgXcQ]")
|
||||
|
||||
def test_two_videos_same_title_differ_by_id(self):
|
||||
one = naming.basename("c", 2026, 8110, "Same", "aaaaaaaaaaa")
|
||||
two = naming.basename("c", 2026, 8111, "Same", "bbbbbbbbbbb")
|
||||
assert one != two
|
||||
|
||||
|
||||
class TestChannelDirName:
|
||||
def test_sanitises_the_title(self):
|
||||
assert naming.channel_dir_name("Tom / Jerry", "UCabc") == "Tom Jerry"
|
||||
|
||||
def test_falls_back_to_channel_id_when_title_is_unusable(self):
|
||||
assert naming.channel_dir_name("???", "UCabc") == "UCabc"
|
||||
@@ -0,0 +1,94 @@
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
from ytstream import nfo
|
||||
|
||||
HOSTILE = (
|
||||
"Ampersands & angle <brackets> and \"quotes\"\n"
|
||||
"control chars: \x00\x07\x1b\n"
|
||||
"emoji 🎬 and em-dash — and links https://example.com/?a=1&b=2"
|
||||
)
|
||||
|
||||
|
||||
class TestEpisodeNfo:
|
||||
def build(self, **overrides):
|
||||
kwargs = dict(
|
||||
title="Video Title",
|
||||
show_title="Some Channel",
|
||||
season=2026,
|
||||
episode=8110,
|
||||
plot="A plot.",
|
||||
aired="2026-08-11",
|
||||
duration_seconds=762,
|
||||
video_id="dQw4w9WgXcQ",
|
||||
)
|
||||
kwargs.update(overrides)
|
||||
return nfo.episode_nfo(**kwargs)
|
||||
|
||||
def test_is_well_formed_xml(self):
|
||||
root = ET.fromstring(self.build())
|
||||
assert root.tag == "episodedetails"
|
||||
|
||||
def test_hostile_description_still_parses(self):
|
||||
root = ET.fromstring(self.build(plot=HOSTILE))
|
||||
plot = root.findtext("plot")
|
||||
assert "&" in plot and "<brackets>" in plot
|
||||
assert "🎬" in plot
|
||||
|
||||
def test_control_characters_are_stripped(self):
|
||||
plot = ET.fromstring(self.build(plot=HOSTILE)).findtext("plot")
|
||||
for bad in ("\x00", "\x07", "\x1b"):
|
||||
assert bad not in plot
|
||||
|
||||
def test_newlines_are_preserved(self):
|
||||
plot = ET.fromstring(self.build(plot="one\ntwo")).findtext("plot")
|
||||
assert plot == "one\ntwo"
|
||||
|
||||
def test_runtime_is_rounded_minutes(self):
|
||||
assert ET.fromstring(self.build(duration_seconds=762)).findtext("runtime") == "13"
|
||||
|
||||
def test_short_video_still_gets_at_least_one_minute(self):
|
||||
assert ET.fromstring(self.build(duration_seconds=20)).findtext("runtime") == "1"
|
||||
|
||||
def test_runtime_omitted_when_duration_unknown(self):
|
||||
assert ET.fromstring(self.build(duration_seconds=None)).find("runtime") is None
|
||||
|
||||
def test_unique_id_marks_youtube_as_default(self):
|
||||
unique = ET.fromstring(self.build()).find("uniqueid")
|
||||
assert unique.get("type") == "youtube"
|
||||
assert unique.get("default") == "true"
|
||||
assert unique.text == "dQw4w9WgXcQ"
|
||||
|
||||
def test_season_and_episode_are_present(self):
|
||||
root = ET.fromstring(self.build())
|
||||
assert root.findtext("season") == "2026"
|
||||
assert root.findtext("episode") == "8110"
|
||||
|
||||
def test_title_keeps_characters_that_the_filename_strips(self):
|
||||
root = ET.fromstring(self.build(title="Hermitcraft S11#11: Expanding Business"))
|
||||
assert root.findtext("title") == "Hermitcraft S11#11: Expanding Business"
|
||||
|
||||
def test_empty_plot_does_not_break(self):
|
||||
assert ET.fromstring(self.build(plot=None)).find("plot") is not None
|
||||
|
||||
|
||||
class TestTvshowNfo:
|
||||
def test_well_formed_and_carries_channel_id(self):
|
||||
root = ET.fromstring(nfo.tvshow_nfo("clabretro", HOSTILE, "UCabc123"))
|
||||
assert root.tag == "tvshow"
|
||||
assert root.findtext("title") == "clabretro"
|
||||
assert root.findtext("studio") == "YouTube"
|
||||
assert root.find("uniqueid").text == "UCabc123"
|
||||
|
||||
|
||||
class TestWrite:
|
||||
def test_write_is_atomic_and_leaves_no_temp_file(self, tmp_path):
|
||||
target = tmp_path / "sub" / "tvshow.nfo"
|
||||
nfo.write(target, b"<tvshow/>")
|
||||
assert target.read_bytes() == b"<tvshow/>"
|
||||
assert list(tmp_path.rglob("*.tmp")) == []
|
||||
|
||||
def test_overwrites_existing(self, tmp_path):
|
||||
target = tmp_path / "tvshow.nfo"
|
||||
nfo.write(target, b"<a/>")
|
||||
nfo.write(target, b"<b/>")
|
||||
assert target.read_bytes() == b"<b/>"
|
||||
@@ -0,0 +1,354 @@
|
||||
"""The streaming proxy: HTTP range serving, and the safety rails.
|
||||
|
||||
Ported from the standalone `proxy/test_range.py` and `proxy/test_limits.py` so that
|
||||
one `pytest` run covers everything. These now drive the real `make_handler(mgr, …)`
|
||||
rather than the PoC's `make_handler(path, done)`, which means the routing, the
|
||||
video-id validation and the wait-for-complete logic are exercised too.
|
||||
|
||||
No network, no yt-dlp, no ffmpeg: the pipeline runner is stubbed and the output
|
||||
file is synthetic.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import http.client
|
||||
import importlib.util
|
||||
import json
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
from http.server import ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
PROXY_PATH = Path(__file__).resolve().parent.parent / "proxy" / "ytstream_proxy.py"
|
||||
|
||||
|
||||
def _load_proxy():
|
||||
spec = importlib.util.spec_from_file_location("ytstream_proxy", PROXY_PATH)
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
return module
|
||||
|
||||
|
||||
proxy = _load_proxy()
|
||||
|
||||
# Position-identifiable body, so a wrong range returns visibly wrong bytes rather
|
||||
# than merely the wrong length.
|
||||
BODY = bytes(range(256)) * 400
|
||||
TOTAL = len(BODY)
|
||||
VIDEO_ID = "dQw4w9WgXcQ"
|
||||
|
||||
|
||||
class StubSession:
|
||||
"""A Session whose output file already exists."""
|
||||
|
||||
def __init__(self, path: Path, *, finished: bool = True):
|
||||
self.video_id = VIDEO_ID
|
||||
self.out_path = str(path)
|
||||
self.final = threading.Event()
|
||||
self.failed = None
|
||||
self.readers = 0
|
||||
self.last_used = time.time()
|
||||
if finished:
|
||||
self.final.set()
|
||||
|
||||
@property
|
||||
def complete(self):
|
||||
return self.final.is_set() and not self.failed
|
||||
|
||||
def size(self):
|
||||
try:
|
||||
return os.path.getsize(self.out_path)
|
||||
except OSError:
|
||||
return 0
|
||||
|
||||
|
||||
class StubManager:
|
||||
def __init__(self, session, *, growing=False):
|
||||
self.session = session
|
||||
self.growing = growing
|
||||
self.lock = threading.Lock()
|
||||
|
||||
def get(self, video_id):
|
||||
return self.session, None
|
||||
|
||||
def status(self):
|
||||
return {"mode": "stub", "cache_used_gb": 0.0}
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def complete_server(tmp_path):
|
||||
"""A server over a finished file, with ranges honoured."""
|
||||
path = tmp_path / "out.mp4"
|
||||
path.write_bytes(BODY)
|
||||
session = StubSession(path, finished=True)
|
||||
yield from _serve(StubManager(session))
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def growing_server(tmp_path):
|
||||
"""A server over a file still being written."""
|
||||
path = tmp_path / "out.mp4"
|
||||
path.write_bytes(BODY[:1000])
|
||||
session = StubSession(path, finished=False)
|
||||
manager = StubManager(session, growing=True)
|
||||
|
||||
def finish():
|
||||
time.sleep(0.2)
|
||||
with open(path, "ab") as handle:
|
||||
handle.write(BODY[1000:])
|
||||
session.final.set()
|
||||
|
||||
threading.Thread(target=finish, daemon=True).start()
|
||||
yield from _serve(manager)
|
||||
|
||||
|
||||
def _serve(manager):
|
||||
server = ThreadingHTTPServer(("127.0.0.1", 0), proxy.make_handler(manager, 30))
|
||||
threading.Thread(target=server.serve_forever, daemon=True).start()
|
||||
try:
|
||||
yield server.server_address[1]
|
||||
finally:
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
|
||||
|
||||
def request(port, headers=None, method="GET", path=f"/watch/{VIDEO_ID}"):
|
||||
conn = http.client.HTTPConnection("127.0.0.1", port, timeout=20)
|
||||
conn.request(method, path, headers=headers or {})
|
||||
response = conn.getresponse()
|
||||
body = response.read()
|
||||
head = dict(response.getheaders())
|
||||
conn.close()
|
||||
return response.status, head, body
|
||||
|
||||
|
||||
# ------------------------------------------------------------- complete file
|
||||
|
||||
|
||||
def test_bounded_range(complete_server):
|
||||
status, head, body = request(complete_server, {"Range": "bytes=1000-1999"})
|
||||
assert status == 206
|
||||
assert len(body) == 1000
|
||||
assert body == BODY[1000:2000]
|
||||
assert head["Content-Range"] == f"bytes 1000-1999/{TOTAL}"
|
||||
assert head["Content-Length"] == "1000"
|
||||
|
||||
|
||||
def test_open_ended_range(complete_server):
|
||||
status, head, body = request(complete_server, {"Range": "bytes=102000-"})
|
||||
assert status == 206
|
||||
assert body == BODY[102000:]
|
||||
assert head["Content-Range"] == f"bytes 102000-{TOTAL - 1}/{TOTAL}"
|
||||
|
||||
|
||||
def test_suffix_range(complete_server):
|
||||
"""bytes=-N means the final N bytes, not the first N."""
|
||||
status, head, body = request(complete_server, {"Range": "bytes=-100"})
|
||||
assert status == 206
|
||||
assert len(body) == 100
|
||||
assert body == BODY[-100:]
|
||||
assert head["Content-Range"] == f"bytes {TOTAL - 100}-{TOTAL - 1}/{TOTAL}"
|
||||
|
||||
|
||||
def test_single_byte_range(complete_server):
|
||||
status, _, body = request(complete_server, {"Range": "bytes=5-5"})
|
||||
assert status == 206
|
||||
assert body == BODY[5:6]
|
||||
|
||||
|
||||
def test_end_beyond_eof_is_clamped(complete_server):
|
||||
status, head, body = request(complete_server,
|
||||
{"Range": f"bytes=102000-{TOTAL + 5000}"})
|
||||
assert status == 206
|
||||
assert body == BODY[102000:]
|
||||
assert head["Content-Range"] == f"bytes 102000-{TOTAL - 1}/{TOTAL}"
|
||||
|
||||
|
||||
def test_range_past_eof_is_416(complete_server):
|
||||
"""And it must carry Content-Range: bytes */total, or clients retry forever."""
|
||||
status, head, _ = request(complete_server, {"Range": f"bytes={TOTAL + 10}-"})
|
||||
assert status == 416
|
||||
assert head["Content-Range"] == f"bytes */{TOTAL}"
|
||||
|
||||
|
||||
def test_no_range_serves_the_whole_body(complete_server):
|
||||
status, head, body = request(complete_server)
|
||||
assert status == 200
|
||||
assert body == BODY
|
||||
assert head["Accept-Ranges"] == "bytes"
|
||||
|
||||
|
||||
def test_multi_range_falls_back_to_the_whole_body(complete_server):
|
||||
"""Legal, and beats mis-serving one part of a multipart response."""
|
||||
status, _, body = request(complete_server, {"Range": "bytes=0-99,200-299"})
|
||||
assert status == 200
|
||||
assert body == BODY
|
||||
|
||||
|
||||
@pytest.mark.parametrize("header", ["bytes=", "bytes=abc-def", "items=0-99", "0-99"])
|
||||
def test_unparseable_range_serves_the_whole_body(complete_server, header):
|
||||
status, _, body = request(complete_server, {"Range": header})
|
||||
assert status == 200
|
||||
assert len(body) == TOTAL
|
||||
|
||||
|
||||
def test_head_returns_headers_and_no_body(complete_server):
|
||||
status, head, body = request(complete_server, method="HEAD")
|
||||
assert status == 200
|
||||
assert body == b""
|
||||
assert head["Content-Length"] == str(TOTAL)
|
||||
|
||||
|
||||
# ------------------------------------------------------------- growing file
|
||||
|
||||
|
||||
def test_growing_file_is_chunked_and_tracks_to_eof(growing_server):
|
||||
"""Ranges cannot be honoured mid-write: there is no reliable time-to-byte
|
||||
mapping into a fragmented MP4 yet, so it becomes a non-seekable stream."""
|
||||
status, head, body = request(growing_server, {"Range": "bytes=500-999"})
|
||||
assert status == 200
|
||||
assert head.get("Transfer-Encoding") == "chunked"
|
||||
assert body == BODY
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- routing
|
||||
|
||||
|
||||
def test_unknown_path_is_404(complete_server):
|
||||
status, _, _ = request(complete_server, path="/nope")
|
||||
assert status == 404
|
||||
|
||||
|
||||
def test_malformed_video_id_is_400(complete_server):
|
||||
status, _, _ = request(complete_server, path="/watch/short")
|
||||
assert status == 400
|
||||
|
||||
|
||||
@pytest.mark.parametrize("bad", ["../../etc/passwd", "abcdefghij", "abcdefghijkl"])
|
||||
def test_video_id_must_be_exactly_eleven_safe_chars(complete_server, bad):
|
||||
status, _, _ = request(complete_server, path=f"/watch/{bad}")
|
||||
assert status in (400, 404)
|
||||
|
||||
|
||||
def test_healthz_returns_json(complete_server):
|
||||
status, head, body = request(complete_server, path="/healthz")
|
||||
assert status == 200
|
||||
assert head["Content-Type"] == "application/json"
|
||||
assert json.loads(body)["mode"] == "stub"
|
||||
|
||||
|
||||
# ------------------------------------------------- safety rails on the manager
|
||||
|
||||
|
||||
def vid(n):
|
||||
return f"vid{n:08d}"
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def manager_factory(tmp_path):
|
||||
made = []
|
||||
|
||||
def build(**kwargs):
|
||||
options = dict(max_pipelines=99, cache_bytes=10 ** 12, no_fetch=False,
|
||||
growing=False, max_retries=0, max_starts=3,
|
||||
starts_window=3600.0)
|
||||
options.update(kwargs)
|
||||
work = tmp_path / f"work{len(made)}"
|
||||
work.mkdir()
|
||||
|
||||
class Recording(proxy.Manager):
|
||||
"""Records starts instead of spawning a pipeline."""
|
||||
|
||||
def __init__(self, *args, **kw):
|
||||
super().__init__(*args, **kw)
|
||||
self.ran = []
|
||||
|
||||
def _run(self, session):
|
||||
self.ran.append(session.video_id)
|
||||
session.final.set() # completes instantly, never stays active
|
||||
|
||||
manager = Recording(str(work), **options)
|
||||
made.append(manager)
|
||||
return manager
|
||||
|
||||
return build
|
||||
|
||||
|
||||
def test_cold_start_budget_is_enforced(manager_factory):
|
||||
manager = manager_factory(max_starts=3)
|
||||
results = [manager.get(vid(i)) for i in range(5)]
|
||||
|
||||
started = [r for r in results if r[0] is not None]
|
||||
refused = [r for r in results if r[0] is None]
|
||||
|
||||
assert len(started) == 3
|
||||
assert len(refused) == 2
|
||||
assert all("cold-start budget" in r[1] for r in refused)
|
||||
assert manager.counters["refused_ratelimit"] == 2
|
||||
assert len(manager.ran) == 3
|
||||
|
||||
|
||||
def test_cache_hits_are_never_rate_limited(manager_factory):
|
||||
"""Re-watching an already-fetched video must keep working with the budget
|
||||
spent, or a metadata refresh would break normal playback for an hour."""
|
||||
manager = manager_factory(max_starts=3)
|
||||
for i in range(5):
|
||||
manager.get(vid(i))
|
||||
|
||||
session, refusal = manager.get(vid(0))
|
||||
|
||||
assert session is not None and refusal is None
|
||||
assert manager.counters["reused"] >= 1
|
||||
assert len(manager.ran) == 3
|
||||
|
||||
|
||||
def test_start_budget_window_prunes(manager_factory):
|
||||
manager = manager_factory(max_starts=2)
|
||||
manager.get(vid(10))
|
||||
manager.get(vid(11))
|
||||
assert manager.get(vid(12))[0] is None
|
||||
|
||||
# Age the recorded starts past the window.
|
||||
manager.start_log = type(manager.start_log)(t - 3601 for t in manager.start_log)
|
||||
|
||||
assert manager.get(vid(13))[0] is not None
|
||||
assert len(manager.start_log) == 1
|
||||
|
||||
|
||||
def test_concurrency_cap(tmp_path):
|
||||
class Blocking(proxy.Manager):
|
||||
"""Pipelines that never finish, so sessions stay active."""
|
||||
|
||||
def _run(self, session):
|
||||
pass
|
||||
|
||||
work = tmp_path / "blocking"
|
||||
work.mkdir()
|
||||
manager = Blocking(str(work), max_pipelines=2, cache_bytes=10 ** 12,
|
||||
no_fetch=False, growing=False, max_retries=0,
|
||||
max_starts=99, starts_window=3600.0)
|
||||
|
||||
results = [manager.get(vid(20 + i)) for i in range(4)]
|
||||
ok = [r for r in results if r[0] is not None]
|
||||
refused = [r for r in results if r[0] is None]
|
||||
|
||||
assert len(ok) == 2
|
||||
assert len(refused) == 2
|
||||
assert all("pipeline cap" in r[1] for r in refused)
|
||||
# A busy refusal must not consume start budget, or a burst of concurrent
|
||||
# requests would exhaust the hourly allowance without fetching anything.
|
||||
assert manager.counters["refused_busy"] == 2
|
||||
assert len(manager.start_log) == 2
|
||||
|
||||
|
||||
def test_no_fetch_mode_refuses_everything(manager_factory):
|
||||
manager = manager_factory(no_fetch=True, max_starts=99)
|
||||
session, refusal = manager.get(vid(30))
|
||||
|
||||
assert session is None
|
||||
assert "no-fetch" in refusal
|
||||
assert manager.ran == []
|
||||
assert len(manager.start_log) == 0
|
||||
@@ -0,0 +1,201 @@
|
||||
"""Retention: the window, the min-keep floor, and the tombstones.
|
||||
|
||||
These are the tests that matter most in this suite. Retention is the only part of
|
||||
ytstream that deletes things, and two of its rules exist because of measured
|
||||
behaviour rather than taste:
|
||||
|
||||
* `min_keep_videos` exists because 52 of 117 real channels upload nothing in 30
|
||||
days and would otherwise be empty, flickering Jellyfin series.
|
||||
* the `aged_out` tombstone exists because without it the poller re-materialises
|
||||
everything the sweep just deleted, forever.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import timedelta
|
||||
|
||||
import pytest
|
||||
|
||||
from ytstream import discovery, reap, strm, util, videos
|
||||
|
||||
from conftest import add_channel, add_video
|
||||
|
||||
|
||||
def _old(days: int) -> str:
|
||||
return (util.today() - timedelta(days=days)).isoformat()
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def materialised(conn, settings, media_root, channel):
|
||||
"""Put videos on disk with a spread of ages. Returns their ids, newest first."""
|
||||
counter = {"n": 0}
|
||||
|
||||
def build(ages: list[int], chan=None):
|
||||
chan = chan or channel
|
||||
ids = []
|
||||
for age in ages:
|
||||
video_id = f"vid{counter['n']:08d}"
|
||||
counter["n"] += 1
|
||||
add_video(conn, chan["id"], video_id, upload_date=_old(age))
|
||||
strm.materialise(conn, settings, chan, videos.get(conn, video_id))
|
||||
ids.append(video_id)
|
||||
return ids
|
||||
|
||||
return build
|
||||
|
||||
|
||||
def test_video_inside_the_window_is_kept(conn, settings, materialised):
|
||||
materialised([1])
|
||||
assert reap.candidates(conn, settings) == []
|
||||
|
||||
|
||||
def test_video_outside_the_window_is_deleted(conn, settings, materialised, media_root):
|
||||
settings.set("min_keep_videos", "0")
|
||||
[video_id] = materialised([99])
|
||||
|
||||
path = media_root / videos.get(conn, video_id)["rel_path"]
|
||||
assert path.exists()
|
||||
|
||||
result = reap.run(conn, settings)
|
||||
|
||||
assert result["aged_out"] == 1
|
||||
assert not path.exists()
|
||||
assert videos.get(conn, video_id)["state"] == videos.AGED_OUT
|
||||
|
||||
|
||||
def test_deleting_removes_the_sidecars_too(conn, settings, materialised, media_root):
|
||||
settings.set("min_keep_videos", "0")
|
||||
[video_id] = materialised([99])
|
||||
strm_path = media_root / videos.get(conn, video_id)["rel_path"]
|
||||
stem = strm_path.name[: -len(".strm")]
|
||||
nfo_path = strm_path.with_name(stem + ".nfo")
|
||||
assert nfo_path.exists()
|
||||
|
||||
reap.run(conn, settings)
|
||||
|
||||
assert not nfo_path.exists()
|
||||
assert not strm_path.exists()
|
||||
|
||||
|
||||
def test_min_keep_videos_protects_the_newest_regardless_of_age(
|
||||
conn, settings, materialised
|
||||
):
|
||||
"""The measured common case: a channel whose only videos are all ancient."""
|
||||
settings.set("min_keep_videos", "5")
|
||||
materialised([400, 500, 600, 700, 800])
|
||||
|
||||
assert reap.candidates(conn, settings) == []
|
||||
assert reap.run(conn, settings)["aged_out"] == 0
|
||||
|
||||
|
||||
def test_min_keep_videos_releases_once_enough_newer_ones_exist(
|
||||
conn, settings, materialised
|
||||
):
|
||||
"""Six old videos with a floor of five: exactly the oldest one goes."""
|
||||
settings.set("min_keep_videos", "5")
|
||||
ids = materialised([100, 200, 300, 400, 500, 600])
|
||||
oldest = ids[-1]
|
||||
|
||||
assert reap.run(conn, settings)["aged_out"] == 1
|
||||
|
||||
assert videos.get(conn, oldest)["state"] == videos.AGED_OUT
|
||||
for kept in ids[:-1]:
|
||||
assert videos.get(conn, kept)["state"] == videos.MATERIALISED
|
||||
|
||||
|
||||
def test_min_keep_floor_counts_per_channel_not_globally(
|
||||
conn, settings, media_root, channel, materialised
|
||||
):
|
||||
other = add_channel(conn, "UCzzzzzzzzzzzzzzzzzzzzzz", "Other", "Other")
|
||||
settings.set("min_keep_videos", "2")
|
||||
materialised([300, 400], chan=channel)
|
||||
materialised([300, 400], chan=other)
|
||||
|
||||
# Two channels, two videos each, floor of two: nothing is eligible. If the
|
||||
# floor were global, two of the four would be deleted.
|
||||
assert reap.candidates(conn, settings) == []
|
||||
|
||||
|
||||
def test_channel_retention_override_beats_the_global_setting(
|
||||
conn, settings, media_root, channel, materialised
|
||||
):
|
||||
settings.set("min_keep_videos", "0")
|
||||
settings.set("retention_days", "365")
|
||||
materialised([90])
|
||||
assert reap.candidates(conn, settings) == []
|
||||
|
||||
with conn:
|
||||
conn.execute("UPDATE channel SET retention_days = 30 WHERE id = ?",
|
||||
(channel["id"],))
|
||||
assert len(reap.candidates(conn, settings)) == 1
|
||||
|
||||
|
||||
def test_aged_out_row_is_a_tombstone_the_poller_will_not_revive(
|
||||
conn, settings, media_root, channel, materialised
|
||||
):
|
||||
"""The failure this prevents: sweep deletes, poll re-adds, forever."""
|
||||
settings.set("min_keep_videos", "0")
|
||||
[video_id] = materialised([99])
|
||||
reap.run(conn, settings)
|
||||
assert videos.get(conn, video_id)["state"] == videos.AGED_OUT
|
||||
|
||||
# The video is still in the feed — YouTube has no idea we deleted it.
|
||||
entry = {"video_id": video_id, "title": "Video", "published": util.today(),
|
||||
"published_at": None}
|
||||
outcome = discovery._record(
|
||||
conn, channel, entry, videos.SOURCE_UULF, util.today() - timedelta(days=30)
|
||||
)
|
||||
|
||||
assert outcome == "known"
|
||||
assert videos.get(conn, video_id)["state"] == videos.AGED_OUT
|
||||
|
||||
|
||||
def test_rescan_revives_skipped_old_but_never_aged_out(
|
||||
conn, settings, media_root, channel, materialised
|
||||
):
|
||||
settings.set("min_keep_videos", "0")
|
||||
[gone] = materialised([99])
|
||||
reap.run(conn, settings)
|
||||
add_video(conn, channel["id"], "skippedold1", upload_date=_old(40),
|
||||
state=videos.SKIPPED_OLD)
|
||||
|
||||
settings.set("retention_days", "365")
|
||||
revived = discovery.rescan_channel(conn, settings, channel)
|
||||
|
||||
assert revived == 1
|
||||
assert videos.get(conn, "skippedold1")["state"] == videos.LISTED
|
||||
assert videos.get(conn, gone)["state"] == videos.AGED_OUT
|
||||
|
||||
|
||||
def test_empty_season_directory_is_pruned(conn, settings, materialised, media_root):
|
||||
settings.set("min_keep_videos", "0")
|
||||
[video_id] = materialised([99])
|
||||
season_dir = (media_root / videos.get(conn, video_id)["rel_path"]).parent
|
||||
assert season_dir.is_dir()
|
||||
|
||||
reap.run(conn, settings)
|
||||
|
||||
assert not season_dir.exists()
|
||||
# The channel directory survives: it still holds tvshow.nfo and artwork, and
|
||||
# an active subscription should not vanish from Jellyfin between uploads.
|
||||
assert season_dir.parent.is_dir()
|
||||
|
||||
|
||||
def test_prune_never_climbs_past_the_media_root(media_root):
|
||||
nested = media_root / "Chan" / "Season 2026"
|
||||
nested.mkdir(parents=True)
|
||||
removed = util.prune_empty_dirs(nested, media_root)
|
||||
assert removed == 2
|
||||
assert media_root.is_dir()
|
||||
|
||||
|
||||
def test_row_without_rel_path_still_gets_a_tombstone(conn, settings, channel):
|
||||
"""A video whose files vanished underneath us must not be retried forever."""
|
||||
settings.set("min_keep_videos", "0")
|
||||
add_video(conn, channel["id"], "orphan0001", upload_date=_old(99),
|
||||
state=videos.MATERIALISED)
|
||||
row = videos.get(conn, "orphan0001")
|
||||
assert row["rel_path"] is None
|
||||
|
||||
assert reap.delete_video(conn, row) is False
|
||||
assert videos.get(conn, "orphan0001")["state"] == videos.AGED_OUT
|
||||
@@ -0,0 +1,232 @@
|
||||
"""Orchestration: ordering, the lock, and what turns the cron check red."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from ytstream import config, discovery, jellyfin, reap, runner, subsync, videos
|
||||
|
||||
from conftest import add_video
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def quiet_jellyfin(monkeypatch):
|
||||
"""Jellyfin refresh is best effort; record calls instead of making them."""
|
||||
calls = []
|
||||
monkeypatch.setattr(jellyfin.Jellyfin, "refresh", lambda self: calls.append(1))
|
||||
return calls
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def no_thumbs(monkeypatch):
|
||||
from ytstream import strm
|
||||
|
||||
monkeypatch.setattr(strm, "fetch_thumbnail", lambda *a, **k: False)
|
||||
|
||||
|
||||
# ----------------------------------------------------------------------- lock
|
||||
|
||||
|
||||
def test_lock_is_exclusive(tmp_path):
|
||||
path = tmp_path / "run.lock"
|
||||
with runner.exclusive_lock(path):
|
||||
with pytest.raises(runner.AlreadyRunning):
|
||||
with runner.exclusive_lock(path):
|
||||
pass # pragma: no cover
|
||||
|
||||
|
||||
def test_lock_is_released_after_use(tmp_path):
|
||||
path = tmp_path / "run.lock"
|
||||
with runner.exclusive_lock(path):
|
||||
pass
|
||||
with runner.exclusive_lock(path):
|
||||
pass
|
||||
|
||||
|
||||
def test_lock_is_released_even_when_the_body_raises(tmp_path):
|
||||
path = tmp_path / "run.lock"
|
||||
with pytest.raises(ValueError):
|
||||
with runner.exclusive_lock(path):
|
||||
raise ValueError("boom")
|
||||
with runner.exclusive_lock(path):
|
||||
pass
|
||||
|
||||
|
||||
# --------------------------------------------------------------- materialising
|
||||
|
||||
|
||||
def test_materialise_all_writes_everything_listed(conn, settings, media_root,
|
||||
channel, no_thumbs):
|
||||
for index in range(3):
|
||||
add_video(conn, channel["id"], f"vid{index:08d}")
|
||||
|
||||
stats = runner.materialise_all(conn, settings)
|
||||
|
||||
assert stats["materialised"] == 3
|
||||
assert stats["errors"] == 0
|
||||
assert videos.queue_depth(conn) == 0
|
||||
|
||||
|
||||
def test_tvshow_nfo_is_written_once_per_channel(conn, settings, media_root,
|
||||
channel, no_thumbs):
|
||||
for index in range(3):
|
||||
add_video(conn, channel["id"], f"vid{index:08d}")
|
||||
|
||||
stats = runner.materialise_all(conn, settings)
|
||||
|
||||
assert stats["shows"] == 1
|
||||
assert (media_root / "clabretro" / "tvshow.nfo").exists()
|
||||
|
||||
|
||||
def test_channel_with_nothing_in_the_window_creates_no_directory(
|
||||
conn, settings, media_root, channel, no_thumbs
|
||||
):
|
||||
"""52 of 117 measured channels are in this state, and an empty series in
|
||||
Jellyfin looks like a bug rather than a quiet channel."""
|
||||
add_video(conn, channel["id"], "vid00000001", state=videos.SKIPPED_OLD)
|
||||
|
||||
runner.materialise_all(conn, settings)
|
||||
|
||||
assert not (media_root / "clabretro").exists()
|
||||
|
||||
|
||||
def test_materialise_all_honours_the_limit(conn, settings, media_root, channel,
|
||||
no_thumbs):
|
||||
for index in range(5):
|
||||
add_video(conn, channel["id"], f"vid{index:08d}")
|
||||
stats = runner.materialise_all(conn, settings, limit=2)
|
||||
assert stats["materialised"] == 2
|
||||
|
||||
|
||||
def test_materialise_skips_a_video_whose_channel_vanished(conn, settings,
|
||||
media_root, channel,
|
||||
no_thumbs):
|
||||
add_video(conn, channel["id"], "vid00000001")
|
||||
# Simulate the channel being unsubscribed between claiming and writing, with
|
||||
# foreign keys off so the row survives to be found.
|
||||
conn.execute("PRAGMA foreign_keys = OFF")
|
||||
with conn:
|
||||
conn.execute("DELETE FROM channel WHERE id = ?", (channel["id"],))
|
||||
|
||||
stats = runner.materialise_all(conn, settings)
|
||||
|
||||
assert stats["materialised"] == 0
|
||||
assert stats["errors"] == 0
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ full cycle
|
||||
|
||||
|
||||
def test_run_order_is_sync_poll_materialise_reap(conn, settings, media_root,
|
||||
channel, monkeypatch,
|
||||
quiet_jellyfin):
|
||||
order = []
|
||||
monkeypatch.setattr(subsync, "sync_all",
|
||||
lambda *a: order.append("sync") or {"added": 0, "queued": 0,
|
||||
"removed": 0, "refused": 0,
|
||||
"sources": 0,
|
||||
"pending_removal": 0})
|
||||
monkeypatch.setattr(discovery, "poll_all",
|
||||
lambda *a: order.append("poll") or {"queued": 0})
|
||||
monkeypatch.setattr(runner, "materialise_all",
|
||||
lambda *a, **k: order.append("materialise") or
|
||||
{"materialised": 0, "shows": 0, "errors": 0})
|
||||
monkeypatch.setattr(reap, "run",
|
||||
lambda *a: order.append("reap") or {"aged_out": 0})
|
||||
|
||||
runner.run(conn, settings)
|
||||
|
||||
assert order == ["sync", "poll", "materialise", "reap"]
|
||||
|
||||
|
||||
def test_single_channel_run_skips_the_sync(conn, settings, channel, monkeypatch,
|
||||
quiet_jellyfin):
|
||||
"""A targeted run is an operator action, not a mirror pass."""
|
||||
monkeypatch.setattr(subsync, "sync_all",
|
||||
lambda *a: pytest.fail("sync should not run"))
|
||||
monkeypatch.setattr(discovery, "poll_all", lambda *a: {"queued": 0})
|
||||
monkeypatch.setattr(reap, "run", lambda *a: {"aged_out": 0})
|
||||
|
||||
result = runner.run(conn, settings, channel_pk=channel["id"])
|
||||
|
||||
assert "sync" not in result
|
||||
|
||||
|
||||
def test_run_records_last_run_at(conn, settings, channel, monkeypatch,
|
||||
quiet_jellyfin):
|
||||
monkeypatch.setattr(subsync, "sync_all", lambda *a: {"added": 0, "queued": 0,
|
||||
"removed": 0, "refused": 0,
|
||||
"sources": 0,
|
||||
"pending_removal": 0})
|
||||
monkeypatch.setattr(discovery, "poll_all", lambda *a: {"queued": 0})
|
||||
monkeypatch.setattr(reap, "run", lambda *a: {"aged_out": 0})
|
||||
|
||||
runner.run(conn, settings)
|
||||
|
||||
assert settings.raw("last_run_at")
|
||||
|
||||
|
||||
def test_jellyfin_is_refreshed_only_when_the_tree_changed(
|
||||
conn, settings, media_root, channel, monkeypatch, quiet_jellyfin, no_thumbs
|
||||
):
|
||||
monkeypatch.setattr(subsync, "sync_all", lambda *a: {"added": 0, "queued": 0,
|
||||
"removed": 0, "refused": 0,
|
||||
"sources": 0,
|
||||
"pending_removal": 0})
|
||||
monkeypatch.setattr(discovery, "poll_all", lambda *a: {"queued": 0})
|
||||
|
||||
runner.run(conn, settings)
|
||||
assert quiet_jellyfin == []
|
||||
|
||||
add_video(conn, channel["id"], "vid00000001")
|
||||
runner.run(conn, settings)
|
||||
assert len(quiet_jellyfin) == 1
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ exit codes
|
||||
|
||||
|
||||
def test_refused_sync_turns_the_check_red():
|
||||
"""A silently-broken mirror is the worst outcome available: nothing looks
|
||||
wrong until someone asks why a channel never appeared."""
|
||||
assert runner.exit_code({"sync": {"refused": 1}}) == 1
|
||||
|
||||
|
||||
def test_materialise_errors_turn_the_check_red():
|
||||
assert runner.exit_code({"materialise": {"errors": 2}}) == 1
|
||||
|
||||
|
||||
def test_poll_failures_alone_do_not_turn_the_check_red():
|
||||
"""Two of 119 measured channels fail permanently — terminated or private."""
|
||||
assert runner.exit_code({"poll": {"failed": 2}}) == 0
|
||||
|
||||
|
||||
def test_clean_run_is_zero():
|
||||
assert runner.exit_code({"sync": {"refused": 0}, "poll": {"failed": 0},
|
||||
"materialise": {"errors": 0}}) == 0
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- summarise
|
||||
|
||||
|
||||
def test_summarise_mentions_a_refusal():
|
||||
text = runner.summarise({
|
||||
"sync": {"added": 0, "queued": 0, "removed": 0, "refused": 1},
|
||||
"poll": {"channels": 5, "queued": 0},
|
||||
"materialise": {"materialised": 0},
|
||||
"reap": {"aged_out": 0},
|
||||
})
|
||||
assert "SYNC_REFUSED" in text
|
||||
|
||||
|
||||
def test_summarise_is_a_single_line():
|
||||
text = runner.summarise({
|
||||
"sync": {"added": 1, "queued": 2, "removed": 3, "refused": 0},
|
||||
"poll": {"channels": 119, "queued": 4, "shorts": 5, "live": 6, "failed": 2},
|
||||
"materialise": {"materialised": 7, "errors": 0},
|
||||
"reap": {"aged_out": 8},
|
||||
})
|
||||
assert "\n" not in text
|
||||
assert "channels=119" in text
|
||||
assert "materialised=7" in text
|
||||
assert "aged_out=8" in text
|
||||
@@ -0,0 +1,125 @@
|
||||
"""Typed settings accessors and form validation."""
|
||||
|
||||
import pytest
|
||||
|
||||
from ytstream import settings as settings_module
|
||||
from ytstream.settings import DEFAULTS, Settings, validate, validate_all
|
||||
|
||||
|
||||
class TestAccessors:
|
||||
def test_missing_key_returns_the_default(self, settings):
|
||||
assert settings.get_int("retention_days") == 30
|
||||
assert settings.get_int("min_keep_videos") == 5
|
||||
assert settings.get_str("proxy_base_url") == "http://127.0.0.1:8099"
|
||||
|
||||
def test_unknown_key_never_raises(self, settings):
|
||||
assert settings.get_str("no_such_key") == ""
|
||||
assert settings.get_int("no_such_key") == 0
|
||||
|
||||
def test_set_then_get(self, settings):
|
||||
settings.set("retention_days", "30")
|
||||
assert settings.get_int("retention_days") == 30
|
||||
|
||||
def test_set_overwrites(self, settings):
|
||||
settings.set("retention_days", "30")
|
||||
settings.set("retention_days", "45")
|
||||
assert settings.get_int("retention_days") == 45
|
||||
|
||||
def test_corrupt_integer_falls_back_to_the_default(self, settings):
|
||||
settings.set("retention_days", "not a number")
|
||||
assert settings.get_int("retention_days") == 30
|
||||
|
||||
def test_reads_are_not_cached(self, conn, settings):
|
||||
"""Rotating the API key must be nothing more than saving a new value —
|
||||
no restart of the hourly job or the admin server."""
|
||||
settings.set("youtube_api_key", "first")
|
||||
assert settings.get_str("youtube_api_key") == "first"
|
||||
|
||||
# A second Settings object standing in for the other process.
|
||||
other = Settings(conn)
|
||||
other.set("youtube_api_key", "second")
|
||||
|
||||
assert settings.get_str("youtube_api_key") == "second"
|
||||
|
||||
def test_all_editable_covers_every_default(self, settings):
|
||||
assert set(settings.all_editable()) == set(DEFAULTS)
|
||||
|
||||
def test_secrets_are_not_editable(self):
|
||||
for key in settings_module.SECRET_KEYS:
|
||||
assert key not in settings_module.EDITABLE
|
||||
|
||||
def test_both_api_keys_are_masked(self):
|
||||
assert "youtube_api_key" in settings_module.MASKED_KEYS
|
||||
assert "jellyfin_api_key" in settings_module.MASKED_KEYS
|
||||
|
||||
def test_download_era_settings_are_gone(self):
|
||||
for gone in ("disk_cap_gb", "write_subs", "sub_langs", "sponsorblock_mark",
|
||||
"max_attempts", "backfill_days"):
|
||||
assert gone not in DEFAULTS
|
||||
|
||||
|
||||
class TestValidation:
|
||||
@pytest.mark.parametrize(
|
||||
"key, value",
|
||||
[
|
||||
("retention_days", "30"),
|
||||
("min_keep_videos", "0"),
|
||||
("min_keep_videos", "5"),
|
||||
("backfill_max_videos", "300"),
|
||||
("max_height", "1080"),
|
||||
("min_duration_seconds", "120"),
|
||||
("subsync_max_new", "25"),
|
||||
("subsync_missing_threshold", "3"),
|
||||
("jellyfin_url", "http://127.0.0.1:8096"),
|
||||
("pot_provider_url", "https://example.com:4416"),
|
||||
("proxy_base_url", "http://127.0.0.1:8099"),
|
||||
("youtube_api_key", ""),
|
||||
],
|
||||
)
|
||||
def test_accepts_good_values(self, key, value):
|
||||
ok, _ = validate(key, value)
|
||||
assert ok
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"key, value",
|
||||
[
|
||||
("retention_days", "abc"),
|
||||
("retention_days", "0"),
|
||||
("retention_days", "-5"),
|
||||
("min_keep_videos", "-1"),
|
||||
("max_height", "10"),
|
||||
("min_duration_seconds", "-1"),
|
||||
("jellyfin_url", "not-a-url"),
|
||||
("jellyfin_url", "ftp://host/"),
|
||||
("jellyfin_url", "http://"),
|
||||
("proxy_base_url", "nonsense"),
|
||||
],
|
||||
)
|
||||
def test_rejects_bad_values(self, key, value):
|
||||
ok, message = validate(key, value)
|
||||
assert not ok
|
||||
assert message
|
||||
|
||||
def test_zero_missing_threshold_is_rejected(self):
|
||||
"""Zero would unsubscribe on the first absent response, which is exactly
|
||||
the failure the threshold exists to prevent."""
|
||||
ok, message = validate("subsync_missing_threshold", "0")
|
||||
assert not ok
|
||||
assert message
|
||||
|
||||
def test_validate_all_reports_each_bad_field(self):
|
||||
errors = validate_all(
|
||||
{"retention_days": "abc", "max_height": "1080", "jellyfin_url": "nope"}
|
||||
)
|
||||
assert set(errors) == {"retention_days", "jellyfin_url"}
|
||||
|
||||
def test_validate_all_ignores_unknown_keys(self):
|
||||
assert validate_all({"not_a_setting": "x"}) == {}
|
||||
|
||||
def test_empty_api_keys_are_acceptable(self):
|
||||
assert validate("jellyfin_api_key", "")[0]
|
||||
assert validate("youtube_api_key", "")[0]
|
||||
|
||||
def test_whitespace_is_tolerated(self):
|
||||
ok, _ = validate("retention_days", " 21 ")
|
||||
assert ok
|
||||
@@ -0,0 +1,260 @@
|
||||
"""Materialising: .strm contents, NFO sidecars, naming, and idempotency."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import date
|
||||
|
||||
import pytest
|
||||
|
||||
from ytstream import config, strm, videos
|
||||
|
||||
from conftest import add_video
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def video(conn, channel):
|
||||
return add_video(conn, channel["id"], "dQw4w9WgXcQ",
|
||||
title="A Video: With/Punctuation",
|
||||
upload_date="2026-08-12", duration=1337)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def no_thumbs(monkeypatch):
|
||||
"""Thumbnails are a network fetch; every test here runs without one."""
|
||||
monkeypatch.setattr(strm, "fetch_thumbnail", lambda *a, **k: False)
|
||||
|
||||
|
||||
def test_strm_contains_only_the_proxy_url(conn, settings, media_root, channel,
|
||||
video, no_thumbs):
|
||||
result = strm.materialise(conn, settings, channel, video)
|
||||
|
||||
path = media_root / result["rel_path"]
|
||||
assert path.read_text() == "http://127.0.0.1:8099/watch/dQw4w9WgXcQ"
|
||||
# No trailing newline, and nothing else in the file.
|
||||
assert path.read_bytes() == b"http://127.0.0.1:8099/watch/dQw4w9WgXcQ"
|
||||
|
||||
|
||||
def test_strm_url_follows_the_configured_base(conn, settings, media_root, channel,
|
||||
video, no_thumbs):
|
||||
settings.set("proxy_base_url", "http://127.0.0.1:9999/")
|
||||
result = strm.materialise(conn, settings, channel, video)
|
||||
assert (media_root / result["rel_path"]).read_text().startswith(
|
||||
"http://127.0.0.1:9999/watch/"
|
||||
)
|
||||
|
||||
|
||||
def test_layout_matches_the_naming_scheme(conn, settings, media_root, channel,
|
||||
video, no_thumbs):
|
||||
result = strm.materialise(conn, settings, channel, video)
|
||||
|
||||
# Season = upload year, episode = MMDD*10 + ordinal.
|
||||
assert result["season"] == 2026
|
||||
assert result["episode"] == 8120
|
||||
assert result["rel_path"] == (
|
||||
"clabretro/Season 2026/"
|
||||
"clabretro - S2026E8120 - A Video With Punctuation [dQw4w9WgXcQ].strm"
|
||||
)
|
||||
|
||||
|
||||
def test_nfo_is_written_alongside(conn, settings, media_root, channel, video,
|
||||
no_thumbs):
|
||||
result = strm.materialise(conn, settings, channel, video)
|
||||
nfo_path = (media_root / result["rel_path"]).with_suffix(".nfo")
|
||||
|
||||
text = nfo_path.read_text()
|
||||
|
||||
assert "<season>2026</season>" in text
|
||||
assert "<episode>8120</episode>" in text
|
||||
assert "<aired>2026-08-12</aired>" in text
|
||||
# durationinseconds is what stops a .strm episode showing a zero runtime
|
||||
# before it has ever been played.
|
||||
assert "<durationinseconds>1337</durationinseconds>" in text
|
||||
assert 'type="youtube"' in text
|
||||
|
||||
|
||||
def test_nfo_has_no_streamdetails(conn, settings, media_root, channel, video,
|
||||
no_thumbs):
|
||||
"""Pre-seeding them was measured to change nothing about Jellyfin probing."""
|
||||
result = strm.materialise(conn, settings, channel, video)
|
||||
text = (media_root / result["rel_path"]).with_suffix(".nfo").read_text()
|
||||
assert "streamdetails" not in text
|
||||
assert "fileinfo" not in text
|
||||
|
||||
|
||||
def test_materialise_is_idempotent(conn, settings, media_root, channel, video,
|
||||
no_thumbs):
|
||||
first = strm.materialise(conn, settings, channel, video)
|
||||
before = (media_root / first["rel_path"]).read_bytes()
|
||||
|
||||
second = strm.materialise(
|
||||
conn, settings, channel, videos.get(conn, "dQw4w9WgXcQ")
|
||||
)
|
||||
|
||||
assert second["rel_path"] == first["rel_path"]
|
||||
assert (media_root / second["rel_path"]).read_bytes() == before
|
||||
|
||||
|
||||
def test_row_is_marked_materialised(conn, settings, media_root, channel, video,
|
||||
no_thumbs):
|
||||
strm.materialise(conn, settings, channel, video)
|
||||
row = videos.get(conn, "dQw4w9WgXcQ")
|
||||
assert row["state"] == videos.MATERIALISED
|
||||
assert row["rel_path"]
|
||||
assert row["materialised_at"]
|
||||
|
||||
|
||||
def test_episode_ordinals_increment_within_a_day(conn, settings, media_root,
|
||||
channel, no_thumbs):
|
||||
for index in range(3):
|
||||
row = add_video(conn, channel["id"], f"vid{index:08d}",
|
||||
upload_date="2026-08-12")
|
||||
result = strm.materialise(conn, settings, channel, row)
|
||||
assert result["episode"] == 8120 + index
|
||||
|
||||
|
||||
def test_ordinals_are_stable_when_an_earlier_video_ages_out(
|
||||
conn, settings, media_root, channel, no_thumbs
|
||||
):
|
||||
"""Aged-out rows keep their episode number, so later ordinals never shift."""
|
||||
first = add_video(conn, channel["id"], "vid00000001", upload_date="2026-08-12")
|
||||
strm.materialise(conn, settings, channel, first)
|
||||
videos.mark_aged_out(conn, "vid00000001")
|
||||
|
||||
second = add_video(conn, channel["id"], "vid00000002", upload_date="2026-08-12")
|
||||
result = strm.materialise(conn, settings, channel, second)
|
||||
|
||||
assert result["episode"] == 8121
|
||||
|
||||
|
||||
def test_title_falls_back_to_the_video_id(conn, settings, media_root, channel,
|
||||
no_thumbs):
|
||||
"""Backfilled rows carry no title until the feed supplies one."""
|
||||
row = add_video(conn, channel["id"], "vid00000001", title="",
|
||||
upload_date="2026-08-12")
|
||||
result = strm.materialise(conn, settings, channel, row)
|
||||
assert "vid00000001" in result["rel_path"]
|
||||
|
||||
|
||||
def test_write_show_creates_tvshow_nfo(media_root, channel):
|
||||
strm.write_show(channel)
|
||||
text = (media_root / "clabretro" / "tvshow.nfo").read_text()
|
||||
assert "<title>clabretro</title>" in text
|
||||
assert channel["channel_id"] in text
|
||||
|
||||
|
||||
def test_show_nfo_uses_the_channel_title_not_a_video_title(conn, media_root, channel):
|
||||
"""The channel/video join has `title` on both sides, and reading the wrong one
|
||||
renamed every series after whichever episode happened to be first."""
|
||||
add_video(conn, channel["id"], "vid00000001", title="Some Episode Title")
|
||||
strm.write_show(channel)
|
||||
text = (media_root / "clabretro" / "tvshow.nfo").read_text()
|
||||
assert "Some Episode Title" not in text
|
||||
|
||||
|
||||
def test_remove_deletes_strm_and_sidecars(conn, settings, media_root, channel,
|
||||
video, no_thumbs):
|
||||
result = strm.materialise(conn, settings, channel, video)
|
||||
strm_path = media_root / result["rel_path"]
|
||||
stem = strm_path.name[: -len(".strm")]
|
||||
thumb = strm_path.with_name(stem + "-thumb.jpg")
|
||||
thumb.write_bytes(b"x" * 2000)
|
||||
|
||||
removed = strm.remove(videos.get(conn, "dQw4w9WgXcQ"))
|
||||
|
||||
assert removed == 3 # .strm, .nfo, -thumb.jpg
|
||||
assert not strm_path.exists()
|
||||
assert not thumb.exists()
|
||||
|
||||
|
||||
def test_remove_leaves_files_it_does_not_own(conn, settings, media_root, channel,
|
||||
video, no_thumbs):
|
||||
result = strm.materialise(conn, settings, channel, video)
|
||||
season_dir = (media_root / result["rel_path"]).parent
|
||||
stranger = season_dir / "someone-elses-file.txt"
|
||||
stranger.write_text("not ours")
|
||||
|
||||
strm.remove(videos.get(conn, "dQw4w9WgXcQ"))
|
||||
|
||||
assert stranger.exists()
|
||||
|
||||
|
||||
def test_remove_channel_tree(conn, media_root, channel):
|
||||
tree = media_root / "clabretro"
|
||||
(tree / "Season 2026").mkdir(parents=True)
|
||||
(tree / "tvshow.nfo").write_text("<tvshow/>")
|
||||
|
||||
assert strm.remove_channel_tree(channel) is True
|
||||
assert not tree.exists()
|
||||
|
||||
|
||||
def test_remove_channel_tree_refuses_the_media_root(conn, media_root):
|
||||
from conftest import add_channel
|
||||
|
||||
row = add_channel(conn, "UC" + "q" * 22, "Blank", "")
|
||||
keep = media_root / "keep"
|
||||
keep.mkdir()
|
||||
|
||||
assert strm.remove_channel_tree(row) is False
|
||||
assert keep.exists()
|
||||
|
||||
|
||||
def test_fetch_thumbnail_rejects_the_grey_placeholder(monkeypatch, tmp_path):
|
||||
"""YouTube serves a tiny placeholder rather than a 404 for missing maxres."""
|
||||
import urllib.request
|
||||
|
||||
class Response:
|
||||
status = 200
|
||||
|
||||
def __init__(self, payload):
|
||||
self.payload = payload
|
||||
|
||||
def read(self):
|
||||
return self.payload
|
||||
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc):
|
||||
return False
|
||||
|
||||
calls = []
|
||||
|
||||
def fake_open(request, timeout=None):
|
||||
calls.append(request.full_url)
|
||||
# maxres returns the placeholder; hq returns something real.
|
||||
return Response(b"x" * 120 if "maxres" in request.full_url else b"y" * 5000)
|
||||
|
||||
monkeypatch.setattr(urllib.request, "urlopen", fake_open)
|
||||
destination = tmp_path / "out-thumb.jpg"
|
||||
|
||||
assert strm.fetch_thumbnail("dQw4w9WgXcQ", destination) is True
|
||||
assert destination.read_bytes() == b"y" * 5000
|
||||
assert len(calls) == 2
|
||||
|
||||
|
||||
def test_fetch_thumbnail_is_skipped_when_one_already_exists(monkeypatch, tmp_path):
|
||||
import urllib.request
|
||||
|
||||
monkeypatch.setattr(urllib.request, "urlopen",
|
||||
lambda *a, **k: pytest.fail("should not fetch"))
|
||||
existing = tmp_path / "out-thumb.jpg"
|
||||
existing.write_bytes(b"cached")
|
||||
|
||||
assert strm.fetch_thumbnail("dQw4w9WgXcQ", existing) is True
|
||||
|
||||
|
||||
def test_untitled_video_does_not_get_its_id_written_back_as_a_title(
|
||||
conn, settings, media_root, channel, no_thumbs
|
||||
):
|
||||
"""Writing the fallback back to the database makes the row look titled, which
|
||||
permanently disables the title repair in discovery._record. That shipped once
|
||||
and left five of twenty episodes named after their video ids."""
|
||||
row = add_video(conn, channel["id"], "vid00000001", title="",
|
||||
upload_date="2026-08-12")
|
||||
|
||||
result = strm.materialise(conn, settings, channel, row)
|
||||
|
||||
# The filename falls back to the id...
|
||||
assert "vid00000001]" in result["rel_path"]
|
||||
# ...but the row stays untitled, so a later feed poll can still repair it.
|
||||
assert videos.get(conn, "vid00000001")["title"] == ""
|
||||
@@ -0,0 +1,366 @@
|
||||
"""Subscription mirroring, and above all its refusals.
|
||||
|
||||
The sync is authoritative in both directions and the removal half deletes a
|
||||
channel's whole tree, so most of what needs pinning down here is what it does
|
||||
with *bad* data. Every one of these failure modes looks identical to "he
|
||||
unsubscribed from everything" on the wire:
|
||||
|
||||
403 subscriptionForbidden he re-ticked the privacy box
|
||||
network error susan's link dropped
|
||||
200 with zero items could be true, could be a broken response
|
||||
|
||||
None of them may delete anything. The tests below are the reason that claim can
|
||||
be made with a straight face.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from ytstream import api, subsync, videos
|
||||
|
||||
from conftest import CHANNEL_ID, FakeApi, add_channel, add_video, patch_api
|
||||
|
||||
BROTHER = "UCPcTWaLV8zwx4WP4QExHj4Q"
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def source(conn):
|
||||
key = subsync.add_source(conn, channel_id=BROTHER, label="C Flux")
|
||||
with conn:
|
||||
conn.execute("UPDATE source SET imported = 1 WHERE key = ?", (key,))
|
||||
return subsync.get_source(conn, key)
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def fresh_source(conn):
|
||||
"""A source that has never imported — the first-sync path."""
|
||||
key = subsync.add_source(conn, channel_id=BROTHER, label="C Flux")
|
||||
return subsync.get_source(conn, key)
|
||||
|
||||
|
||||
def sub(channel_id, title):
|
||||
return {"channel_id": channel_id, "title": title}
|
||||
|
||||
|
||||
def _fake(monkeypatch, **kwargs):
|
||||
return patch_api(monkeypatch, subsync, FakeApi(**kwargs))
|
||||
|
||||
|
||||
# ------------------------------------------------------------------- additions
|
||||
|
||||
|
||||
def test_first_sync_queues_everything_and_adds_nothing(
|
||||
conn, settings, fresh_source, monkeypatch
|
||||
):
|
||||
"""119 subscriptions would trip any cap, so day one is approval-only."""
|
||||
_fake(monkeypatch, subs=[sub(f"UC{i:022d}", f"Chan {i}") for i in range(40)])
|
||||
|
||||
stats = subsync.sync_source(conn, settings, fresh_source)
|
||||
|
||||
assert stats["added"] == 0
|
||||
assert stats["queued"] == 40
|
||||
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 0
|
||||
assert len(subsync.pending(conn)) == 40
|
||||
# And it does not queue them again on the next pass.
|
||||
assert subsync.get_source(conn, fresh_source["key"])["imported"] == 1
|
||||
|
||||
|
||||
def test_second_sync_adds_within_the_cap(conn, settings, source, monkeypatch):
|
||||
settings.set("subsync_max_new", "25")
|
||||
_fake(monkeypatch, subs=[sub("UCaaaaaaaaaaaaaaaaaaaaaa", "Alpha")],
|
||||
channel={"channel_id": "UCaaaaaaaaaaaaaaaaaaaaaa", "title": "Alpha",
|
||||
"description": "", "handle": None, "avatar_url": None})
|
||||
|
||||
stats = subsync.sync_source(conn, settings, source)
|
||||
|
||||
assert stats["added"] == 1
|
||||
row = conn.execute("SELECT * FROM channel").fetchone()
|
||||
assert row["title"] == "Alpha"
|
||||
assert row["source"] == "youtube"
|
||||
|
||||
|
||||
def test_burst_over_the_cap_adds_nothing_and_queues_all(
|
||||
conn, settings, source, monkeypatch
|
||||
):
|
||||
settings.set("subsync_max_new", "3")
|
||||
_fake(monkeypatch, subs=[sub(f"UC{i:022d}", f"Chan {i}") for i in range(10)])
|
||||
|
||||
stats = subsync.sync_source(conn, settings, source)
|
||||
|
||||
assert stats["added"] == 0
|
||||
assert stats["queued"] == 10
|
||||
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 0
|
||||
|
||||
|
||||
def test_rejected_channels_are_never_queued_again(
|
||||
conn, settings, fresh_source, monkeypatch
|
||||
):
|
||||
entries = [sub("UCaaaaaaaaaaaaaaaaaaaaaa", "Alpha")]
|
||||
_fake(monkeypatch, subs=entries)
|
||||
subsync.sync_source(conn, settings, fresh_source)
|
||||
|
||||
queued = subsync.pending(conn)
|
||||
assert len(queued) == 1
|
||||
subsync.resolve(conn, [queued[0]["id"]], "rejected")
|
||||
|
||||
source = subsync.get_source(conn, fresh_source["key"])
|
||||
stats = subsync.sync_source(conn, settings, source)
|
||||
|
||||
assert stats["queued"] == 0
|
||||
assert stats["added"] == 0
|
||||
assert subsync.pending(conn) == []
|
||||
|
||||
|
||||
def test_approving_subscribes(conn, settings, fresh_source, monkeypatch):
|
||||
_fake(monkeypatch, subs=[sub("UCaaaaaaaaaaaaaaaaaaaaaa", "Alpha")],
|
||||
channel={"channel_id": "UCaaaaaaaaaaaaaaaaaaaaaa", "title": "Alpha",
|
||||
"description": "Desc", "handle": "@alpha", "avatar_url": None})
|
||||
subsync.sync_source(conn, settings, fresh_source)
|
||||
queued = subsync.pending(conn)
|
||||
|
||||
stats = subsync.approve(conn, settings, [row["id"] for row in queued])
|
||||
|
||||
assert stats == {"added": 1, "failed": 0}
|
||||
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 1
|
||||
assert subsync.pending(conn) == []
|
||||
|
||||
|
||||
# -------------------------------------------------------------------- refusals
|
||||
|
||||
|
||||
def test_private_subscriptions_change_nothing(conn, settings, source, monkeypatch):
|
||||
add_channel(conn, CHANNEL_ID, "Existing", "Existing")
|
||||
_fake(monkeypatch, raises=api.SubscriptionsPrivate(
|
||||
403, "subscriptionForbidden", "not allowed"))
|
||||
|
||||
stats = subsync.sync_source(conn, settings, source)
|
||||
|
||||
assert stats["refused"]
|
||||
assert "private" in stats["refused"]
|
||||
assert stats["removed"] == 0
|
||||
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 1
|
||||
# The channel's miss counter must not move either, or three consecutive
|
||||
# outages would delete the library without a single healthy response.
|
||||
assert conn.execute("SELECT missing_syncs FROM channel").fetchone()[0] == 0
|
||||
|
||||
|
||||
def test_empty_response_is_treated_as_suspect(conn, settings, source, monkeypatch):
|
||||
"""A genuinely empty list and a broken one are indistinguishable, so assume
|
||||
the harmless reading."""
|
||||
add_channel(conn, CHANNEL_ID, "Existing", "Existing")
|
||||
_fake(monkeypatch, subs=[])
|
||||
|
||||
stats = subsync.sync_source(conn, settings, source)
|
||||
|
||||
assert stats["refused"]
|
||||
assert stats["removed"] == 0
|
||||
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 1
|
||||
|
||||
|
||||
def test_api_not_configured_changes_nothing(conn, settings, source, monkeypatch):
|
||||
add_channel(conn, CHANNEL_ID, "Existing", "Existing")
|
||||
_fake(monkeypatch, raises=api.NotConfigured(403, "forbidden", "blocked"))
|
||||
|
||||
stats = subsync.sync_source(conn, settings, source)
|
||||
|
||||
assert stats["refused"]
|
||||
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 1
|
||||
|
||||
|
||||
def test_network_error_changes_nothing(conn, settings, source, monkeypatch):
|
||||
add_channel(conn, CHANNEL_ID, "Existing", "Existing")
|
||||
_fake(monkeypatch, raises=api.ApiError(0, "network", "connection reset"))
|
||||
|
||||
stats = subsync.sync_source(conn, settings, source)
|
||||
|
||||
assert stats["refused"]
|
||||
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 1
|
||||
|
||||
|
||||
def test_refusal_is_recorded_on_the_source(conn, settings, source, monkeypatch):
|
||||
_fake(monkeypatch, raises=api.ApiError(0, "network", "boom"))
|
||||
subsync.sync_source(conn, settings, source)
|
||||
|
||||
row = subsync.get_source(conn, source["key"])
|
||||
assert row["last_sync_ok"] == 0
|
||||
assert row["consecutive_failures"] == 1
|
||||
assert "boom" in row["last_error"]
|
||||
|
||||
|
||||
def test_three_outages_in_a_row_still_delete_nothing(
|
||||
conn, settings, source, monkeypatch
|
||||
):
|
||||
"""The threshold counts absences from healthy responses, not failures."""
|
||||
add_channel(conn, CHANNEL_ID, "Existing", "Existing")
|
||||
_fake(monkeypatch, raises=api.ApiError(0, "network", "down"))
|
||||
|
||||
for _ in range(5):
|
||||
subsync.sync_source(conn, settings, subsync.get_source(conn, source["key"]))
|
||||
|
||||
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 1
|
||||
|
||||
|
||||
# -------------------------------------------------------------------- removals
|
||||
|
||||
|
||||
def test_absence_counts_up_but_does_not_delete_below_the_threshold(
|
||||
conn, settings, source, monkeypatch
|
||||
):
|
||||
settings.set("subsync_missing_threshold", "3")
|
||||
add_channel(conn, CHANNEL_ID, "Doomed", "Doomed")
|
||||
_fake(monkeypatch, subs=[sub("UCaaaaaaaaaaaaaaaaaaaaaa", "Alpha")])
|
||||
settings.set("subsync_max_new", "0") # keep the addition path out of this
|
||||
|
||||
for expected in (1, 2):
|
||||
stats = subsync.sync_source(
|
||||
conn, settings, subsync.get_source(conn, source["key"])
|
||||
)
|
||||
assert stats["removed"] == 0
|
||||
assert stats["pending_removal"] == 1
|
||||
assert conn.execute(
|
||||
"SELECT missing_syncs FROM channel WHERE dir_name = 'Doomed'"
|
||||
).fetchone()[0] == expected
|
||||
|
||||
|
||||
def test_deletion_happens_on_the_threshold_sync(
|
||||
conn, settings, source, media_root, monkeypatch
|
||||
):
|
||||
settings.set("subsync_missing_threshold", "3")
|
||||
settings.set("subsync_max_new", "0")
|
||||
doomed = add_channel(conn, CHANNEL_ID, "Doomed", "Doomed")
|
||||
tree = media_root / "Doomed"
|
||||
(tree / "Season 2026").mkdir(parents=True)
|
||||
(tree / "tvshow.nfo").write_text("<tvshow/>")
|
||||
add_video(conn, doomed["id"], "vid00000001")
|
||||
_fake(monkeypatch, subs=[sub("UCaaaaaaaaaaaaaaaaaaaaaa", "Alpha")])
|
||||
|
||||
for _ in range(3):
|
||||
stats = subsync.sync_source(
|
||||
conn, settings, subsync.get_source(conn, source["key"])
|
||||
)
|
||||
|
||||
assert stats["removed"] == 1
|
||||
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 0
|
||||
assert not tree.exists()
|
||||
# The videos went with it, via ON DELETE CASCADE.
|
||||
assert conn.execute("SELECT COUNT(*) FROM video").fetchone()[0] == 0
|
||||
|
||||
|
||||
def test_reappearing_resets_the_counter(conn, settings, source, monkeypatch):
|
||||
settings.set("subsync_missing_threshold", "3")
|
||||
settings.set("subsync_max_new", "0")
|
||||
add_channel(conn, CHANNEL_ID, "Flaky", "Flaky")
|
||||
|
||||
_fake(monkeypatch, subs=[sub("UCaaaaaaaaaaaaaaaaaaaaaa", "Alpha")])
|
||||
subsync.sync_source(conn, settings, subsync.get_source(conn, source["key"]))
|
||||
assert conn.execute("SELECT missing_syncs FROM channel").fetchone()[0] == 1
|
||||
|
||||
_fake(monkeypatch, subs=[sub(CHANNEL_ID, "Flaky")])
|
||||
subsync.sync_source(conn, settings, subsync.get_source(conn, source["key"]))
|
||||
|
||||
assert conn.execute("SELECT missing_syncs FROM channel").fetchone()[0] == 0
|
||||
|
||||
|
||||
def test_manual_channels_are_never_removed(
|
||||
conn, settings, source, media_root, monkeypatch
|
||||
):
|
||||
settings.set("subsync_missing_threshold", "1")
|
||||
settings.set("subsync_max_new", "0")
|
||||
add_channel(conn, CHANNEL_ID, "Pinned", "Pinned", source="manual")
|
||||
_fake(monkeypatch, subs=[sub("UCaaaaaaaaaaaaaaaaaaaaaa", "Alpha")])
|
||||
|
||||
for _ in range(3):
|
||||
stats = subsync.sync_source(
|
||||
conn, settings, subsync.get_source(conn, source["key"])
|
||||
)
|
||||
|
||||
assert stats["removed"] == 0
|
||||
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 1
|
||||
assert conn.execute("SELECT missing_syncs FROM channel").fetchone()[0] == 0
|
||||
|
||||
|
||||
def test_threshold_of_one_deletes_on_the_first_absence(
|
||||
conn, settings, source, monkeypatch
|
||||
):
|
||||
"""Configurable so tests need not loop; the default stays at 3."""
|
||||
settings.set("subsync_missing_threshold", "1")
|
||||
settings.set("subsync_max_new", "0")
|
||||
add_channel(conn, CHANNEL_ID, "Doomed", "Doomed")
|
||||
_fake(monkeypatch, subs=[sub("UCaaaaaaaaaaaaaaaaaaaaaa", "Alpha")])
|
||||
|
||||
stats = subsync.sync_source(conn, settings, source)
|
||||
|
||||
assert stats["removed"] == 1
|
||||
|
||||
|
||||
def test_zero_threshold_is_clamped_to_one(conn, settings, source, monkeypatch):
|
||||
"""A stored 0 must not mean "delete before any absence is confirmed"."""
|
||||
settings.set("subsync_missing_threshold", "0")
|
||||
settings.set("subsync_max_new", "0")
|
||||
add_channel(conn, CHANNEL_ID, "Doomed", "Doomed")
|
||||
_fake(monkeypatch, subs=[sub("UCaaaaaaaaaaaaaaaaaaaaaa", "Alpha")])
|
||||
|
||||
stats = subsync.sync_source(conn, settings, source)
|
||||
|
||||
# Clamped to 1, so the first absence is enough — but it took one absence,
|
||||
# not zero, and the row was actually observed missing.
|
||||
assert stats["removed"] == 1
|
||||
assert stats["seen"] == 1
|
||||
|
||||
|
||||
# --------------------------------------------------------------------- general
|
||||
|
||||
|
||||
def test_unsafe_channel_dir_is_not_deleted(conn, settings, media_root):
|
||||
"""A blank dir_name must never resolve the delete to the media root."""
|
||||
row = add_channel(conn, CHANNEL_ID, "Bad", " ")
|
||||
(media_root / "keepme").mkdir()
|
||||
|
||||
from ytstream import strm
|
||||
|
||||
assert strm.remove_channel_tree(row) is False
|
||||
assert (media_root / "keepme").exists()
|
||||
|
||||
|
||||
def test_one_unresolvable_channel_does_not_abort_the_sync(
|
||||
conn, settings, source, monkeypatch
|
||||
):
|
||||
class Exploding(FakeApi):
|
||||
def channel(self, channel_id):
|
||||
if channel_id.endswith("bad"):
|
||||
raise api.ApiError(500, "backendError", "boom")
|
||||
return {"channel_id": channel_id, "title": "Fine", "description": "",
|
||||
"handle": None, "avatar_url": None}
|
||||
|
||||
monkeypatch.setattr(
|
||||
subsync.channels, "subscribe_from_sync",
|
||||
lambda conn, settings, cid, title: (_ for _ in ()).throw(RuntimeError("no"))
|
||||
if cid.endswith("bad") else add_channel(conn, cid, title, title),
|
||||
)
|
||||
patch_api(monkeypatch, subsync,
|
||||
FakeApi(subs=[sub("UC" + "a" * 19 + "bad", "Bad"),
|
||||
sub("UC" + "b" * 22, "Good")]))
|
||||
|
||||
stats = subsync.sync_source(conn, settings, source)
|
||||
|
||||
assert stats["added"] == 1
|
||||
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 1
|
||||
|
||||
|
||||
def test_sync_all_aggregates_and_counts_refusals(conn, settings, source, monkeypatch):
|
||||
_fake(monkeypatch, raises=api.ApiError(0, "network", "boom"))
|
||||
totals = subsync.sync_all(conn, settings)
|
||||
assert totals["sources"] == 1
|
||||
assert totals["refused"] == 1
|
||||
|
||||
|
||||
def test_disabled_source_is_skipped(conn, settings, source, monkeypatch):
|
||||
with conn:
|
||||
conn.execute("UPDATE source SET enabled = 0")
|
||||
fake = _fake(monkeypatch, subs=[sub("UCaaaaaaaaaaaaaaaaaaaaaa", "Alpha")])
|
||||
|
||||
totals = subsync.sync_all(conn, settings)
|
||||
|
||||
assert totals["sources"] == 0
|
||||
assert fake.subscription_calls == 0
|
||||
@@ -0,0 +1,209 @@
|
||||
"""The video state machine and episode numbering."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import date
|
||||
|
||||
import pytest
|
||||
|
||||
from ytstream import videos
|
||||
|
||||
from conftest import add_channel, add_video
|
||||
|
||||
|
||||
def test_insert_and_get(conn, channel):
|
||||
row = add_video(conn, channel["id"], "vid00000001")
|
||||
assert row["state"] == videos.LISTED
|
||||
assert videos.exists(conn, "vid00000001")
|
||||
assert not videos.exists(conn, "nosuchvideo")
|
||||
|
||||
|
||||
def test_insert_is_idempotent(conn, channel):
|
||||
add_video(conn, channel["id"], "vid00000001", title="First")
|
||||
add_video(conn, channel["id"], "vid00000001", title="Second")
|
||||
assert conn.execute("SELECT COUNT(*) FROM video").fetchone()[0] == 1
|
||||
assert videos.get(conn, "vid00000001")["title"] == "First"
|
||||
|
||||
|
||||
def test_aged_out_is_terminal_and_never_revived():
|
||||
assert videos.AGED_OUT in videos.TERMINAL
|
||||
assert videos.AGED_OUT in videos.NEVER_REVIVE
|
||||
# skipped_old is terminal but IS revivable by rescan; conflating the two
|
||||
# would resurrect months of deleted episodes as new.
|
||||
assert videos.SKIPPED_OLD in videos.TERMINAL
|
||||
assert videos.SKIPPED_OLD not in videos.NEVER_REVIVE
|
||||
|
||||
|
||||
def test_no_download_era_states_survive():
|
||||
"""Materialising a text file cannot fail, so there is no retry ladder."""
|
||||
for gone in ("pending", "downloading", "downloaded", "failed", "deferred"):
|
||||
assert gone not in (
|
||||
videos.LISTED, videos.MATERIALISED, videos.SKIPPED_SHORT,
|
||||
videos.SKIPPED_LIVE, videos.SKIPPED_OLD, videos.AGED_OUT,
|
||||
)
|
||||
|
||||
|
||||
def test_mark_aged_out_clears_rel_path_but_keeps_the_row(conn, channel):
|
||||
add_video(conn, channel["id"], "vid00000001", state=videos.MATERIALISED)
|
||||
with conn:
|
||||
conn.execute("UPDATE video SET rel_path = 'a/b.strm' WHERE video_id = ?",
|
||||
("vid00000001",))
|
||||
|
||||
videos.mark_aged_out(conn, "vid00000001")
|
||||
|
||||
row = videos.get(conn, "vid00000001")
|
||||
assert row is not None
|
||||
assert row["state"] == videos.AGED_OUT
|
||||
assert row["rel_path"] is None
|
||||
assert row["deleted_at"]
|
||||
|
||||
|
||||
def test_mark_materialised_records_everything(conn, channel):
|
||||
add_video(conn, channel["id"], "vid00000001")
|
||||
videos.mark_materialised(conn, "vid00000001", rel_path="a/b.strm", season=2026,
|
||||
episode=8120, upload_date="2026-08-12", duration=99,
|
||||
title="T")
|
||||
row = videos.get(conn, "vid00000001")
|
||||
assert (row["state"], row["season"], row["episode"], row["duration"]) == (
|
||||
videos.MATERIALISED, 2026, 8120, 99
|
||||
)
|
||||
|
||||
|
||||
def test_set_duration_alone(conn, channel):
|
||||
add_video(conn, channel["id"], "vid00000001", duration=None)
|
||||
videos.set_duration(conn, "vid00000001", 1234)
|
||||
assert videos.get(conn, "vid00000001")["duration"] == 1234
|
||||
|
||||
|
||||
# ------------------------------------------------------------ episode numbers
|
||||
|
||||
|
||||
def test_first_episode_of_a_day(conn, channel):
|
||||
add_video(conn, channel["id"], "vid00000001")
|
||||
assert videos.next_episode(
|
||||
conn, channel["id"], date(2026, 8, 12), "vid00000001"
|
||||
) == (2026, 8120)
|
||||
|
||||
|
||||
def test_ordinal_counts_existing_rows_for_that_day(conn, channel):
|
||||
add_video(conn, channel["id"], "vid00000001")
|
||||
with conn:
|
||||
conn.execute("UPDATE video SET season = 2026, episode = 8120 "
|
||||
"WHERE video_id = ?", ("vid00000001",))
|
||||
add_video(conn, channel["id"], "vid00000002")
|
||||
|
||||
assert videos.next_episode(
|
||||
conn, channel["id"], date(2026, 8, 12), "vid00000002"
|
||||
) == (2026, 8121)
|
||||
|
||||
|
||||
def test_ordinal_ignores_other_days(conn, channel):
|
||||
add_video(conn, channel["id"], "vid00000001")
|
||||
with conn:
|
||||
conn.execute("UPDATE video SET season = 2026, episode = 8110 "
|
||||
"WHERE video_id = ?", ("vid00000001",))
|
||||
add_video(conn, channel["id"], "vid00000002")
|
||||
|
||||
assert videos.next_episode(
|
||||
conn, channel["id"], date(2026, 8, 12), "vid00000002"
|
||||
) == (2026, 8120)
|
||||
|
||||
|
||||
def test_ordinal_ignores_other_channels(conn, channel):
|
||||
other = add_channel(conn, "UC" + "z" * 22, "Other", "Other")
|
||||
add_video(conn, other["id"], "vid00000001")
|
||||
with conn:
|
||||
conn.execute("UPDATE video SET season = 2026, episode = 8120 "
|
||||
"WHERE video_id = ?", ("vid00000001",))
|
||||
add_video(conn, channel["id"], "vid00000002")
|
||||
|
||||
assert videos.next_episode(
|
||||
conn, channel["id"], date(2026, 8, 12), "vid00000002"
|
||||
) == (2026, 8120)
|
||||
|
||||
|
||||
def test_ordinal_clamps_at_ten_uploads_a_day(conn, channel):
|
||||
for index in range(10):
|
||||
video_id = f"vid{index:08d}"
|
||||
add_video(conn, channel["id"], video_id)
|
||||
with conn:
|
||||
conn.execute("UPDATE video SET season = 2026, episode = ? "
|
||||
"WHERE video_id = ?", (8120 + index, video_id))
|
||||
add_video(conn, channel["id"], "vid00000099")
|
||||
|
||||
season, episode = videos.next_episode(
|
||||
conn, channel["id"], date(2026, 8, 12), "vid00000099"
|
||||
)
|
||||
|
||||
assert (season, episode) == (2026, 8129)
|
||||
|
||||
|
||||
def test_a_videos_own_row_does_not_bump_its_ordinal(conn, channel):
|
||||
"""Re-materialising must produce the same number, not the next one."""
|
||||
add_video(conn, channel["id"], "vid00000001")
|
||||
with conn:
|
||||
conn.execute("UPDATE video SET season = 2026, episode = 8120 "
|
||||
"WHERE video_id = ?", ("vid00000001",))
|
||||
|
||||
assert videos.next_episode(
|
||||
conn, channel["id"], date(2026, 8, 12), "vid00000001"
|
||||
) == (2026, 8120)
|
||||
|
||||
|
||||
# --------------------------------------------------------------------- queues
|
||||
|
||||
|
||||
def test_claim_listed_returns_only_listed_rows(conn, channel):
|
||||
add_video(conn, channel["id"], "listed00001", state=videos.LISTED)
|
||||
add_video(conn, channel["id"], "done0000001", state=videos.MATERIALISED)
|
||||
add_video(conn, channel["id"], "short000001", state=videos.SKIPPED_SHORT)
|
||||
add_video(conn, channel["id"], "aged0000001", state=videos.AGED_OUT)
|
||||
|
||||
rows = videos.claim_listed(conn)
|
||||
|
||||
assert [row["video_id"] for row in rows] == ["listed00001"]
|
||||
|
||||
|
||||
def test_claim_listed_is_oldest_upload_first(conn, channel):
|
||||
add_video(conn, channel["id"], "newer000001", upload_date="2026-08-10")
|
||||
add_video(conn, channel["id"], "older000001", upload_date="2026-01-01")
|
||||
|
||||
rows = videos.claim_listed(conn)
|
||||
|
||||
assert [row["video_id"] for row in rows] == ["older000001", "newer000001"]
|
||||
|
||||
|
||||
def test_claim_listed_honours_the_limit(conn, channel):
|
||||
for index in range(5):
|
||||
add_video(conn, channel["id"], f"vid{index:08d}")
|
||||
assert len(videos.claim_listed(conn, limit=2)) == 2
|
||||
|
||||
|
||||
def test_materialised_for_channel_is_newest_first(conn, channel):
|
||||
"""The retention sweep counts down from the newest to honour min_keep_videos."""
|
||||
add_video(conn, channel["id"], "older000001", upload_date="2026-01-01",
|
||||
state=videos.MATERIALISED)
|
||||
add_video(conn, channel["id"], "newer000001", upload_date="2026-08-10",
|
||||
state=videos.MATERIALISED)
|
||||
|
||||
rows = videos.materialised_for_channel(conn, channel["id"])
|
||||
|
||||
assert [row["video_id"] for row in rows] == ["newer000001", "older000001"]
|
||||
|
||||
|
||||
def test_queue_depth_and_counts(conn, channel):
|
||||
add_video(conn, channel["id"], "listed00001", state=videos.LISTED)
|
||||
add_video(conn, channel["id"], "listed00002", state=videos.LISTED)
|
||||
add_video(conn, channel["id"], "done0000001", state=videos.MATERIALISED)
|
||||
|
||||
assert videos.queue_depth(conn) == 2
|
||||
counts = videos.counts_by_state(conn)
|
||||
assert counts[videos.LISTED] == 2
|
||||
assert counts[videos.MATERIALISED] == 1
|
||||
|
||||
|
||||
def test_deleting_a_channel_cascades_to_its_videos(conn, channel):
|
||||
add_video(conn, channel["id"], "vid00000001")
|
||||
with conn:
|
||||
conn.execute("DELETE FROM channel WHERE id = ?", (channel["id"],))
|
||||
assert conn.execute("SELECT COUNT(*) FROM video").fetchone()[0] == 0
|
||||
@@ -0,0 +1,146 @@
|
||||
"""The admin UI: rendering, auth gating and CSRF.
|
||||
|
||||
The templates are f-strings over a dict, so a renamed key is a KeyError at render
|
||||
time rather than a type error at import time — which is exactly the kind of
|
||||
breakage that only shows up when somebody opens the page. These tests render every
|
||||
page for real.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from ytstream import videos
|
||||
from ytstream.web import auth, templates
|
||||
|
||||
from conftest import add_channel, add_video
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def channel_row(conn, settings, channel):
|
||||
"""The dict shape server.py builds for the channel table."""
|
||||
add_video(conn, channel["id"], "vid00000001", state=videos.MATERIALISED)
|
||||
return {
|
||||
"id": channel["id"],
|
||||
"title": channel["title"],
|
||||
"handle": channel["handle"],
|
||||
"channel_id": channel["channel_id"],
|
||||
"retention_days": None,
|
||||
"global_retention": 30,
|
||||
"last_polled_at": "2026-08-12T14:00:00+00:00",
|
||||
"last_poll_ok": 1,
|
||||
"consecutive_poll_failures": 0,
|
||||
"episodes": 1,
|
||||
"source": "youtube",
|
||||
"missing_syncs": 0,
|
||||
"missing_threshold": 3,
|
||||
"latest": "2026-08-12",
|
||||
}
|
||||
|
||||
|
||||
def test_login_page_renders(settings):
|
||||
html = templates.login_page().decode()
|
||||
assert "ytstream" in html
|
||||
assert "youtube-automate" not in html
|
||||
assert 'type="password"' in html
|
||||
|
||||
|
||||
def test_login_page_shows_an_error():
|
||||
assert "Wrong" in templates.login_page("Wrong password").decode()
|
||||
|
||||
|
||||
def test_index_page_renders(channel_row):
|
||||
html = templates.index_page(
|
||||
channels=[channel_row], settings_values={"retention_days": "30"},
|
||||
settings_errors={}, csrf="tok", add_error=None, queue_depth=0,
|
||||
).decode()
|
||||
|
||||
assert "clabretro" in html
|
||||
assert "ytstream" in html
|
||||
# The provenance column replaced the meaningless size column.
|
||||
assert "youtube" in html
|
||||
assert "Size" not in html
|
||||
|
||||
|
||||
def test_index_page_renders_with_no_channels():
|
||||
html = templates.index_page(
|
||||
channels=[], settings_values={}, settings_errors={}, csrf="tok",
|
||||
add_error=None, queue_depth=0,
|
||||
).decode()
|
||||
assert "0 channel(s)" in html
|
||||
|
||||
|
||||
def test_absence_badge_appears_before_the_channel_disappears(channel_row):
|
||||
"""A channel counting towards removal must be visible while it still exists."""
|
||||
channel_row["missing_syncs"] = 2
|
||||
html = templates.index_page(
|
||||
channels=[channel_row], settings_values={}, settings_errors={}, csrf="tok",
|
||||
add_error=None, queue_depth=0,
|
||||
).decode()
|
||||
assert "absent 2/3 syncs" in html
|
||||
|
||||
|
||||
def test_no_absence_badge_when_healthy(channel_row):
|
||||
html = templates.index_page(
|
||||
channels=[channel_row], settings_values={}, settings_errors={}, csrf="tok",
|
||||
add_error=None, queue_depth=0,
|
||||
).decode()
|
||||
assert "absent" not in html
|
||||
|
||||
|
||||
def test_hostile_channel_title_is_escaped(channel_row):
|
||||
channel_row["title"] = '<script>alert("x")</script>'
|
||||
html = templates.index_page(
|
||||
channels=[channel_row], settings_values={}, settings_errors={}, csrf="tok",
|
||||
add_error=None, queue_depth=0,
|
||||
).decode()
|
||||
assert "<script>alert" not in html
|
||||
assert "<script>" in html
|
||||
|
||||
|
||||
def test_settings_errors_are_rendered_inline(channel_row):
|
||||
html = templates.index_page(
|
||||
channels=[channel_row], settings_values={"retention_days": "abc"},
|
||||
settings_errors={"retention_days": "must be a whole number"},
|
||||
csrf="tok", add_error=None, queue_depth=0,
|
||||
).decode()
|
||||
assert "must be a whole number" in html
|
||||
|
||||
|
||||
def test_masked_settings_are_not_rendered_in_plaintext(channel_row):
|
||||
"""Both API keys are secrets; neither belongs in the HTML."""
|
||||
html = templates.index_page(
|
||||
channels=[channel_row],
|
||||
settings_values={"youtube_api_key": "AIzaSECRETVALUE",
|
||||
"jellyfin_api_key": "JELLYSECRET"},
|
||||
settings_errors={}, csrf="tok", add_error=None, queue_depth=0,
|
||||
).decode()
|
||||
assert "AIzaSECRETVALUE" not in html
|
||||
assert "JELLYSECRET" not in html
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------ csrf
|
||||
|
||||
|
||||
def test_csrf_token_round_trips():
|
||||
secret = auth.new_secret()
|
||||
session = "session-token"
|
||||
token = auth.csrf_token(secret, session)
|
||||
assert auth.verify_csrf(secret, session, token)
|
||||
|
||||
|
||||
def test_csrf_token_is_bound_to_the_session():
|
||||
secret = auth.new_secret()
|
||||
token = auth.csrf_token(secret, "session-a")
|
||||
assert not auth.verify_csrf(secret, "session-b", token)
|
||||
|
||||
|
||||
def test_csrf_token_rejects_tampering():
|
||||
secret = auth.new_secret()
|
||||
token = auth.csrf_token(secret, "s")
|
||||
assert not auth.verify_csrf(secret, "s", token[:-1] + "x")
|
||||
|
||||
|
||||
def test_csrf_rejects_an_empty_token():
|
||||
secret = auth.new_secret()
|
||||
assert not auth.verify_csrf(secret, "s", "")
|
||||
Reference in New Issue
Block a user