Files
ytstream/ytstream/strm.py
T
Tom FluxandClaude Opus 5 155f05773d Build ytstream: catalogue, retention, subscription mirror, proxy
Phases 1-4 of plan.md §13. Forked from youtube-automate as planned rather than
written from scratch: naming, NFO, auth, the admin UI, settings and the DB layer
came across largely unchanged, download.py is gone, and the pieces that only make
sense for a streaming library are new.

  api.py       YouTube Data API v3 client. The whole metadata path.
  strm.py      Materialising: a .strm, an .nfo and a thumbnail. Replaces the
               330-line download.py, because the job is writing a URL to a file.
  subsync.py   The subscription mirror, most of which is refusals.
  reap.py      Retention, rewritten around the 30-day window and min_keep_videos.
  discovery.py RSS polling plus an API-backed, resumable, bounded backfill.
  proxy/       The verified PoC, moved in with a systemd unit.

330 tests, all passing, no network and no yt-dlp in any of them. The suite leans
towards the failure paths, because that is where this design can actually hurt
someone: a 403 that looks like an unsubscribe, a video that ages out and comes
back, a title that never arrives. tests/test_proxy.py replaces the two standalone
scripts under proxy/ and now drives the real make_handler(mgr, ...) rather than the
PoC's make_handler(path, done), so routing and video-id validation are covered too.

Ran it end to end against the live API and it found three real bugs.

The first was mine and the tests caught it: strm.remove pruned empty directories
up to the media root, so a channel directory whose tvshow.nfo happened to be
missing would be deleted along with the season. It only looked safe because
tvshow.nfo normally stops the walk. The prune boundary is now the channel
directory explicitly.

The other two only showed up against real data, and they compounded. The backfill
inserted rows with no title and left the RSS poll to fill them in — but RSS returns
15 entries, which for Pitch Side spans 23 days against a 30-day window, so five of
twenty episodes were named after their video ids. Worse, strm.materialise wrote
that fallback back to the database as the title, which made the row look titled and
permanently disabled the repair path. Both fixed: playlistItems.list now requests
snippet as well as contentDetails, which costs the same single quota unit and
carries the title alongside the exact publish date, and the fallback is used for the
filename without being persisted. A title that does arrive late now also removes the
badly-named files and re-queues, so the episode is rewritten rather than keeping its
video-id name forever. Verified against the live API: all twenty Pitch Side episodes
now carry real titles.

Measured on the real account: 119 subscriptions queued for approval and none added
on the first sync, then a two-channel run backfilled and materialised 26 episodes in
under seven seconds.

Two deliberate departures from plan.md, both recorded there:

min_keep_videos defaults to 5 rather than being left as an open question. Without
it 52 of 117 measured channels are empty Jellyfin series that flicker in and out as
their single video crosses the retention line, and the plan already recommended it.

The Jellyfin refresh is a bare /Library/Refresh with a comment explaining why it
must stay that way. A normal scan makes zero media probes; FullRefresh does probe,
and at 400 episodes that is 400 cold starts.

Not yet done: no systemd units are installed (needs root — deploy/deploy.sh), the
admin UI has no routes for sources or the approval queue yet, and nothing has been
pointed at the real media root.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-08-12 16:35:23 +01:00

208 lines
7.4 KiB
Python

"""Materialising a video: write a `.strm`, an `.nfo` and a thumbnail.
This is what replaced youtube-automate's 330-line `download.py`. There is no
subprocess, no format selection, no progress parsing and no retry ladder, because
the whole job is writing a URL into a text file. Everything expensive was moved to
the moment somebody presses play, which is the point of the design.
Thumbnails come from `i.ytimg.com`, which needs no API key and no quota.
"""
from __future__ import annotations
import logging
import shutil
import sqlite3
import urllib.error
import urllib.request
from pathlib import Path
from . import config, naming, nfo, util, videos
from .settings import Settings
log = logging.getLogger(__name__)
THUMB_SUFFIX = "-thumb.jpg"
# maxres does not exist for every video; hq always does.
_THUMB_URLS = (
"https://i.ytimg.com/vi/{vid}/maxresdefault.jpg",
"https://i.ytimg.com/vi/{vid}/hqdefault.jpg",
)
def watch_url(settings: Settings, video_id: str) -> str:
base = settings.get_str("proxy_base_url").rstrip("/")
return f"{base}/watch/{video_id}"
def channel_dir(channel: sqlite3.Row) -> Path:
return config.MEDIA_ROOT / channel["dir_name"]
def episode_paths(channel: sqlite3.Row, season: int, episode: int,
title: str, video_id: str) -> tuple[Path, str]:
"""(.strm path, path relative to the media root)."""
stem = naming.basename(channel["dir_name"], season, episode, title, video_id)
relative = Path(channel["dir_name"]) / naming.season_dir_name(season) / (stem + ".strm")
return config.MEDIA_ROOT / relative, str(relative)
def write_show(channel: sqlite3.Row) -> None:
"""tvshow.nfo for a channel.
Takes the row's show fields explicitly rather than trusting a joined row: the
channel/video join has `title` on both sides, and reading the wrong one
renamed every series after whichever episode happened to be first. That
shipped once already.
"""
directory = channel_dir(channel)
directory.mkdir(parents=True, exist_ok=True)
nfo.write(
directory / "tvshow.nfo",
nfo.tvshow_nfo(
title=channel["title"],
plot=channel["description"],
channel_id=channel["channel_id"],
),
)
def fetch_thumbnail(video_id: str, destination: Path, *, timeout: float = 20.0) -> bool:
"""Best effort. A missing thumbnail is cosmetic, never a failure."""
if destination.exists():
return True
for template in _THUMB_URLS:
url = template.format(vid=video_id)
request = urllib.request.Request(url, headers={"User-Agent": config.USER_AGENT})
try:
with urllib.request.urlopen(request, timeout=timeout) as response:
if response.status != 200:
continue
payload = response.read()
except (urllib.error.HTTPError, urllib.error.URLError, OSError):
continue
if len(payload) < 1024:
# YouTube serves a 120-byte grey placeholder rather than a 404.
continue
destination.parent.mkdir(parents=True, exist_ok=True)
temporary = destination.with_name(destination.name + ".tmp")
temporary.write_bytes(payload)
temporary.replace(destination)
return True
log.debug("no thumbnail available for %s", video_id)
return False
def materialise(
conn: sqlite3.Connection,
settings: Settings,
channel: sqlite3.Row,
video: sqlite3.Row,
) -> dict:
"""Write one episode's `.strm`, `.nfo` and thumbnail, and record it.
Idempotent: re-running over an already-materialised video rewrites the same
bytes to the same paths.
"""
upload_date = naming.parse_upload_date(video["upload_date"])
season, episode = videos.next_episode(
conn, channel["id"], upload_date, video["video_id"]
)
# `title` is for the filename; `stored_title` is what goes back to the
# database. They differ when we have no title yet: writing the video id back
# would make the row look titled, permanently disabling the repair path in
# discovery._record that fills titles in from a later feed poll. That bug
# shipped once and renamed five of twenty episodes after their video ids.
stored_title = video["title"] or ""
title = stored_title or video["video_id"]
strm_path, relative = episode_paths(
channel, season, episode, title, video["video_id"]
)
strm_path.parent.mkdir(parents=True, exist_ok=True)
# No trailing newline: some Jellyfin versions have historically been fussy
# about trailing whitespace in .strm files, and there is nothing to gain.
temporary = strm_path.with_name(strm_path.name + ".tmp")
temporary.write_text(watch_url(settings, video["video_id"]), encoding="utf-8")
temporary.replace(strm_path)
nfo.write(
strm_path.with_suffix(".nfo"),
nfo.episode_nfo(
title=title,
show_title=channel["title"],
season=season,
episode=episode,
plot=None,
aired=upload_date.isoformat(),
duration_seconds=video["duration"],
video_id=video["video_id"],
),
)
stem = strm_path.name[: -len(".strm")]
fetch_thumbnail(video["video_id"], strm_path.with_name(stem + THUMB_SUFFIX))
videos.mark_materialised(
conn,
video["video_id"],
rel_path=relative,
season=season,
episode=episode,
upload_date=upload_date.isoformat(),
duration=video["duration"],
title=stored_title,
)
return {"video_id": video["video_id"], "rel_path": relative,
"season": season, "episode": episode}
def remove(video: sqlite3.Row) -> int:
"""Delete an episode's files. Returns how many were removed.
Only touches suffixes we know we wrote. A stray file someone else put in the
season directory is not ours to delete.
"""
if not video["rel_path"]:
return 0
strm_path = config.MEDIA_ROOT / video["rel_path"]
stem = strm_path.name[: -len(".strm")]
removed = 0
for path in [strm_path] + [
strm_path.with_name(stem + suffix) for suffix in config.SIDECAR_SUFFIXES
]:
try:
path.unlink()
removed += 1
except FileNotFoundError:
continue
except OSError as exc:
log.warning("could not remove %s: %s", path, exc)
# Prune empty season directories, but stop at the channel directory — it
# holds tvshow.nfo and the artwork, and an active subscription must not
# vanish from Jellyfin just because it published nothing this year. Passing
# MEDIA_ROOT as the boundary instead would delete the channel directory on
# any channel whose tvshow.nfo happened to be missing.
channel_dir = config.MEDIA_ROOT / Path(video["rel_path"]).parts[0]
util.prune_empty_dirs(strm_path.parent, channel_dir)
return removed
def remove_channel_tree(channel: sqlite3.Row) -> bool:
"""Delete a whole channel directory, on unsubscribe.
Guarded against deleting the media root itself, which a channel whose
dir_name somehow ended up empty would otherwise do.
"""
directory = channel_dir(channel)
if directory == config.MEDIA_ROOT or not str(channel["dir_name"]).strip():
log.error("refusing to remove %s: unsafe channel directory", directory)
return False
if not directory.exists():
return False
shutil.rmtree(directory)
log.info("removed %s", directory)
return True