tom/ytstream
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Build ytstream: catalogue, retention, subscription mirror, proxy

Browse Source 
Phases 1-4 of plan.md §13. Forked from youtube-automate as planned rather than
written from scratch: naming, NFO, auth, the admin UI, settings and the DB layer
came across largely unchanged, download.py is gone, and the pieces that only make
sense for a streaming library are new.

  api.py       YouTube Data API v3 client. The whole metadata path.
  strm.py      Materialising: a .strm, an .nfo and a thumbnail. Replaces the
               330-line download.py, because the job is writing a URL to a file.
  subsync.py   The subscription mirror, most of which is refusals.
  reap.py      Retention, rewritten around the 30-day window and min_keep_videos.
  discovery.py RSS polling plus an API-backed, resumable, bounded backfill.
  proxy/       The verified PoC, moved in with a systemd unit.

330 tests, all passing, no network and no yt-dlp in any of them. The suite leans
towards the failure paths, because that is where this design can actually hurt
someone: a 403 that looks like an unsubscribe, a video that ages out and comes
back, a title that never arrives. tests/test_proxy.py replaces the two standalone
scripts under proxy/ and now drives the real make_handler(mgr, ...) rather than the
PoC's make_handler(path, done), so routing and video-id validation are covered too.

Ran it end to end against the live API and it found three real bugs.

The first was mine and the tests caught it: strm.remove pruned empty directories
up to the media root, so a channel directory whose tvshow.nfo happened to be
missing would be deleted along with the season. It only looked safe because
tvshow.nfo normally stops the walk. The prune boundary is now the channel
directory explicitly.

The other two only showed up against real data, and they compounded. The backfill
inserted rows with no title and left the RSS poll to fill them in — but RSS returns
15 entries, which for Pitch Side spans 23 days against a 30-day window, so five of
twenty episodes were named after their video ids. Worse, strm.materialise wrote
that fallback back to the database as the title, which made the row look titled and
permanently disabled the repair path. Both fixed: playlistItems.list now requests
snippet as well as contentDetails, which costs the same single quota unit and
carries the title alongside the exact publish date, and the fallback is used for the
filename without being persisted. A title that does arrive late now also removes the
badly-named files and re-queues, so the episode is rewritten rather than keeping its
video-id name forever. Verified against the live API: all twenty Pitch Side episodes
now carry real titles.

Measured on the real account: 119 subscriptions queued for approval and none added
on the first sync, then a two-channel run backfilled and materialised 26 episodes in
under seven seconds.

Two deliberate departures from plan.md, both recorded there:

min_keep_videos defaults to 5 rather than being left as an open question. Without
it 52 of 117 measured channels are empty Jellyfin series that flicker in and out as
their single video crosses the retention line, and the plan already recommended it.

The Jellyfin refresh is a bare /Library/Refresh with a comment explaining why it
must stay that way. A normal scan makes zero media probes; FullRefresh does probe,
and at 400 episodes that is 400 cold starts.

Not yet done: no systemd units are installed (needs root — deploy/deploy.sh), the
admin UI has no routes for sources or the approval queue yet, and nothing has been
pointed at the real media root.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Tom Flux
2026-08-12 16:35:23 +01:00
co-authored by Claude Opus 5
parent f640c064c6
commit 155f05773d
48 changed files with 8964 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
deploy/bootstrap.sh
Executable
+46
View File
@@ -0,0 +1,46 @@
#!/bin/bash
# Everything that does NOT need root. Run as susan, before deploy.sh.
#
# /opt/ytstream/deploy/bootstrap.sh
set -euo pipefail
STATE=/var/lib/ytstream
VENV=$STATE/venv
REPO=/opt/ytstream
say() { printf '\n\033[1m==> %s\033[0m\n' "$1"; }
if [[ ! -d $STATE ]]; then
echo "$STATE does not exist yet — run 'sudo $REPO/deploy/deploy.sh' first," >&2
echo "or create it with: sudo install -d -o susan -g automation -m 0770 $STATE" >&2
exit 1
fi
say "Creating the virtualenv"
python3 -m venv "$VENV"
"$VENV/bin/pip" install --quiet --upgrade pip
"$VENV/bin/pip" install --quiet -e "$REPO"
say "Verifying yt-dlp and the POT plugin"
# The plugin is what makes YouTube reachable at all. A venv without it looks fine
# until the first playback fails with a bare 403.
"$VENV/bin/yt-dlp" --version
"$VENV/bin/python3" - <<'PY'
import importlib.util
missing = [name for name in ("yt_dlp_plugins",) if importlib.util.find_spec(name) is None]
print("POT plugin:", "MISSING" if missing else "present")
raise SystemExit(1 if missing else 0)
PY
say "Starting the POT provider container if it is not already up"
if ! curl -sf --max-time 5 http://127.0.0.1:4416/ping >/dev/null 2>&1; then
docker run -d --name bgutil-provider --restart unless-stopped \
-p 4416:4416 brainicism/bgutil-ytdlp-pot-provider || true
else
echo " already responding on 127.0.0.1:4416"
fi
say "Initialising the database"
"$VENV/bin/ytstream" status || true
say "Done — now run 'sudo $REPO/deploy/deploy.sh'"
deploy/crontab.fragment
+15
View File
@@ -0,0 +1,15 @@
# ytstream — add these to susan's crontab (`crontab -e`).
#
# HC_API_URL is already set at the top of susan's crontab; these follow the
# existing one-UUID-per-job convention. `sg mediaserver` guarantees new files are
# group-owned by mediaserver even if the invoking shell's primary group differs.
#
# Get fresh UUIDs from hc.jihakuz.xyz before enabling these.
# Sync subscriptions, poll, materialise, reap. Hourly. Runs at :23 to stay clear
# of youtube-automate's :17 entry during the overlap period.
23 * * * * runitor -uuid REPLACE-WITH-UUID -- sg mediaserver "/usr/local/bin/ytstream run"
# Keep yt-dlp current — this is the thing that keeps playback working as YouTube
# changes. Mondays 04:50, after youtube-automate's 04:40 slot.
50 4 * * 1 runitor -uuid REPLACE-WITH-UUID -- /opt/ytstream/deploy/update-ytdlp.sh
deploy/deploy.sh
Executable
+89
View File
@@ -0,0 +1,89 @@
#!/bin/bash
# Root-requiring installation steps for ytstream.
#
# susan has no passwordless sudo, so everything needing root is collected here for
# the operator to run in one go:
#
# sudo /opt/ytstream/deploy/deploy.sh
#
# Everything that does NOT need root — the venv, the database, the POT provider
# container, subscriptions, the API key — is handled by `bootstrap.sh` and the
# application itself. Run bootstrap.sh (as susan) first.
set -euo pipefail
REPO=/opt/ytstream
STATE=/var/lib/ytstream
VENV=$STATE/venv
HOSTNAME_=tube.jihakuz.xyz
if [[ $EUID -ne 0 ]]; then
echo "This script needs root. Run: sudo $0" >&2
exit 1
fi
say() { printf '\n\033[1m==> %s\033[0m\n' "$1"; }
say "Creating $STATE"
# root-owned directory, group-writable by `automation` so susan's cron job and the
# admin server can both write the database.
install -d -o susan -g automation -m 0770 "$STATE"
say "Installing the /usr/local/bin shim"
cat > /usr/local/bin/ytstream <<EOF
#!/bin/sh
# Thin shim onto the venv entry point.
exec $VENV/bin/ytstream "\$@"
EOF
chmod 0755 /usr/local/bin/ytstream
chown root:automation /usr/local/bin/ytstream
echo " /usr/local/bin/ytstream"
say "Preparing the media root"
# setgid so new directories inherit `mediaserver` — without it Jellyfin loses
# access to anything created after the fact. See plan.md §9.
install -d -o susan -g mediaserver -m 2770 /disks/Plex/_ytstream
say "Installing systemd units"
for unit in ytstream-proxy ytstream-admin; do
install -m 0644 "$REPO/deploy/$unit.service" "/etc/systemd/system/$unit.service"
echo " $unit.service"
done
systemctl daemon-reload
systemctl enable --now ytstream-proxy.service ytstream-admin.service
for unit in ytstream-proxy ytstream-admin; do
systemctl --no-pager --lines=5 status "$unit.service" || true
done
say "nginx vhost for $HOSTNAME_"
# tube.jihakuz.xyz is served by a leftover TubeArchivist server block inside
# sites-available/jihakuz.xyz, which owns the Let's Encrypt certificate and wins
# because nginx uses the FIRST server block matching a name. Installing a second
# vhost for the same name silently does nothing. youtube-automate repointed that
# block at 8085; ytstream needs it on 8086.
if grep -rql "server_name $HOSTNAME_" /etc/nginx/sites-enabled/ 2>/dev/null; then
echo " $HOSTNAME_ is already served by an existing vhost."
echo " Repoint its proxy_pass to http://127.0.0.1:8086 by hand, then:"
echo " nginx -t && systemctl reload nginx"
echo " (Deliberately not edited automatically — that block also serves"
echo " other names and owns the TLS certificate.)"
else
echo " No existing vhost found. Install one proxying to 127.0.0.1:8086"
echo " and run: certbot --nginx -d $HOSTNAME_"
fi
say "Done"
cat <<'EOF'
Remaining steps, all as susan and none needing root:
ytstream set-password # admin UI login
ytstream set-jellyfin-key # verified against the live server
ytstream set-youtube-key # verified against the live API
ytstream add-source @cflux1030 # the mirrored account
ytstream sync # queues the subscriptions
ytstream pending # review them
ytstream approve --all # or approve a subset by id
ytstream run # first real cycle
ytstream doctor # confirm everything is wired up
Then add the cron entries from deploy/crontab.fragment to susan's crontab.
EOF
deploy/update-ytdlp.sh
Executable
+23
View File
@@ -0,0 +1,23 @@
#!/bin/bash
# Keep yt-dlp current inside ytstream's venv, then confirm the install still works.
#
# This is the single most important recurring job: YouTube changes its player and
# streaming protocol frequently, and a stale yt-dlp means every playback fails.
set -euo pipefail
VENV=/var/lib/ytstream/venv
before=$("$VENV/bin/yt-dlp" --version 2>/dev/null || echo "none")
"$VENV/bin/pip" install --quiet --upgrade "yt-dlp[default]"
after=$("$VENV/bin/yt-dlp" --version)
echo "yt-dlp: $before -> $after"
if [[ "$before" != "$after" ]]; then
# A new yt-dlp can change format availability, so restart the proxy to drop
# any cached extraction state and re-run the checks.
systemctl restart ytstream-proxy.service 2>/dev/null \
|| echo "could not restart the proxy (needs root); do it by hand"
fi
exec "$VENV/bin/ytstream" doctor
deploy/ytstream-admin.service
+34
View File
@@ -0,0 +1,34 @@
[Unit]
Description=ytstream admin server
Documentation=file:///opt/ytstream/plan.md
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
# See plan.md §9 — the same ownership rules as the proxy, because subscribing a
# channel through the UI writes into the media tree.
User=susan
Group=mediaserver
UMask=0002
Environment=PATH=/var/lib/ytstream/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Environment=PYTHONUNBUFFERED=1
WorkingDirectory=/opt/ytstream
# Port 8086: youtube-automate holds 8085 for as long as the two run side by side.
ExecStart=/var/lib/ytstream/venv/bin/ytstream serve --host 127.0.0.1 --port 8086
Restart=always
RestartSec=5
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ProtectControlGroups=true
ProtectKernelTunables=true
RestrictSUIDSGID=true
ReadWritePaths=/var/lib/ytstream /disks/Plex/_ytstream
[Install]
WantedBy=multi-user.target
deploy/ytstream-proxy.service
+46
View File
@@ -0,0 +1,46 @@
[Unit]
Description=ytstream just-in-time YouTube streaming proxy
Documentation=file:///opt/ytstream/plan.md
After=network-online.target docker.service
Wants=network-online.target
# The bgutil POT provider runs in Docker on 127.0.0.1:4416 and yt-dlp cannot
# fetch anything without it.
Requires=docker.service
[Service]
Type=simple
# Group=mediaserver and UMask=0002 are load-bearing: Jellyfin reaches the media
# tree only through the mediaserver group. See plan.md §9.
User=susan
Group=mediaserver
UMask=0002
# LOAD-BEARING. The only yt-dlp with the bgutil POT plugin is the one in this
# venv. /usr/local/bin/yt-dlp is a 2023.11.16 binary that cannot talk to YouTube
# at all, and if it wins the PATH race every playback fails with no clear reason.
Environment=PATH=/var/lib/ytstream/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Environment=PYTHONUNBUFFERED=1
WorkingDirectory=/opt/ytstream
ExecStart=/var/lib/ytstream/venv/bin/python3 /opt/ytstream/proxy/ytstream_proxy.py \
--host 127.0.0.1 --port 8099 \
--work /dev/shm/ytstream \
--cache-gb 8 \
--max-pipelines 2 \
--max-retries 2 \
--max-starts 20 --starts-window 3600
Restart=always
RestartSec=5
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ProtectControlGroups=true
ProtectKernelTunables=true
RestrictSUIDSGID=true
# /dev/shm is the cache; nothing else needs to be writable.
ReadWritePaths=/dev/shm
[Install]
WantedBy=multi-user.target
proxy/ytstream_proxy.py
+755
View File
@@ -0,0 +1,755 @@
#!/usr/bin/env python3
"""
ytstream -- just-in-time YouTube streaming proxy for Jellyfin.
Serves one endpoint per video:
GET /watch/<video_id> muxed avc1+mp4a MP4, range-capable
GET /healthz JSON status (sessions, cache use, counters)
A `.strm` file whose contents are `http://127.0.0.1:8099/watch/<video_id>` plays
that video without any of its bytes ever having been stored on disk beforehand.
ffmpeg reads only local FIFOs; every piece of YouTube protocol handling stays
inside yt-dlp. See FINDINGS.md for what has and has not been verified.
Safety rails, because a Jellyfin library scan can ask for every episode at once:
--no-fetch never start a pipeline. Log the request and return 503.
Use this for a first library scan: it reveals whether
Jellyfin probes .strm targets with zero YouTube traffic.
--max-pipelines cap on CONCURRENT pipelines (default 2). Excess -> 503.
--max-starts cap on TOTAL cold starts per window (default 20/hour). This
is the one that bounds a runaway metadata refresh: concurrency
alone only slows a 1249-episode churn down, it does not stop
it. A person watching podcasts starts a handful of videos an
hour; a refresh storm hits the cap in seconds.
--cache-gb tmpfs budget; least-recently-used complete files are evicted.
Two serving modes, as in the PoC:
default wait for the mux to finish, then serve. Correct duration, ranges
and seeking. Costs time-to-first-byte (~60x realtime pull, so a
46-minute video is ready in about 50s).
--growing serve while writing. Low TTFB, but a probe of a partially written
fragmented MP4 reports only the duration written so far. ffmpeg
ignores both mvhd.duration and an injected mehd box, so this
cannot be fixed in the container.
"""
import argparse
import collections
import json
import os
import re
import shutil
import subprocess
import sys
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
POT_ARGS = "youtubepot-bgutilhttp:base_url=http://127.0.0.1:4416"
# player_client: `mweb` is deliberately NOT included. Measured 2026-08-12:
# * mweb formats 403 on every attempt, both directly and via --load-info-json,
# even though they are the only ones carrying a PO token in the URL.
# * mweb is also the ONLY source of the DRC and dubbed-language variants that
# caused the two original picker bugs. Dropping it removes both hazards
# before the picker ever sees them.
# * `default` resolves to android_vr for avc1+mp4a, which works without a PO
# token. web / ios / web_safari / tv are SABR-only and yield no usable
# formats at all. Left as `default` rather than pinned to `android_vr` so a
# yt-dlp update can follow YouTube if android_vr stops working.
CLIENT_ARGS = "youtube:player_client=default"
MAX_HEIGHT = 720
STALL_TIMEOUT = 45.0
VIDEO_ID_RE = re.compile(r"^[A-Za-z0-9_-]{11}$")
_log_lock = threading.Lock()
_access_log = None
def log(msg):
print(f"[ytstream] {msg}", file=sys.stderr, flush=True)
def access(msg):
"""Append to the access log. This is the evidence trail for the scan-probe
question, so it is written unbuffered and never dropped."""
line = f"{time.strftime('%Y-%m-%dT%H:%M:%S')} {msg}"
with _log_lock:
print(line, file=sys.stderr, flush=True)
if _access_log:
with open(_access_log, "a") as f:
f.write(line + "\n")
# --------------------------------------------------------------------------
# Extraction and format selection
# --------------------------------------------------------------------------
def extract_info(url, work):
"""One extraction. Both downloads reuse it via --load-info-json, which is
verified not to re-extract."""
info_path = os.path.join(work, "info.json")
cmd = [
"yt-dlp", "-J", "--no-warnings",
"--extractor-args", CLIENT_ARGS,
"--extractor-args", POT_ARGS,
url,
]
t0 = time.monotonic()
out = subprocess.run(cmd, capture_output=True, text=True)
if out.returncode != 0:
raise RuntimeError(f"yt-dlp -J failed: {out.stderr.strip()[:500]}")
info = json.loads(out.stdout)
with open(info_path, "w") as f:
json.dump(info, f)
log(f"extracted {info.get('id')} in {time.monotonic() - t0:.1f}s -- "
f"{info.get('title')!r}")
return info, info_path
def pick_formats(info):
"""Prefer avc1 video and the original-language mp4a audio so the mux is a
pure copy and Jellyfin can direct-play."""
fmts = info.get("formats", [])
def usable(f):
return f.get("url") and f.get("protocol", "").startswith("http")
def is_drc(f):
# YouTube advertises DRC (dynamic-range-compressed) audio variants that
# carry the same abr as their plain counterparts but 403 on download.
# Verified against NH2MhBFQm9w: 140-drc -> 403, 140 -> fine.
return "-drc" in (f.get("format_id") or "") or "DRC" in (f.get("format_note") or "")
vids = [
f for f in fmts
if usable(f)
and f.get("vcodec") not in (None, "none")
and f.get("acodec") in (None, "none")
and (f.get("height") or 0) <= MAX_HEIGHT
]
auds = [
f for f in fmts
if usable(f)
and f.get("acodec") not in (None, "none")
and f.get("vcodec") in (None, "none")
]
# Ranking notes, all learned the hard way against real videos:
#
# * language_preference outranks everything. Multi-language uploads expose
# 140-0..140-N with IDENTICAL abr, so max() would otherwise tie-break on
# list order and silently pick whichever dub YouTube listed first --
# German audio on an English podcast (verified: J9O3sxoMs5U). The
# original track carries language_preference 10, dubs carry -1.
# * non-DRC next, because DRC variants 403. Language correctness ranks
# above this deliberately: a 403 is a loud failure, wrong-language audio
# is a silent one that would ship.
# * only then codec and bitrate.
def langpref(f):
return f.get("language_preference") or 0
def vkey(f):
return (
not is_drc(f),
f.get("vcodec", "").startswith("avc1"),
f.get("height") or 0,
f.get("tbr") or 0,
)
def akey(f):
return (
langpref(f),
not is_drc(f),
f.get("acodec", "").startswith("mp4a"),
f.get("abr") or 0,
)
if not vids or not auds:
raise RuntimeError(
"no separate video+audio pair; formats present: " + ", ".join(
f"{f.get('format_id')}({f.get('vcodec')}/{f.get('acodec')})"
for f in fmts[:15]))
v = max(vids, key=vkey)
a = max(auds, key=akey)
log(f" video {v['format_id']} {v.get('vcodec')} {v.get('height')}p | "
f"audio {a['format_id']} {a.get('acodec')} lang={a.get('language')}")
if not v.get("vcodec", "").startswith("avc1"):
log(" WARNING: no h264 at this height -- Jellyfin may transcode")
if len({f.get("language") for f in auds}) > 1 and langpref(a) <= 0:
log(f" WARNING: multi-language upload and the chosen track "
f"({a.get('language')}) is not the original -- expect a dub")
return v["format_id"], a["format_id"]
# --------------------------------------------------------------------------
# Producer: two yt-dlp -> FIFOs -> ffmpeg -c copy -> fragmented MP4
# --------------------------------------------------------------------------
def start_producer(info_path, vfmt, afmt, work):
vfifo = os.path.join(work, "v.fifo")
afifo = os.path.join(work, "a.fifo")
for p in (vfifo, afifo):
if os.path.exists(p):
os.unlink(p)
os.mkfifo(p)
out_path = os.path.join(work, "out.mp4")
done_path = out_path + ".done"
err_path = out_path + ".err"
procs = []
def feed(fmt, fifo, tag):
# Opening a FIFO for write blocks until a reader attaches, so ffmpeg
# must already be starting -- it is, just below.
fh = open(fifo, "wb")
p = subprocess.Popen(
[
"yt-dlp", "--load-info-json", info_path,
"-f", fmt, "-o", "-", "--quiet", "--no-warnings",
"--extractor-args", CLIENT_ARGS,
"--extractor-args", POT_ARGS,
],
stdout=fh, stderr=subprocess.PIPE,
)
procs.append((tag, p))
fh.close()
ff = subprocess.Popen(
[
"ffmpeg", "-y", "-loglevel", "error",
"-i", vfifo, "-i", afifo,
"-c", "copy",
"-movflags", "frag_keyframe+empty_moov+default_base_moof",
"-f", "mp4", out_path,
],
stderr=subprocess.PIPE,
)
threading.Thread(target=feed, args=(vfmt, vfifo, "video"), daemon=True).start()
threading.Thread(target=feed, args=(afmt, afifo, "audio"), daemon=True).start()
def reap():
rc = ff.wait()
err = ff.stderr.read().decode(errors="replace").strip()
problems = []
if rc != 0:
problems.append(f"ffmpeg exited {rc}: {err[:400]}")
for tag, p in procs:
if p.poll() not in (0, None):
problems.append(
f"yt-dlp[{tag}] exited {p.returncode}: "
f"{p.stderr.read().decode(errors='replace')[:300]}")
if problems:
# A single 403 on either stream kills the run and there is no retry
# yet; record it so /healthz and the caller can see why.
with open(err_path, "w") as f:
f.write("\n".join(problems))
log(f"producer FAILED in {work}: {problems[0]}")
else:
log(f"mux complete: {out_path} {os.path.getsize(out_path)} bytes")
open(done_path, "w").close()
threading.Thread(target=reap, daemon=True).start()
return out_path, done_path, err_path
# --------------------------------------------------------------------------
# Session manager
# --------------------------------------------------------------------------
class Session:
"""One video's pipeline and its output file.
`final` is the terminal signal: set once the session has either produced a
complete file or given up after retries. It is deliberately separate from
the producer's own done-marker, because a failed attempt writes that marker
too and a waiting request must not mistake a retry for a finished file.
"""
def __init__(self, video_id, work):
self.video_id = video_id
self.work = work
self.out_path = os.path.join(work, "out.mp4")
self.started = time.time()
self.last_used = time.time()
self.final = threading.Event()
self.failed = None
self.attempts = 0
self.readers = 0
@property
def complete(self):
"""True only when the file is finished AND good."""
return self.final.is_set() and not self.failed
def size(self):
try:
return os.path.getsize(self.out_path)
except OSError:
return 0
class Manager:
def __init__(self, work_root, max_pipelines, cache_bytes, no_fetch, growing,
max_retries=2, max_starts=20, starts_window=3600.0):
self.work_root = work_root
self.max_pipelines = max_pipelines
self.max_retries = max_retries
self.max_starts = max_starts
self.starts_window = starts_window
# Timestamps of cold starts, pruned to the window. Cache hits are not
# recorded: re-watching or resuming must never be rate limited.
self.start_log = collections.deque()
self.cache_bytes = cache_bytes
self.no_fetch = no_fetch
self.growing = growing
self.lock = threading.Lock()
self.sessions = {}
self.counters = {"requests": 0, "started": 0, "reused": 0,
"refused_nofetch": 0, "refused_busy": 0,
"refused_ratelimit": 0, "retried": 0, "failed": 0,
"evicted": 0}
def _active(self):
return [s for s in self.sessions.values() if not s.complete]
def get(self, video_id):
"""Return (session, error_string). Never raises."""
with self.lock:
self.counters["requests"] += 1
s = self.sessions.get(video_id)
if s is not None:
s.last_used = time.time()
self.counters["reused"] += 1
return s, None
if self.no_fetch:
self.counters["refused_nofetch"] += 1
return None, "no-fetch mode: refusing to start a pipeline"
if len(self._active()) >= self.max_pipelines:
self.counters["refused_busy"] += 1
return None, (f"at pipeline cap ({self.max_pipelines}); "
f"refusing to start another")
# Cold-start budget. Refusing produces a short retry burst from
# libavformat and then aborts whatever refresh triggered it, which
# is the intended protective outcome: an aborted refresh costs some
# re-derivable metadata, a runaway one costs the whole catalogue in
# YouTube traffic.
now = time.monotonic()
while self.start_log and now - self.start_log[0] > self.starts_window:
self.start_log.popleft()
if len(self.start_log) >= self.max_starts:
self.counters["refused_ratelimit"] += 1
oldest = self.starts_window - (now - self.start_log[0])
log(f"RATE LIMIT: {len(self.start_log)} cold starts in the last "
f"{self.starts_window / 60:.0f}min, refusing {video_id}. "
f"Budget frees in {oldest:.0f}s. If this was a library "
f"refresh, that refresh is being stopped on purpose.")
return None, (f"cold-start budget exhausted "
f"({self.max_starts} per "
f"{self.starts_window / 60:.0f}min)")
self.start_log.append(now)
work = os.path.join(self.work_root, video_id)
shutil.rmtree(work, ignore_errors=True)
os.makedirs(work, exist_ok=True)
s = Session(video_id, work)
self.sessions[video_id] = s
self.counters["started"] += 1
threading.Thread(target=self._run, args=(s,), daemon=True).start()
return s, None
def _run(self, s):
"""Run the pipeline, retrying on failure with a fresh extraction.
Intermittent 403s do happen -- observed on both the video and the audio
stream of videos that succeeded minutes earlier, clustered after heavy
use, so most likely transient rate limiting. A single one used to kill
playback outright. Each retry re-extracts, because the resolved URLs and
their PO token binding are the most likely thing to have gone stale.
Only retried while nothing has been served yet (size == 0). If bytes
already went out we cannot rewind under a reader.
"""
url = f"https://www.youtube.com/watch?v={s.video_id}"
last = None
for attempt in range(1, self.max_retries + 2):
s.attempts = attempt
try:
shutil.rmtree(s.work, ignore_errors=True)
os.makedirs(s.work, exist_ok=True)
info, info_path = extract_info(url, s.work)
vfmt, afmt = pick_formats(info)
_out, done_path, err_path = start_producer(
info_path, vfmt, afmt, s.work)
while not os.path.exists(done_path):
time.sleep(0.2)
if not os.path.exists(err_path):
last = None
break
with open(err_path) as f:
last = f.read().strip()
except Exception as e: # noqa: BLE001 - report anything
last = str(e)[:500]
if s.size() > 0:
log(f"{s.video_id}: attempt {attempt} failed after serving "
f"bytes -- not retrying")
break
if attempt <= self.max_retries:
with self.lock:
self.counters["retried"] += 1
log(f"{s.video_id}: attempt {attempt} failed "
f"({(last or '')[:90]}) -- retrying")
time.sleep(2.0 * attempt)
s.failed = last
if last:
with self.lock:
self.counters["failed"] += 1
log(f"session {s.video_id} gave up after {s.attempts} attempt(s)")
s.final.set()
self.evict()
def evict(self):
"""Drop least-recently-used complete sessions until under budget."""
with self.lock:
done = [s for s in self.sessions.values()
if s.complete and s.readers == 0]
total = sum(s.size() for s in self.sessions.values())
for s in sorted(done, key=lambda x: x.last_used):
if total <= self.cache_bytes:
break
total -= s.size()
self.sessions.pop(s.video_id, None)
shutil.rmtree(s.work, ignore_errors=True)
self.counters["evicted"] += 1
log(f"evicted {s.video_id}")
def status(self):
with self.lock:
return {
"mode": "growing" if self.growing else "wait-for-complete",
"no_fetch": self.no_fetch,
"max_pipelines": self.max_pipelines,
"max_starts": self.max_starts,
"starts_in_window": len(self.start_log),
"starts_window_min": round(self.starts_window / 60),
"cache_gb": round(self.cache_bytes / 2**30, 2),
"cache_used_gb": round(
sum(s.size() for s in self.sessions.values()) / 2**30, 3),
"counters": dict(self.counters),
"sessions": [
{"video_id": s.video_id,
"complete": s.complete,
"bytes": s.size(),
"readers": s.readers,
"age_s": round(time.time() - s.started, 1),
"attempts": s.attempts,
"error": s.failed}
for s in self.sessions.values()
],
}
# --------------------------------------------------------------------------
# HTTP
# --------------------------------------------------------------------------
def make_handler(mgr, wait_timeout):
class Handler(BaseHTTPRequestHandler):
protocol_version = "HTTP/1.1"
def log_message(self, *a):
pass # we do our own logging
# -- range helpers (verified by test_range.py) ----------------------
def _size(self, path):
return os.path.getsize(path) if os.path.exists(path) else 0
def _wait_for_bytes(self, path, offset, complete):
deadline = time.monotonic() + STALL_TIMEOUT
while True:
size = self._size(path)
if size > offset or complete():
return size
if time.monotonic() > deadline:
raise TimeoutError
time.sleep(0.05)
def _parse_range(self, rng, total):
"""Parse a single byte range against a known total size.
Returns (start, end) inclusive, "unsatisfiable", or None when there
is no usable range and the whole body should be sent. Multi-range
requests fall into the None case: answering with the whole body is
legal and beats mis-serving one part.
"""
if not rng or not rng.startswith("bytes=") or "," in rng:
return None
first, sep, last = rng[6:].strip().partition("-")
if not sep:
return None
try:
if not first: # bytes=-N -> final N bytes
n = int(last)
return (max(0, total - n), total - 1) if n > 0 else "unsatisfiable"
start = int(first)
end = int(last) if last else total - 1
except ValueError:
return None
if start >= total or start > end:
return "unsatisfiable"
return start, min(end, total - 1)
# -- routing --------------------------------------------------------
def do_HEAD(self):
self._route(True)
def do_GET(self):
self._route(False)
def _route(self, head_only):
rng = self.headers.get("Range", "-")
ua = (self.headers.get("User-Agent") or "-")[:60]
access(f"{self.command} {self.path} range={rng} ua={ua!r}")
if self.path == "/healthz":
body = json.dumps(mgr.status(), indent=2).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
if not head_only:
self.wfile.write(body)
return
if not self.path.startswith("/watch/"):
self._fail(404, "not found")
return
video_id = self.path[len("/watch/"):].split("?")[0]
if not VIDEO_ID_RE.match(video_id):
self._fail(400, "bad video id")
return
sess, refusal = mgr.get(video_id)
if refusal:
access(f" -> 503 {refusal}")
self._fail(503, refusal, retry_after=30)
return
if not mgr.growing:
# Correct duration and working seeks require a finished file.
if not sess.final.is_set():
access(f" -> waiting for {video_id} to finish muxing")
if not sess.final.wait(wait_timeout):
self._fail(504, "mux did not finish in time")
return
if sess.failed:
access(f" -> 502 producer failed: {sess.failed[:120]}")
self._fail(502, f"producer failed: {sess.failed[:200]}")
return
else:
# Growing mode still needs the first bytes to exist. Retries
# happen underneath while size is still 0, so wait on both.
deadline = time.monotonic() + STALL_TIMEOUT
while sess.size() == 0 and not sess.final.is_set():
if time.monotonic() > deadline:
self._fail(504, "producer wrote nothing")
return
time.sleep(0.1)
if sess.failed and sess.size() == 0:
self._fail(502, f"producer failed: {sess.failed[:200]}")
return
with mgr.lock:
sess.readers += 1
sess.last_used = time.time()
try:
self._serve(sess, head_only)
finally:
with mgr.lock:
sess.readers -= 1
sess.last_used = time.time()
def _fail(self, code, msg, retry_after=None):
body = (msg + "\n").encode()
try:
self.send_response(code)
if retry_after:
self.send_header("Retry-After", str(retry_after))
self.send_header("Content-Type", "text/plain")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
except (BrokenPipeError, ConnectionResetError):
pass
# -- body -----------------------------------------------------------
def _serve(self, sess, head_only):
path = sess.out_path
complete = lambda: sess.complete # noqa: E731
# Ranges are only honoured once the file is complete; while it is
# still growing there is no reliable time-to-byte mapping into a
# fragmented MP4, so we present a non-seekable stream instead.
chunked = not complete()
start, end, is_range = 0, None, False
if not chunked:
spec = self._parse_range(self.headers.get("Range"),
self._size(path))
if spec == "unsatisfiable":
total = self._size(path)
self.send_response(416)
self.send_header("Content-Range", f"bytes */{total}")
self.send_header("Content-Length", "0")
self.end_headers()
return
if spec:
start, end = spec
is_range = True
try:
self._wait_for_bytes(path, start, complete)
except TimeoutError:
self._fail(504, "producer stalled")
return
if chunked:
self.send_response(200)
self.send_header("Transfer-Encoding", "chunked")
else:
total = self._size(path)
if end is None:
end = total - 1
if is_range:
self.send_response(206)
self.send_header("Content-Range", f"bytes {start}-{end}/{total}")
else:
self.send_response(200)
self.send_header("Content-Length", str(end - start + 1))
self.send_header("Accept-Ranges", "bytes")
self.send_header("Content-Type", "video/mp4")
self.end_headers()
if head_only:
return
sent = start
# None while chunked: the growing case has no known end and stops
# when the producer does. A bounded range must send exactly the
# bytes it promised in Content-Length, no more.
remaining = None if end is None else end - start + 1
try:
with open(path, "rb") as f:
f.seek(start)
while True:
if remaining is not None and remaining <= 0:
break
buf = f.read(65536 if remaining is None
else min(65536, remaining))
if buf:
if chunked:
self.wfile.write(b"%X\r\n" % len(buf))
self.wfile.write(buf)
self.wfile.write(b"\r\n")
else:
self.wfile.write(buf)
sent += len(buf)
if remaining is not None:
remaining -= len(buf)
continue
if complete() and sent >= self._size(path):
break
try:
self._wait_for_bytes(path, sent, complete)
except TimeoutError:
break
if chunked:
self.wfile.write(b"0\r\n\r\n")
except (BrokenPipeError, ConnectionResetError):
# Normal: a probe reads the header then hangs up.
pass
return Handler
# --------------------------------------------------------------------------
def main():
ap = argparse.ArgumentParser(
description="just-in-time YouTube streaming proxy for Jellyfin")
ap.add_argument("--host", default="127.0.0.1")
ap.add_argument("--port", type=int, default=8099)
ap.add_argument("--work", default="/dev/shm/ytstream")
ap.add_argument("--cache-gb", type=float, default=8.0)
ap.add_argument("--max-pipelines", type=int, default=2)
ap.add_argument("--wait-timeout", type=float, default=300.0,
help="how long a request may block waiting for the mux")
ap.add_argument("--max-starts", type=int, default=20,
help="max cold starts per window; bounds a runaway library "
"refresh (default 20)")
ap.add_argument("--starts-window", type=float, default=3600.0,
help="rate-limit window in seconds (default 3600)")
ap.add_argument("--max-retries", type=int, default=2,
help="retries after a failed pipeline, each with a fresh "
"extraction (intermittent 403s do happen)")
ap.add_argument("--no-fetch", action="store_true",
help="never start a pipeline; log and 503. Use for a first "
"library scan to detect probing with no YouTube traffic")
ap.add_argument("--growing", action="store_true",
help="serve while still writing (low TTFB, wrong duration)")
ap.add_argument("--access-log", default=None)
args = ap.parse_args()
if not shutil.which("yt-dlp") or not shutil.which("ffmpeg"):
sys.exit("need yt-dlp and ffmpeg on PATH")
ver = subprocess.run(["yt-dlp", "--version"], capture_output=True, text=True)
version = ver.stdout.strip()
if version < "2025":
log(f"WARNING: yt-dlp {version} looks far too old, and a POT plugin is "
f"required. Expected the automation venv "
f"(/var/lib/youtube-automate/venv/bin) on PATH.")
else:
log(f"yt-dlp {version}")
global _access_log
_access_log = args.access_log
os.makedirs(args.work, exist_ok=True)
mgr = Manager(args.work, args.max_pipelines,
int(args.cache_gb * 2**30), args.no_fetch, args.growing,
args.max_retries, args.max_starts, args.starts_window)
if args.no_fetch:
log("NO-FETCH MODE: every /watch request will be logged and refused. "
"No YouTube traffic will be generated.")
if args.growing:
log("growing mode: probes will see a partial duration")
srv = ThreadingHTTPServer((args.host, args.port),
make_handler(mgr, args.wait_timeout))
log(f"listening on http://{args.host}:{args.port} "
f"(/watch/<video_id>, /healthz)")
try:
srv.serve_forever()
except KeyboardInterrupt:
log("shutting down")
if __name__ == "__main__":
main()
pyproject.toml
+33
View File
@@ -0,0 +1,33 @@
[project]
name = "ytstream"
version = "1.0.0"
description = "A just-in-time YouTube library for Jellyfin. Stores no video bytes."
requires-python = ">=3.11"
# Pins that matter, inherited from youtube-automate's specs.md §3:
# - the [default] extra ships yt-dlp-ejs, the JS challenge solver, which is
# mandatory. Only the proxy uses yt-dlp; cataloguing runs on the Data API.
# - curl-cffi must stay below 0.16 or yt-dlp rejects it as unsupported
# - the bgutil plugin must match the POT container tag
dependencies = [
"yt-dlp[default]",
"bgutil-ytdlp-pot-provider==1.3.1",
"curl-cffi<0.16",
]
[project.optional-dependencies]
dev = ["pytest>=8"]
[project.scripts]
ytstream = "ytstream.cli:main"
[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"
[tool.setuptools]
packages = ["ytstream", "ytstream.web"]
[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-q"
tests/conftest.py
+202
View File
@@ -0,0 +1,202 @@
"""Test fixtures.
Every path the application uses is redirected into a tmpdir. No test touches the
network, a real yt-dlp, a real Jellyfin, or the real media tree — the YouTube API
client is always a stub (see `FakeApi`), because the point of a test suite here is
to pin down behaviour that only shows up on the failure paths: a 403, an empty
response, a video that ages out and must not come back.
"""
from __future__ import annotations
import os
import tempfile
from pathlib import Path
# config resolves its paths at import time, so the environment has to be set
# before anything from the package is imported.
_SANDBOX = Path(tempfile.mkdtemp(prefix="yts-tests-"))
os.environ.setdefault("YTS_STATE_DIR", str(_SANDBOX / "state"))
os.environ.setdefault("YTS_MEDIA_ROOT", str(_SANDBOX / "media"))
os.environ.setdefault("YTS_DB_PATH", str(_SANDBOX / "state" / "ytstream.db"))
os.environ.setdefault("YTS_LOCK_PATH", str(_SANDBOX / "state" / "run.lock"))
os.environ.setdefault("YTS_VENV_BIN", str(_SANDBOX / "venv" / "bin"))
import pytest # noqa: E402
from ytstream import api, config, db, util, videos # noqa: E402
from ytstream.settings import Settings # noqa: E402
FIXTURES = Path(__file__).parent / "fixtures"
# A real-looking channel id: UC + 22 chars.
CHANNEL_ID = "UCW7jUEpYT_t0Gsf632d6_wQ"
@pytest.fixture()
def media_root(tmp_path, monkeypatch):
"""Point the media root at a per-test tmpdir."""
root = tmp_path / "media"
root.mkdir(parents=True)
monkeypatch.setattr(config, "MEDIA_ROOT", root)
return root
@pytest.fixture()
def conn(tmp_path):
connection = db.connect(tmp_path / "ytstream.db")
yield connection
connection.close()
@pytest.fixture()
def settings(conn):
settings = Settings(conn)
# Tests that reach the API go through FakeApi, but the code refuses to call
# out at all without a key, so give it one that is never used for real.
settings.set("youtube_api_key", "test-key")
return settings
@pytest.fixture()
def channel(conn):
"""One subscribed channel, returned as a row."""
return add_channel(conn, CHANNEL_ID, "clabretro", "clabretro")
@pytest.fixture()
def no_network(monkeypatch):
"""Fail loudly if anything tries to open a socket.
Belt and braces: a test that accidentally hits the network would pass locally
and fail in a different week for reasons nobody could reproduce.
"""
import urllib.request
def forbidden(*args, **kwargs):
raise AssertionError("test attempted a network call")
monkeypatch.setattr(urllib.request, "urlopen", forbidden)
# --------------------------------------------------------------------- helpers
def add_channel(conn, channel_id: str, title: str, dir_name: str, **kwargs):
fields = {"source": "youtube", "backfilled": 1, "uploads_playlist": "UULF"}
fields.update(kwargs)
columns = ", ".join(fields)
marks = ", ".join("?" * len(fields))
with conn:
conn.execute(
f"INSERT INTO channel (channel_id, handle, title, description, dir_name, "
f"added_at, {columns}) VALUES (?, ?, ?, ?, ?, ?, {marks})",
(channel_id, f"@{dir_name}", title, "A channel", dir_name,
util.utcnow_iso(), *fields.values()),
)
return conn.execute(
"SELECT * FROM channel WHERE channel_id = ?", (channel_id,)
).fetchone()
def add_video(conn, channel_pk, video_id, **kwargs):
"""Insert a video row with sensible defaults."""
defaults = {
"title": f"Video {video_id}",
"upload_date": "2026-08-01",
"state": videos.LISTED,
"discovery_source": videos.SOURCE_UULF,
"duration": 900,
}
defaults.update(kwargs)
videos.insert(conn, channel_pk=channel_pk, video_id=video_id, **defaults)
return videos.get(conn, video_id)
def feed_bytes(name: str) -> bytes:
return (FIXTURES / name).read_bytes()
def make_feed(entries: list[dict], *, playlist_published: str = "2019-01-01T00:00:00+00:00") -> bytes:
"""Build an Atom feed shaped like YouTube's.
Includes the feed-level <published> that is NOT an entry, because scraping
timestamps instead of walking atom:entry picks it up and yields nonsense
upload rates — a mistake made once while measuring.
"""
items = "".join(
f"""
<entry>
<id>yt:video:{e['video_id']}</id>
<yt:videoId>{e['video_id']}</yt:videoId>
<title>{e.get('title', 'Untitled')}</title>
<published>{e['published']}</published>
</entry>"""
for e in entries
)
return f"""<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns:yt="http://www.youtube.com/xml/schemas/2015"
xmlns:media="http://search.yahoo.com/mrss/"
xmlns="http://www.w3.org/2005/Atom">
<title>Videos</title>
<published>{playlist_published}</published>{items}
</feed>""".encode()
class FakeApi:
"""Stand-in for `api.Api` with scriptable responses.
Counts calls so tests can assert the batching actually batches — the whole
quota argument in the plan rests on 50 ids per call, and a regression to one
call per video would be silent and expensive.
"""
def __init__(self, *, subs=None, uploads=None, durations=None, channel=None,
raises=None):
self._subs = subs
self._uploads = uploads or []
self._durations = durations or {}
self._channel = channel
self._raises = raises
self.calls = 0
self.duration_calls = 0
self.subscription_calls = 0
def subscriptions(self, channel_id):
self.subscription_calls += 1
self.calls += 1
if isinstance(self._raises, Exception):
raise self._raises
return list(self._subs or [])
def uploads(self, channel_id, *, kind="UULF", since=None, limit=None,
page_token=None):
self.calls += 1
produced = 0
for entry, token in self._uploads:
if since is not None and entry["published"] < since:
return
yield entry, token
produced += 1
if limit is not None and produced >= limit:
return
def durations(self, video_ids):
self.duration_calls += 1
self.calls += 1
return {vid: self._durations[vid] for vid in video_ids
if vid in self._durations}
def channel(self, channel_id):
self.calls += 1
return self._channel
def resolve_handle(self, handle):
self.calls += 1
return self._channel
def patch_api(monkeypatch, module, fake: FakeApi):
"""Make `module.api.Api(...)` return `fake` regardless of arguments."""
monkeypatch.setattr(module.api, "Api", lambda *a, **kw: fake)
return fake
tests/fixtures/uulf_feed.xml
Vendored
+37
View File
@@ -0,0 +1,37 @@
<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns:yt="http://www.youtube.com/xml/schemas/2015"
xmlns:media="http://search.yahoo.com/mrss/"
xmlns="http://www.w3.org/2005/Atom">
<id>yt:playlist:UULFW7jUEpYT_t0Gsf632d6_wQ</id>
<title>Uploads from clabretro</title>
<entry>
<id>yt:video:08Ajr5fP52I</id>
<yt:videoId>08Ajr5fP52I</yt:videoId>
<yt:channelId>UCW7jUEpYT_t0Gsf632d6_wQ</yt:channelId>
<title>Learning to Design 3D Prints</title>
<published>2026-08-07T15:00:11+00:00</published>
<media:group>
<media:description>Tinkercad &amp; a cheap printer. Part 1/3 &lt;of a series&gt;.</media:description>
</media:group>
</entry>
<entry>
<id>yt:video:8k8nAQq0s_s</id>
<yt:videoId>8k8nAQq0s_s</yt:videoId>
<yt:channelId>UCW7jUEpYT_t0Gsf632d6_wQ</yt:channelId>
<title>Trying to use a Nortel PBX: part two</title>
<published>2026-08-02T14:30:00+00:00</published>
<media:group>
<media:description>Telephony experiments.</media:description>
</media:group>
</entry>
<entry>
<id>yt:video:vcYYcQyecNQ</id>
<yt:videoId>vcYYcQyecNQ</yt:videoId>
<yt:channelId>UCW7jUEpYT_t0Gsf632d6_wQ</yt:channelId>
<title>IBM Director on an xSeries 346 from 2004</title>
<published>2026-06-17T12:00:00+00:00</published>
<media:group>
<media:description>Old enterprise management software.</media:description>
</media:group>
</entry>
</feed>
tests/test_api.py
+329
View File
@@ -0,0 +1,329 @@
"""The YouTube Data API client, with no YouTube involved.
Two things here earn their keep beyond ordinary coverage:
* **Error classification.** Three distinct operator mistakes all arrive as HTTP
403 `forbidden`, and the only reliable signal is `error.details[].reason`. That
cost three attempts to diagnose by hand once; it should cost nobody a second one.
* **`totalResults` is not the list length.** It reported 127 against 119 actually
returned, because terminated and private channels still count as subscriptions.
"""
from __future__ import annotations
import io
import json
import urllib.error
import pytest
from ytstream import api
# ------------------------------------------------------------------- durations
@pytest.mark.parametrize("text,expected", [
("PT1H2M3S", 3723),
("PT21M", 1260),
("PT45S", 45),
("P1DT2H", 93600),
("P1W", 604800),
("PT0S", 0), # live and upcoming videos report this
("", None),
("garbage", None),
("1H2M", None),
])
def test_parse_duration(text, expected):
assert api.parse_duration(text) == expected
def test_parse_published_keeps_the_exact_string():
"""The date drives naming; the exact stamp is kept because approximate_date
is wrong by up to two days and episode numbers cannot be re-derived later."""
day, exact = api.parse_published("2026-08-11T16:32:10Z")
assert day.isoformat() == "2026-08-11"
assert exact == "2026-08-11T16:32:10Z"
def test_parse_published_normalises_to_utc():
day, _ = api.parse_published("2026-08-11T23:30:00-08:00")
assert day.isoformat() == "2026-08-12"
def test_parse_published_tolerates_rubbish():
assert api.parse_published("") == (None, None)
assert api.parse_published("not a date")[0] is None
# ------------------------------------------------------------------- playlists
def test_uploads_playlist_id():
assert api.uploads_playlist_id("UCjCJ2LaOIsPzOoXUTMDI3wg") == "UULFjCJ2LaOIsPzOoXUTMDI3wg"
assert api.uploads_playlist_id("UCjCJ2LaOIsPzOoXUTMDI3wg", "UU") == "UUjCJ2LaOIsPzOoXUTMDI3wg"
def test_uploads_playlist_id_rejects_non_channel_ids():
with pytest.raises(ValueError):
api.uploads_playlist_id("PLnotachannel")
# -------------------------------------------------------- error classification
def _http_error(code, body):
return urllib.error.HTTPError(
"https://example/", code, "Forbidden", {},
io.BytesIO(json.dumps(body).encode()),
)
def _classify(code, body):
return api.Api("k")._classify(_http_error(code, body))
def test_service_disabled_is_not_configured():
"""The API is not enabled on the project. Carries an activation URL."""
error = _classify(403, {"error": {
"code": 403,
"message": "YouTube Data API v3 has not been used in project 510818173753 "
"before or it is disabled.",
"errors": [{"reason": "accessNotConfigured"}],
"details": [{"reason": "SERVICE_DISABLED",
"metadata": {"activationUrl": "https://console/enable"}}],
}})
assert isinstance(error, api.NotConfigured)
def test_key_service_blocked_is_not_configured():
"""A different mistake entirely — the key's own API restrictions — and it
reports only `forbidden` in errors[]."""
error = _classify(403, {"error": {
"code": 403,
"message": "Requests to this API youtube method "
"youtube.api.v3.V3DataVideoService.List are blocked.",
"errors": [{"reason": "forbidden"}],
"details": [{"reason": "API_KEY_SERVICE_BLOCKED"}],
}})
assert isinstance(error, api.NotConfigured)
def test_subscription_forbidden_is_its_own_type():
"""Must never be mistaken for an empty subscription list."""
error = _classify(403, {"error": {
"code": 403,
"message": "The requester is not allowed to access the requested subscriptions.",
"errors": [{"reason": "subscriptionForbidden"}],
}})
assert isinstance(error, api.SubscriptionsPrivate)
assert not isinstance(error, api.NotConfigured)
def test_transient_server_error_is_plain_api_error():
error = _classify(500, {"error": {
"code": 500, "message": "Backend Error",
"errors": [{"reason": "backendError"}],
}})
assert type(error) is api.ApiError
def test_quota_exceeded_is_plain_api_error():
"""Retrying tomorrow helps, so it must not be classed as misconfiguration."""
error = _classify(403, {"error": {
"code": 403, "message": "The request cannot be completed because you have "
"exceeded your quota.",
"errors": [{"reason": "quotaExceeded"}],
}})
assert type(error) is api.ApiError
def test_unparseable_body_still_yields_an_error():
exc = urllib.error.HTTPError(
"https://example/", 502, "Bad Gateway", {}, io.BytesIO(b"<html>nope</html>")
)
error = api.Api("k")._classify(exc)
assert isinstance(error, api.ApiError)
assert error.status == 502
def test_missing_key_raises_before_any_request(monkeypatch):
import urllib.request
monkeypatch.setattr(urllib.request, "urlopen", lambda *a, **k:
(_ for _ in ()).throw(AssertionError("should not be called")))
with pytest.raises(api.NotConfigured):
api.Api("").durations(["dQw4w9WgXcQ"])
# -------------------------------------------------------------- request shapes
class Recorder:
"""Captures the URLs requested and replays canned pages."""
def __init__(self, pages):
self.pages = list(pages)
self.urls = []
def __call__(self, request, timeout=None):
self.urls.append(request.full_url)
payload = json.dumps(self.pages.pop(0)).encode()
class Response(io.BytesIO):
def __enter__(self_inner):
return self_inner
def __exit__(self_inner, *exc):
return False
return Response(payload)
@pytest.fixture()
def recorder(monkeypatch):
def install(pages):
rec = Recorder(pages)
monkeypatch.setattr(api.urllib.request, "urlopen", rec)
return rec
return install
def test_subscriptions_pages_to_exhaustion(recorder):
rec = recorder([
{"pageInfo": {"totalResults": 127},
"items": [{"snippet": {"title": f"C{i}",
"resourceId": {"channelId": f"UC{i:022d}"}}}
for i in range(50)],
"nextPageToken": "T2"},
{"pageInfo": {"totalResults": 127},
"items": [{"snippet": {"title": "Last",
"resourceId": {"channelId": "UC" + "z" * 22}}}]},
])
found = api.Api("k").subscriptions("UCbrother")
assert len(found) == 51
assert len(rec.urls) == 2
assert "pageToken=T2" in rec.urls[1]
def test_subscriptions_returns_the_fetched_list_not_total_results(recorder):
"""Measured: totalResults said 127, pagination returned 119."""
recorder([{"pageInfo": {"totalResults": 127},
"items": [{"snippet": {"title": "One",
"resourceId": {"channelId": "UC" + "a" * 22}}}]}])
found = api.Api("k").subscriptions("UCbrother")
assert len(found) == 1
def test_subscriptions_deduplicates(recorder):
recorder([{"items": [
{"snippet": {"title": "Dup", "resourceId": {"channelId": "UC" + "a" * 22}}},
{"snippet": {"title": "Dup", "resourceId": {"channelId": "UC" + "a" * 22}}},
]}])
assert len(api.Api("k").subscriptions("UCbrother")) == 1
def test_subscriptions_skips_entries_without_a_channel_id(recorder):
recorder([{"items": [
{"snippet": {"title": "Broken", "resourceId": {}}},
{"snippet": {"title": "Fine", "resourceId": {"channelId": "UC" + "a" * 22}}},
]}])
assert [entry["title"] for entry in api.Api("k").subscriptions("UCb")] == ["Fine"]
def test_durations_batches_fifty_ids_per_call(recorder):
"""The whole quota argument rests on this. One call per video would be a
silent 50x regression."""
ids = [f"vid{i:08d}" for i in range(120)]
rec = recorder([
{"items": [{"id": vid, "contentDetails": {"duration": "PT10M"}}
for vid in ids[start:start + 50]]}
for start in (0, 50, 100)
])
found = api.Api("k").durations(ids)
assert len(found) == 120
assert len(rec.urls) == 3
def test_durations_flags_livestreams(recorder):
recorder([{"items": [
{"id": "live0000001", "contentDetails": {"duration": "PT0S"},
"liveStreamingDetails": {"actualStartTime": "2026-08-01T00:00:00Z"}},
{"id": "vod00000001", "contentDetails": {"duration": "PT30M"}},
]}])
found = api.Api("k").durations(["live0000001", "vod00000001"])
assert found["live0000001"]["is_live"] is True
assert found["vod00000001"]["is_live"] is False
def test_uploads_stops_at_the_since_date(recorder):
recorder([{"items": [
{"contentDetails": {"videoId": "new00000001",
"videoPublishedAt": "2026-08-10T00:00:00Z"}},
{"contentDetails": {"videoId": "old00000001",
"videoPublishedAt": "2026-01-01T00:00:00Z"}},
{"contentDetails": {"videoId": "old00000002",
"videoPublishedAt": "2025-01-01T00:00:00Z"}},
], "nextPageToken": "T2"}])
from datetime import date
got = list(api.Api("k").uploads("UC" + "a" * 22, since=date(2026, 8, 1)))
assert [entry["video_id"] for entry, _ in got] == ["new00000001"]
def test_uploads_honours_the_limit(recorder):
recorder([{"items": [
{"contentDetails": {"videoId": f"vid{i:08d}",
"videoPublishedAt": "2026-08-10T00:00:00Z"}}
for i in range(10)
]}])
got = list(api.Api("k").uploads("UC" + "a" * 22, limit=3))
assert len(got) == 3
def test_uploads_skips_entries_with_no_publish_date(recorder):
"""A private or deleted video keeps its playlist slot but loses its date.
Skipping is right; stopping would truncate the backfill at that point."""
recorder([{"items": [
{"contentDetails": {"videoId": "priv0000001"}},
{"contentDetails": {"videoId": "good0000001",
"videoPublishedAt": "2026-08-10T00:00:00Z"}},
]}])
got = list(api.Api("k").uploads("UC" + "a" * 22))
assert [entry["video_id"] for entry, _ in got] == ["good0000001"]
def test_uploads_yields_the_next_page_token_for_resumability(recorder):
recorder([{"items": [
{"contentDetails": {"videoId": "vid00000001",
"videoPublishedAt": "2026-08-10T00:00:00Z"}}
], "nextPageToken": "CARRY_ON"}])
# Take only the first entry: the generator would otherwise follow the token
# on to a page the recorder has not been given, which is correct paging
# behaviour and covered elsewhere.
_, token = next(api.Api("k").uploads("UC" + "a" * 22))
assert token == "CARRY_ON"
def test_call_counter_tracks_quota(recorder):
recorder([{"items": []}, {"items": []}])
client = api.Api("k")
client.durations(["a"])
client.durations(["b"])
assert client.calls == 2
tests/test_auth.py
+203
View File
@@ -0,0 +1,203 @@
"""Password hashing, session cookies, CSRF tokens and login throttling."""
import time
import pytest
from ytstream.web import auth
class TestPasswords:
def test_round_trip(self):
stored = auth.hash_password("correct horse battery staple")
assert auth.verify_password(stored, "correct horse battery staple")
def test_wrong_password_rejected(self):
stored = auth.hash_password("secret")
assert not auth.verify_password(stored, "Secret")
assert not auth.verify_password(stored, "")
def test_salt_makes_hashes_unique(self):
assert auth.hash_password("same") != auth.hash_password("same")
def test_hash_is_not_the_plaintext(self):
assert "secret" not in auth.hash_password("secret")
def test_empty_stored_hash_rejects_everything(self):
assert not auth.verify_password("", "anything")
def test_malformed_stored_hash_does_not_raise(self):
for junk in ("nonsense", "scrypt$bad", "a$b$c$d$e$f", "scrypt$x$y$z$q$r"):
assert auth.verify_password(junk, "anything") is False
def test_unicode_password(self):
stored = auth.hash_password("pässwörd🎬")
assert auth.verify_password(stored, "pässwörd🎬")
class TestSessions:
def test_issue_and_verify(self):
secret = auth.new_secret()
token = auth.issue_session(secret)
assert auth.verify_session(secret, token)
def test_a_different_secret_rejects(self):
token = auth.issue_session(auth.new_secret())
assert not auth.verify_session(auth.new_secret(), token)
def test_tampered_payload_rejected(self):
secret = auth.new_secret()
token = auth.issue_session(secret)
payload, signature = token.split(".", 1)
assert not auth.verify_session(secret, f"{payload}x.{signature}")
def test_tampered_signature_rejected(self):
secret = auth.new_secret()
payload, _ = auth.issue_session(secret).split(".", 1)
assert not auth.verify_session(secret, f"{payload}.deadbeef")
def test_garbage_rejected(self):
secret = auth.new_secret()
for junk in ("", "no-dot", "a.b.c", "...."):
assert auth.verify_session(secret, junk) is False
def test_expires_after_a_year(self):
secret = auth.new_secret()
issued = time.time() - auth.SESSION_MAX_AGE - 10
token = auth.issue_session(secret, issued_at=issued)
assert not auth.verify_session(secret, token)
def test_still_valid_just_inside_a_year(self):
secret = auth.new_secret()
issued = time.time() - auth.SESSION_MAX_AGE + 60
token = auth.issue_session(secret, issued_at=issued)
assert auth.verify_session(secret, token)
def test_a_token_from_the_future_is_rejected(self):
secret = auth.new_secret()
token = auth.issue_session(secret, issued_at=time.time() + 3600)
assert not auth.verify_session(secret, token)
class TestCookie:
def test_carries_the_hardening_flags(self):
header = auth.cookie_header("abc")
for flag in ("HttpOnly", "Secure", "SameSite=Lax", "Path=/"):
assert flag in header
assert f"Max-Age={auth.SESSION_MAX_AGE}" in header
def test_secure_can_be_omitted_for_local_http_testing(self):
assert "Secure" not in auth.cookie_header("abc", secure=False)
def test_clear_cookie_expires_immediately(self):
assert "Max-Age=0" in auth.clear_cookie_header()
class TestCookieParsing:
"""Regression tests for a real failure: http.cookies.SimpleCookie silently
drops everything after a value it dislikes, which made valid sessions
invisible and bounced users back to the login page with no error."""
def test_finds_our_cookie_alone(self):
assert auth.cookie_value("yta_session=abc") == "abc"
def test_finds_it_after_a_neighbour(self):
assert auth.cookie_value("sessionid=xyz; yta_session=abc") == "abc"
def test_finds_it_before_a_neighbour(self):
assert auth.cookie_value("yta_session=abc; sessionid=xyz") == "abc"
@pytest.mark.parametrize(
"neighbour",
[
'prefs={"a":1}', # JSON value — what actually broke it
"junk=[1,2,3]",
"weird=a b c",
"empty=",
"novalue",
"quoted=\"has spaces\"",
"path=/a/b/c",
"colons=a:b:c",
"comma=a,b",
],
)
def test_survives_hostile_neighbours(self, neighbour):
assert auth.cookie_value(f"{neighbour}; yta_session=abc") == "abc"
assert auth.cookie_value(f"yta_session=abc; {neighbour}") == "abc"
def test_strips_surrounding_quotes(self):
assert auth.cookie_value('yta_session="abc"') == "abc"
def test_tolerates_whitespace(self):
assert auth.cookie_value(" yta_session = abc ") == "abc"
def test_absent_cookie_returns_empty(self):
assert auth.cookie_value("sessionid=xyz") == ""
def test_empty_header_returns_empty(self):
assert auth.cookie_value("") == ""
assert auth.cookie_value(None) == ""
def test_does_not_match_a_name_that_merely_contains_ours(self):
assert auth.cookie_value("not_yta_session=nope") == ""
def test_real_token_round_trips_through_the_header(self):
secret = auth.new_secret()
token = auth.issue_session(secret)
header = f'prefs={{"theme":"dark"}}; yta_session={token}; other=1'
assert auth.verify_session(secret, auth.cookie_value(header))
class TestCsrf:
def test_token_verifies(self):
secret, session = auth.new_secret(), auth.issue_session(auth.new_secret())
token = auth.csrf_token(secret, session)
assert auth.verify_csrf(secret, session, token)
def test_token_is_bound_to_the_session(self):
secret = auth.new_secret()
one = auth.issue_session(secret, issued_at=1000)
two = auth.issue_session(secret, issued_at=2000)
assert not auth.verify_csrf(secret, two, auth.csrf_token(secret, one))
def test_empty_token_rejected(self):
secret, session = auth.new_secret(), "sess"
assert not auth.verify_csrf(secret, session, "")
def test_wrong_token_rejected(self):
secret, session = auth.new_secret(), "sess"
assert not auth.verify_csrf(secret, session, "deadbeef")
class TestThrottle:
def test_allows_up_to_the_limit(self):
throttle = auth.LoginThrottle(max_failures=3, lockout=60)
for _ in range(2):
throttle.record_failure("1.2.3.4", now=1000)
assert not throttle.locked("1.2.3.4", now=1000)
def test_locks_after_the_limit(self):
throttle = auth.LoginThrottle(max_failures=3, lockout=60)
for _ in range(3):
throttle.record_failure("1.2.3.4", now=1000)
assert throttle.locked("1.2.3.4", now=1000)
def test_lock_expires(self):
throttle = auth.LoginThrottle(max_failures=3, lockout=60)
for _ in range(3):
throttle.record_failure("1.2.3.4", now=1000)
assert not throttle.locked("1.2.3.4", now=1061)
def test_success_clears_the_counter(self):
throttle = auth.LoginThrottle(max_failures=3, lockout=60)
for _ in range(2):
throttle.record_failure("1.2.3.4", now=1000)
throttle.record_success("1.2.3.4")
assert not throttle.locked("1.2.3.4", now=1000)
def test_addresses_are_tracked_separately(self):
throttle = auth.LoginThrottle(max_failures=2, lockout=60)
for _ in range(2):
throttle.record_failure("1.1.1.1", now=1000)
assert throttle.locked("1.1.1.1", now=1000)
assert not throttle.locked("2.2.2.2", now=1000)
tests/test_channels.py
+151
View File
@@ -0,0 +1,151 @@
"""Channel subscribe/unsubscribe, and the API-only sync path."""
from __future__ import annotations
import pytest
from ytstream import api, channels, config
from conftest import CHANNEL_ID, FakeApi, add_channel, add_video
def test_normalise_url_accepts_every_reference_form():
assert channels.normalise_url(CHANNEL_ID).endswith("/channel/" + CHANNEL_ID)
assert channels.normalise_url("@clabretro") == "https://www.youtube.com/@clabretro"
assert channels.normalise_url("clabretro") == "https://www.youtube.com/@clabretro"
assert channels.normalise_url("https://youtube.com/x") == "https://youtube.com/x"
assert channels.normalise_url("youtube.com/x") == "https://youtube.com/x"
def test_normalise_url_rejects_nonsense():
with pytest.raises(channels.ResolutionError):
channels.normalise_url("")
with pytest.raises(channels.ResolutionError):
channels.normalise_url("not a handle!!")
def test_uulf_playlist_id():
assert channels.uulf_playlist_id(CHANNEL_ID) == "UULF" + CHANNEL_ID[2:]
# ---------------------------------------------------------- subscribe_from_sync
@pytest.fixture()
def api_channel(monkeypatch):
"""Patch channels' lazily-imported api module."""
fake = FakeApi(channel={"channel_id": "UC" + "a" * 22, "title": "Alpha Channel",
"description": "About alpha", "handle": "@alpha",
"avatar_url": None})
monkeypatch.setattr(api, "Api", lambda *a, **kw: fake)
return fake
def test_subscribe_from_sync_never_calls_yt_dlp(conn, settings, media_root,
api_channel, monkeypatch):
"""119 channels' worth of yt-dlp resolution is the request burst this design
exists to avoid."""
monkeypatch.setattr(channels.ytdlp, "run_json",
lambda *a, **k: pytest.fail("yt-dlp must not be called"))
row = channels.subscribe_from_sync(conn, settings, "UC" + "a" * 22, "Alpha")
assert row["title"] == "Alpha Channel"
assert row["source"] == "youtube"
def test_subscribe_from_sync_uses_the_api_description(conn, settings, media_root,
api_channel):
row = channels.subscribe_from_sync(conn, settings, "UC" + "a" * 22, "Alpha")
assert row["description"] == "About alpha"
def test_subscribe_from_sync_is_idempotent(conn, settings, media_root, api_channel):
first = channels.subscribe_from_sync(conn, settings, "UC" + "a" * 22, "Alpha")
second = channels.subscribe_from_sync(conn, settings, "UC" + "a" * 22, "Alpha")
assert first["id"] == second["id"]
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 1
def test_subscribe_from_sync_survives_a_metadata_failure(conn, settings, media_root,
monkeypatch):
"""A channel we cannot describe is still a channel we can mirror."""
class Failing(FakeApi):
def channel(self, channel_id):
raise api.ApiError(500, "backendError", "boom")
monkeypatch.setattr(api, "Api", lambda *a, **kw: Failing())
row = channels.subscribe_from_sync(conn, settings, "UC" + "a" * 22, "Fallback Name")
assert row["title"] == "Fallback Name"
def test_subscribe_from_sync_makes_no_directory(conn, settings, media_root,
monkeypatch):
"""The tree appears with the first episode, so a channel with nothing inside
the window leaves no empty series behind."""
class NoArt(FakeApi):
def channel(self, channel_id):
return {"channel_id": channel_id, "title": "Quiet", "description": "",
"handle": None, "avatar_url": None}
monkeypatch.setattr(api, "Api", lambda *a, **kw: NoArt())
row = channels.subscribe_from_sync(conn, settings, "UC" + "a" * 22, "Quiet")
assert not (media_root / row["dir_name"]).exists()
def test_directory_names_are_deduplicated(conn, settings, media_root, monkeypatch):
"""Two channels can legitimately share a title; dir_name is UNIQUE."""
class Same(FakeApi):
def channel(self, channel_id):
return {"channel_id": channel_id, "title": "Same Name",
"description": "", "handle": None, "avatar_url": None}
monkeypatch.setattr(api, "Api", lambda *a, **kw: Same())
first = channels.subscribe_from_sync(conn, settings, "UC" + "a" * 22, "Same Name")
second = channels.subscribe_from_sync(conn, settings, "UC" + "b" * 22, "Same Name")
assert first["dir_name"] == "Same Name"
assert second["dir_name"] == "Same Name (2)"
# ------------------------------------------------------------------ unsubscribe
def test_unsubscribe_removes_the_tree_and_the_rows(conn, media_root, channel):
tree = media_root / "clabretro"
(tree / "Season 2026").mkdir(parents=True)
(tree / "tvshow.nfo").write_text("<tvshow/>")
add_video(conn, channel["id"], "vid00000001")
title = channels.unsubscribe(conn, channel["id"])
assert title == "clabretro"
assert not tree.exists()
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 0
assert conn.execute("SELECT COUNT(*) FROM video").fetchone()[0] == 0
def test_unsubscribe_rejects_an_unknown_id(conn):
with pytest.raises(LookupError):
channels.unsubscribe(conn, 999)
def test_lookup_helpers(conn, channel):
assert channels.get(conn, channel["id"])["title"] == "clabretro"
assert channels.get_by_channel_id(conn, CHANNEL_ID)["id"] == channel["id"]
assert channels.get(conn, 999) is None
assert channels.get_by_channel_id(conn, "UCnope") is None
def test_all_channels_is_sorted_case_insensitively(conn):
add_channel(conn, "UC" + "a" * 22, "zebra", "zebra")
add_channel(conn, "UC" + "b" * 22, "Apple", "Apple")
titles = [row["title"] for row in channels.all_channels(conn)]
assert titles == ["Apple", "zebra"]
tests/test_discovery.py
+391
View File
@@ -0,0 +1,391 @@
"""Discovery: feed parsing, the API backfill, and duration enrichment."""
from __future__ import annotations
from datetime import date, timedelta
import pytest
from ytstream import api, discovery, util, videos
from conftest import FakeApi, add_video, make_feed, patch_api
def _entry(video_id, published, title="A video"):
return {"video_id": video_id, "published": published, "title": title}
@pytest.fixture()
def offline(monkeypatch):
"""No feed fetches and no API calls unless a test asks for them."""
monkeypatch.setattr(discovery, "fetch_feed",
lambda *a, **k: pytest.fail("unexpected feed fetch"))
patch_api(monkeypatch, discovery, FakeApi())
# --------------------------------------------------------------- feed parsing
def test_parse_entries_reads_only_atom_entries():
"""The feed-level <published> is the playlist's creation date, sometimes years
old. Treating it as a video produced a nonsense 0.01/day upload rate once."""
payload = make_feed(
[_entry("vid00000001", "2026-08-11T10:00:00+00:00")],
playlist_published="2019-03-01T00:00:00+00:00",
)
entries = discovery.parse_entries(payload)
assert len(entries) == 1
assert entries[0]["published"] == date(2026, 8, 11)
def test_parse_entries_keeps_the_exact_timestamp():
payload = make_feed([_entry("vid00000001", "2026-08-11T16:32:10+00:00")])
assert discovery.parse_entries(payload)[0]["published_at"].startswith(
"2026-08-11T16:32:10"
)
def test_parse_entries_skips_undated_entries():
payload = make_feed([_entry("vid00000001", "not-a-date"),
_entry("vid00000002", "2026-08-11T10:00:00+00:00")])
assert [e["video_id"] for e in discovery.parse_entries(payload)] == ["vid00000002"]
def test_parse_entries_rejects_garbage():
with pytest.raises(discovery.FeedUnavailable):
discovery.parse_entries(b"<not xml")
def test_empty_feed_is_not_an_error():
assert discovery.parse_entries(make_feed([])) == []
def test_feed_urls_use_the_long_form_playlist():
url = discovery.uulf_feed_url("UCjCJ2LaOIsPzOoXUTMDI3wg")
assert "playlist_id=UULFjCJ2LaOIsPzOoXUTMDI3wg" in url
assert "channel_id=UCjCJ2LaOIsPzOoXUTMDI3wg" in discovery.uc_feed_url(
"UCjCJ2LaOIsPzOoXUTMDI3wg"
)
# -------------------------------------------------------------------- polling
def _install_feed(monkeypatch, entries, *, uulf=True):
payload = make_feed(entries)
def fetch(url, timeout=30.0):
if "playlist_id=UULF" in url:
return payload if uulf else None
return payload if not uulf else None
monkeypatch.setattr(discovery, "fetch_feed", fetch)
def test_poll_queues_videos_inside_the_window(conn, settings, channel, monkeypatch):
today = util.today()
_install_feed(monkeypatch, [_entry("vid00000001", f"{today}T10:00:00+00:00")])
patch_api(monkeypatch, discovery,
FakeApi(durations={"vid00000001": {"duration": 900, "is_live": False}}))
stats = discovery.poll_channel(conn, settings, channel)
assert stats["queued"] == 1
assert stats["source"] == videos.SOURCE_UULF
row = videos.get(conn, "vid00000001")
assert row["state"] == videos.LISTED
assert row["duration"] == 900
def test_poll_marks_older_videos_skipped_old(conn, settings, channel, monkeypatch):
old = util.today() - timedelta(days=90)
_install_feed(monkeypatch, [_entry("vid00000001", f"{old}T10:00:00+00:00")])
patch_api(monkeypatch, discovery, FakeApi())
stats = discovery.poll_channel(conn, settings, channel)
assert stats["old"] == 1
assert videos.get(conn, "vid00000001")["state"] == videos.SKIPPED_OLD
def test_poll_falls_back_to_the_channel_feed(conn, settings, channel, monkeypatch):
today = util.today()
_install_feed(monkeypatch, [_entry("vid00000001", f"{today}T10:00:00+00:00")],
uulf=False)
patch_api(monkeypatch, discovery, FakeApi())
stats = discovery.poll_channel(conn, settings, channel)
assert stats["source"] == videos.SOURCE_UC
assert videos.get(conn, "vid00000001")["discovery_source"] == videos.SOURCE_UC
def test_poll_records_feed_failure_without_raising(conn, settings, channel, monkeypatch):
def boom(url, timeout=30.0):
raise discovery.FeedUnavailable("HTTP 404")
monkeypatch.setattr(discovery, "fetch_feed", boom)
stats = discovery.poll_channel(conn, settings, channel)
assert "error" in stats
row = conn.execute("SELECT * FROM channel WHERE id = ?", (channel["id"],)).fetchone()
assert row["last_poll_ok"] == 0
assert row["consecutive_poll_failures"] == 1
def test_two_of_119_channels_failing_is_survivable(conn, settings, channel, monkeypatch):
"""Measured: terminated channels stay in the subscription list and 404 here."""
monkeypatch.setattr(discovery, "fetch_feed",
lambda *a, **k: (_ for _ in ()).throw(
discovery.FeedUnavailable("HTTP 404")))
totals = discovery.poll_all(conn, settings)
assert totals["failed"] == 1
assert totals["channels"] == 1
def test_poll_fills_in_a_missing_title(conn, settings, channel, monkeypatch):
"""Backfill inserts rows with no title; the feed is where titles come from."""
today = util.today()
add_video(conn, channel["id"], "vid00000001", title="",
upload_date=today.isoformat())
_install_feed(monkeypatch,
[_entry("vid00000001", f"{today}T10:00:00+00:00", "Real Title")])
patch_api(monkeypatch, discovery, FakeApi())
stats = discovery.poll_channel(conn, settings, channel)
assert stats["titled"] == 1
assert videos.get(conn, "vid00000001")["title"] == "Real Title"
def test_uulf_repairs_a_video_the_fallback_called_short(
conn, settings, channel, monkeypatch
):
today = util.today()
add_video(conn, channel["id"], "vid00000001", state=videos.SKIPPED_SHORT,
discovery_source=videos.SOURCE_UC, upload_date=today.isoformat())
_install_feed(monkeypatch, [_entry("vid00000001", f"{today}T10:00:00+00:00")])
patch_api(monkeypatch, discovery, FakeApi())
stats = discovery.poll_channel(conn, settings, channel)
assert stats["repaired"] == 1
row = videos.get(conn, "vid00000001")
assert row["state"] == videos.LISTED
assert row["discovery_source"] == videos.SOURCE_UULF
# ---------------------------------------------------------------- enrichment
def test_enrich_filters_shorts(conn, settings, channel, monkeypatch):
"""Measured: 38 of 50 consecutive uploads on a real channel were <=120s."""
add_video(conn, channel["id"], "short000001", duration=None)
patch_api(monkeypatch, discovery,
FakeApi(durations={"short000001": {"duration": 45, "is_live": False}}))
stats = discovery.enrich_durations(conn, settings, ["short000001"])
assert stats["shorts"] == 1
assert videos.get(conn, "short000001")["state"] == videos.SKIPPED_SHORT
def test_enrich_filters_livestreams_regardless_of_duration(
conn, settings, channel, monkeypatch
):
"""Live and upcoming both report PT0S, so duration cannot be the signal."""
add_video(conn, channel["id"], "live0000001", duration=None)
patch_api(monkeypatch, discovery,
FakeApi(durations={"live0000001": {"duration": 0, "is_live": True}}))
stats = discovery.enrich_durations(conn, settings, ["live0000001"])
assert stats["live"] == 1
assert videos.get(conn, "live0000001")["state"] == videos.SKIPPED_LIVE
def test_enrich_keeps_long_videos(conn, settings, channel, monkeypatch):
add_video(conn, channel["id"], "long0000001", duration=None)
patch_api(monkeypatch, discovery,
FakeApi(durations={"long0000001": {"duration": 2790, "is_live": False}}))
discovery.enrich_durations(conn, settings, ["long0000001"])
row = videos.get(conn, "long0000001")
assert row["state"] == videos.LISTED
assert row["duration"] == 2790
def test_enrich_survives_an_api_failure(conn, settings, channel, monkeypatch):
"""A NULL duration costs a runtime display, not a working library."""
add_video(conn, channel["id"], "vid00000001", duration=None)
class Failing(FakeApi):
def durations(self, ids):
raise api.ApiError(500, "backendError", "boom")
patch_api(monkeypatch, discovery, Failing())
stats = discovery.enrich_durations(conn, settings, ["vid00000001"])
assert stats["resolved"] == 0
assert videos.get(conn, "vid00000001")["state"] == videos.LISTED
def test_enrich_with_no_ids_makes_no_call(conn, settings, monkeypatch):
fake = patch_api(monkeypatch, discovery, FakeApi())
discovery.enrich_durations(conn, settings, [])
assert fake.calls == 0
# ------------------------------------------------------------------- backfill
def test_backfill_queues_the_window(conn, settings, channel, monkeypatch):
today = util.today()
uploads = [
({"video_id": f"vid{i:08d}", "published": today - timedelta(days=i),
"published_at": f"{today - timedelta(days=i)}T00:00:00Z"}, None)
for i in range(3)
]
patch_api(monkeypatch, discovery, FakeApi(
uploads=uploads,
durations={f"vid{i:08d}": {"duration": 900, "is_live": False}
for i in range(3)}))
stats = discovery.backfill_channel(conn, settings, channel)
assert stats["queued"] == 3
assert videos.get(conn, "vid00000000")["state"] == videos.LISTED
assert conn.execute("SELECT backfilled FROM channel WHERE id = ?",
(channel["id"],)).fetchone()[0] == 1
def test_backfill_stores_the_exact_publish_time(conn, settings, channel, monkeypatch):
today = util.today()
patch_api(monkeypatch, discovery, FakeApi(uploads=[
({"video_id": "vid00000001", "published": today,
"published_at": "2026-08-11T16:32:10Z"}, None)]))
discovery.backfill_channel(conn, settings, channel)
assert videos.get(conn, "vid00000001")["published_at"] == "2026-08-11T16:32:10Z"
def test_backfill_respects_the_video_cap(conn, settings, channel, monkeypatch):
settings.set("backfill_max_videos", "2")
today = util.today()
patch_api(monkeypatch, discovery, FakeApi(uploads=[
({"video_id": f"vid{i:08d}", "published": today, "published_at": None}, None)
for i in range(10)]))
stats = discovery.backfill_channel(conn, settings, channel)
assert stats["queued"] == 2
def test_backfill_clears_the_cursor_when_complete(conn, settings, channel, monkeypatch):
patch_api(monkeypatch, discovery, FakeApi(uploads=[
({"video_id": "vid00000001", "published": util.today(),
"published_at": None}, "TOKEN")]))
discovery.backfill_channel(conn, settings, channel)
assert conn.execute("SELECT backfill_cursor FROM channel WHERE id = ?",
(channel["id"],)).fetchone()[0] is None
def test_backfill_leaves_the_flag_unset_when_the_api_is_unusable(
conn, settings, channel, monkeypatch
):
"""So it retries once a key is configured, rather than silently never running."""
with conn:
conn.execute("UPDATE channel SET backfilled = 0 WHERE id = ?", (channel["id"],))
class Failing(FakeApi):
def uploads(self, *a, **kw):
raise api.NotConfigured(403, "forbidden", "blocked")
yield # pragma: no cover
patch_api(monkeypatch, discovery, Failing())
stats = discovery.backfill_channel(conn, settings, channel)
assert "error" in stats
assert conn.execute("SELECT backfilled FROM channel WHERE id = ?",
(channel["id"],)).fetchone()[0] == 0
def test_backfill_does_not_duplicate_known_videos(conn, settings, channel, monkeypatch):
add_video(conn, channel["id"], "vid00000001")
patch_api(monkeypatch, discovery, FakeApi(uploads=[
({"video_id": "vid00000001", "published": util.today(),
"published_at": None}, None)]))
stats = discovery.backfill_channel(conn, settings, channel)
assert stats["queued"] == 0
assert conn.execute("SELECT COUNT(*) FROM video").fetchone()[0] == 1
# ------------------------------------------------------- titles (regression)
def test_backfill_takes_the_title_from_the_api(conn, settings, channel, monkeypatch):
"""Not an optimisation. RSS returns 15 entries, which for a channel posting
under one long-form video a day reaches back only ~23 days against a 30-day
window — so the oldest ~5 of every 20-episode backfill was being named after
its video id."""
patch_api(monkeypatch, discovery, FakeApi(uploads=[
({"video_id": "vid00000001", "published": util.today(),
"published_at": None, "title": "A Real Title"}, None)]))
discovery.backfill_channel(conn, settings, channel)
assert videos.get(conn, "vid00000001")["title"] == "A Real Title"
def test_backfill_tolerates_a_missing_title(conn, settings, channel, monkeypatch):
patch_api(monkeypatch, discovery, FakeApi(uploads=[
({"video_id": "vid00000001", "published": util.today(),
"published_at": None}, None)]))
discovery.backfill_channel(conn, settings, channel)
assert videos.get(conn, "vid00000001")["title"] == ""
def test_a_late_title_renames_an_already_materialised_episode(
conn, settings, media_root, channel, monkeypatch
):
"""The repair has to move the file, not just the row — otherwise the episode
keeps its video-id filename forever."""
from ytstream import strm
monkeypatch.setattr(strm, "fetch_thumbnail", lambda *a, **k: False)
today = util.today()
add_video(conn, channel["id"], "vid00000001", title="",
upload_date=today.isoformat())
first = strm.materialise(conn, settings, channel,
videos.get(conn, "vid00000001"))
assert "vid00000001]" in first["rel_path"]
old_path = media_root / first["rel_path"]
assert old_path.exists()
outcome = discovery._record(
conn, channel,
{"video_id": "vid00000001", "title": "Proper Name", "published": today,
"published_at": None},
videos.SOURCE_UULF, today - timedelta(days=30),
)
assert outcome == "titled"
assert videos.get(conn, "vid00000001")["state"] == videos.LISTED
assert not old_path.exists()
second = strm.materialise(conn, settings, channel,
videos.get(conn, "vid00000001"))
assert "Proper Name" in second["rel_path"]
tests/test_naming.py
+128
View File
@@ -0,0 +1,128 @@
from datetime import date
import pytest
from ytstream import naming
class TestSanitise:
@pytest.mark.parametrize(
"raw, expected",
[
("Hermitcraft S11#11: Expanding Business", "Hermitcraft S11#11 Expanding Business"),
("A/B", "A B"),
('Say "hello" <now>', "Say hello now"),
("path\\to\\thing", "path to thing"),
("what? really * | yes", "what really yes"),
(" .leading and trailing. ", "leading and trailing"),
("line one\nline two", "line one line two"),
("tabs\tand\r\nnewlines", "tabs and newlines"),
],
)
def test_removes_illegal_characters(self, raw, expected):
assert naming.sanitize_component(raw) == expected
def test_empty_input_is_empty(self):
assert naming.sanitize_component("") == ""
assert naming.sanitize_component(None) == ""
def test_title_that_is_only_illegal_characters_collapses_to_empty(self):
assert naming.sanitize_component("///???") == ""
def test_truncates_to_120_characters(self):
long = "word " * 60
result = naming.sanitize_component(long)
assert len(result) <= naming.MAX_TITLE_LEN
def test_truncation_prefers_a_word_boundary(self):
text = "alpha bravo charlie delta echo foxtrot golf hotel india juliet " * 3
result = naming.sanitize_component(text)
assert not result.endswith(" ")
# Should not cut mid-word when a boundary is available late enough.
assert result == result.rstrip()
assert " " in result
def test_hard_cuts_when_no_late_word_boundary_exists(self):
text = "a" + "b" * 400
result = naming.sanitize_component(text)
assert len(result) == naming.MAX_TITLE_LEN
class TestEpisodeNumbering:
@pytest.mark.parametrize(
"day, ordinal, expected",
[
(date(2026, 8, 11), 0, 8110),
(date(2026, 8, 11), 1, 8111),
(date(2026, 8, 12), 0, 8120),
(date(2026, 1, 1), 0, 1010),
(date(2026, 12, 31), 0, 12310),
(date(2026, 12, 31), 9, 12319),
],
)
def test_episode_number(self, day, ordinal, expected):
assert naming.episode_number(day, ordinal) == expected
def test_ordinal_is_clamped_at_nine(self):
assert naming.episode_number(date(2026, 8, 11), 12) == 8119
def test_negative_ordinal_clamps_to_zero(self):
assert naming.episode_number(date(2026, 8, 11), -3) == 8110
def test_numbers_sort_chronologically_across_the_year(self):
days = [date(2026, 1, 1), date(2026, 6, 15), date(2026, 8, 11), date(2026, 12, 31)]
numbers = [naming.episode_number(day, 0) for day in days]
assert numbers == sorted(numbers)
def test_episode_range_covers_ten_slots(self):
low, high = naming.episode_range(date(2026, 8, 11))
assert (low, high) == (8110, 8119)
def test_season_is_the_upload_year(self):
assert naming.season_for(date(2026, 8, 11)) == 2026
class TestParseUploadDate:
def test_accepts_ytdlp_compact_form(self):
assert naming.parse_upload_date("20260811") == date(2026, 8, 11)
def test_accepts_iso_form(self):
assert naming.parse_upload_date("2026-08-11") == date(2026, 8, 11)
def test_accepts_iso_timestamp(self):
assert naming.parse_upload_date("2026-08-11T12:00:00+00:00") == date(2026, 8, 11)
def test_passes_through_a_date(self):
assert naming.parse_upload_date(date(2026, 8, 11)) == date(2026, 8, 11)
def test_rejects_nonsense(self):
with pytest.raises(ValueError):
naming.parse_upload_date("not a date")
class TestBasename:
def test_includes_video_id_for_uniqueness(self):
stem = naming.basename("clabretro", 2026, 8110, "A Title", "dQw4w9WgXcQ")
assert stem == "clabretro - S2026E8110 - A Title [dQw4w9WgXcQ]"
def test_episode_is_unpadded_to_match_the_nfo(self):
stem = naming.basename("c", 2026, 8110, "t", "id")
assert "S2026E8110" in stem
assert "E08110" not in stem
def test_falls_back_to_video_id_when_the_title_sanitises_away(self):
stem = naming.basename("c", 2026, 8110, "///", "dQw4w9WgXcQ")
assert stem.endswith("dQw4w9WgXcQ [dQw4w9WgXcQ]")
def test_two_videos_same_title_differ_by_id(self):
one = naming.basename("c", 2026, 8110, "Same", "aaaaaaaaaaa")
two = naming.basename("c", 2026, 8111, "Same", "bbbbbbbbbbb")
assert one != two
class TestChannelDirName:
def test_sanitises_the_title(self):
assert naming.channel_dir_name("Tom / Jerry", "UCabc") == "Tom Jerry"
def test_falls_back_to_channel_id_when_title_is_unusable(self):
assert naming.channel_dir_name("???", "UCabc") == "UCabc"
tests/test_nfo.py
+94
View File
@@ -0,0 +1,94 @@
import xml.etree.ElementTree as ET
from ytstream import nfo
HOSTILE = (
"Ampersands & angle <brackets> and \"quotes\"\n"
"control chars: \x00\x07\x1b\n"
"emoji 🎬 and em-dash — and links https://example.com/?a=1&b=2"
)
class TestEpisodeNfo:
def build(self, **overrides):
kwargs = dict(
title="Video Title",
show_title="Some Channel",
season=2026,
episode=8110,
plot="A plot.",
aired="2026-08-11",
duration_seconds=762,
video_id="dQw4w9WgXcQ",
)
kwargs.update(overrides)
return nfo.episode_nfo(**kwargs)
def test_is_well_formed_xml(self):
root = ET.fromstring(self.build())
assert root.tag == "episodedetails"
def test_hostile_description_still_parses(self):
root = ET.fromstring(self.build(plot=HOSTILE))
plot = root.findtext("plot")
assert "&" in plot and "<brackets>" in plot
assert "🎬" in plot
def test_control_characters_are_stripped(self):
plot = ET.fromstring(self.build(plot=HOSTILE)).findtext("plot")
for bad in ("\x00", "\x07", "\x1b"):
assert bad not in plot
def test_newlines_are_preserved(self):
plot = ET.fromstring(self.build(plot="one\ntwo")).findtext("plot")
assert plot == "one\ntwo"
def test_runtime_is_rounded_minutes(self):
assert ET.fromstring(self.build(duration_seconds=762)).findtext("runtime") == "13"
def test_short_video_still_gets_at_least_one_minute(self):
assert ET.fromstring(self.build(duration_seconds=20)).findtext("runtime") == "1"
def test_runtime_omitted_when_duration_unknown(self):
assert ET.fromstring(self.build(duration_seconds=None)).find("runtime") is None
def test_unique_id_marks_youtube_as_default(self):
unique = ET.fromstring(self.build()).find("uniqueid")
assert unique.get("type") == "youtube"
assert unique.get("default") == "true"
assert unique.text == "dQw4w9WgXcQ"
def test_season_and_episode_are_present(self):
root = ET.fromstring(self.build())
assert root.findtext("season") == "2026"
assert root.findtext("episode") == "8110"
def test_title_keeps_characters_that_the_filename_strips(self):
root = ET.fromstring(self.build(title="Hermitcraft S11#11: Expanding Business"))
assert root.findtext("title") == "Hermitcraft S11#11: Expanding Business"
def test_empty_plot_does_not_break(self):
assert ET.fromstring(self.build(plot=None)).find("plot") is not None
class TestTvshowNfo:
def test_well_formed_and_carries_channel_id(self):
root = ET.fromstring(nfo.tvshow_nfo("clabretro", HOSTILE, "UCabc123"))
assert root.tag == "tvshow"
assert root.findtext("title") == "clabretro"
assert root.findtext("studio") == "YouTube"
assert root.find("uniqueid").text == "UCabc123"
class TestWrite:
def test_write_is_atomic_and_leaves_no_temp_file(self, tmp_path):
target = tmp_path / "sub" / "tvshow.nfo"
nfo.write(target, b"<tvshow/>")
assert target.read_bytes() == b"<tvshow/>"
assert list(tmp_path.rglob("*.tmp")) == []
def test_overwrites_existing(self, tmp_path):
target = tmp_path / "tvshow.nfo"
nfo.write(target, b"<a/>")
nfo.write(target, b"<b/>")
assert target.read_bytes() == b"<b/>"
tests/test_proxy.py
+354
View File
@@ -0,0 +1,354 @@
"""The streaming proxy: HTTP range serving, and the safety rails.
Ported from the standalone `proxy/test_range.py` and `proxy/test_limits.py` so that
one `pytest` run covers everything. These now drive the real `make_handler(mgr, …)`
rather than the PoC's `make_handler(path, done)`, which means the routing, the
video-id validation and the wait-for-complete logic are exercised too.
No network, no yt-dlp, no ffmpeg: the pipeline runner is stubbed and the output
file is synthetic.
"""
from __future__ import annotations
import http.client
import importlib.util
import json
import os
import threading
import time
from http.server import ThreadingHTTPServer
from pathlib import Path
import pytest
PROXY_PATH = Path(__file__).resolve().parent.parent / "proxy" / "ytstream_proxy.py"
def _load_proxy():
spec = importlib.util.spec_from_file_location("ytstream_proxy", PROXY_PATH)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
proxy = _load_proxy()
# Position-identifiable body, so a wrong range returns visibly wrong bytes rather
# than merely the wrong length.
BODY = bytes(range(256)) * 400
TOTAL = len(BODY)
VIDEO_ID = "dQw4w9WgXcQ"
class StubSession:
"""A Session whose output file already exists."""
def __init__(self, path: Path, *, finished: bool = True):
self.video_id = VIDEO_ID
self.out_path = str(path)
self.final = threading.Event()
self.failed = None
self.readers = 0
self.last_used = time.time()
if finished:
self.final.set()
@property
def complete(self):
return self.final.is_set() and not self.failed
def size(self):
try:
return os.path.getsize(self.out_path)
except OSError:
return 0
class StubManager:
def __init__(self, session, *, growing=False):
self.session = session
self.growing = growing
self.lock = threading.Lock()
def get(self, video_id):
return self.session, None
def status(self):
return {"mode": "stub", "cache_used_gb": 0.0}
@pytest.fixture()
def complete_server(tmp_path):
"""A server over a finished file, with ranges honoured."""
path = tmp_path / "out.mp4"
path.write_bytes(BODY)
session = StubSession(path, finished=True)
yield from _serve(StubManager(session))
@pytest.fixture()
def growing_server(tmp_path):
"""A server over a file still being written."""
path = tmp_path / "out.mp4"
path.write_bytes(BODY[:1000])
session = StubSession(path, finished=False)
manager = StubManager(session, growing=True)
def finish():
time.sleep(0.2)
with open(path, "ab") as handle:
handle.write(BODY[1000:])
session.final.set()
threading.Thread(target=finish, daemon=True).start()
yield from _serve(manager)
def _serve(manager):
server = ThreadingHTTPServer(("127.0.0.1", 0), proxy.make_handler(manager, 30))
threading.Thread(target=server.serve_forever, daemon=True).start()
try:
yield server.server_address[1]
finally:
server.shutdown()
server.server_close()
def request(port, headers=None, method="GET", path=f"/watch/{VIDEO_ID}"):
conn = http.client.HTTPConnection("127.0.0.1", port, timeout=20)
conn.request(method, path, headers=headers or {})
response = conn.getresponse()
body = response.read()
head = dict(response.getheaders())
conn.close()
return response.status, head, body
# ------------------------------------------------------------- complete file
def test_bounded_range(complete_server):
status, head, body = request(complete_server, {"Range": "bytes=1000-1999"})
assert status == 206
assert len(body) == 1000
assert body == BODY[1000:2000]
assert head["Content-Range"] == f"bytes 1000-1999/{TOTAL}"
assert head["Content-Length"] == "1000"
def test_open_ended_range(complete_server):
status, head, body = request(complete_server, {"Range": "bytes=102000-"})
assert status == 206
assert body == BODY[102000:]
assert head["Content-Range"] == f"bytes 102000-{TOTAL - 1}/{TOTAL}"
def test_suffix_range(complete_server):
"""bytes=-N means the final N bytes, not the first N."""
status, head, body = request(complete_server, {"Range": "bytes=-100"})
assert status == 206
assert len(body) == 100
assert body == BODY[-100:]
assert head["Content-Range"] == f"bytes {TOTAL - 100}-{TOTAL - 1}/{TOTAL}"
def test_single_byte_range(complete_server):
status, _, body = request(complete_server, {"Range": "bytes=5-5"})
assert status == 206
assert body == BODY[5:6]
def test_end_beyond_eof_is_clamped(complete_server):
status, head, body = request(complete_server,
{"Range": f"bytes=102000-{TOTAL + 5000}"})
assert status == 206
assert body == BODY[102000:]
assert head["Content-Range"] == f"bytes 102000-{TOTAL - 1}/{TOTAL}"
def test_range_past_eof_is_416(complete_server):
"""And it must carry Content-Range: bytes */total, or clients retry forever."""
status, head, _ = request(complete_server, {"Range": f"bytes={TOTAL + 10}-"})
assert status == 416
assert head["Content-Range"] == f"bytes */{TOTAL}"
def test_no_range_serves_the_whole_body(complete_server):
status, head, body = request(complete_server)
assert status == 200
assert body == BODY
assert head["Accept-Ranges"] == "bytes"
def test_multi_range_falls_back_to_the_whole_body(complete_server):
"""Legal, and beats mis-serving one part of a multipart response."""
status, _, body = request(complete_server, {"Range": "bytes=0-99,200-299"})
assert status == 200
assert body == BODY
@pytest.mark.parametrize("header", ["bytes=", "bytes=abc-def", "items=0-99", "0-99"])
def test_unparseable_range_serves_the_whole_body(complete_server, header):
status, _, body = request(complete_server, {"Range": header})
assert status == 200
assert len(body) == TOTAL
def test_head_returns_headers_and_no_body(complete_server):
status, head, body = request(complete_server, method="HEAD")
assert status == 200
assert body == b""
assert head["Content-Length"] == str(TOTAL)
# ------------------------------------------------------------- growing file
def test_growing_file_is_chunked_and_tracks_to_eof(growing_server):
"""Ranges cannot be honoured mid-write: there is no reliable time-to-byte
mapping into a fragmented MP4 yet, so it becomes a non-seekable stream."""
status, head, body = request(growing_server, {"Range": "bytes=500-999"})
assert status == 200
assert head.get("Transfer-Encoding") == "chunked"
assert body == BODY
# ------------------------------------------------------------------- routing
def test_unknown_path_is_404(complete_server):
status, _, _ = request(complete_server, path="/nope")
assert status == 404
def test_malformed_video_id_is_400(complete_server):
status, _, _ = request(complete_server, path="/watch/short")
assert status == 400
@pytest.mark.parametrize("bad", ["../../etc/passwd", "abcdefghij", "abcdefghijkl"])
def test_video_id_must_be_exactly_eleven_safe_chars(complete_server, bad):
status, _, _ = request(complete_server, path=f"/watch/{bad}")
assert status in (400, 404)
def test_healthz_returns_json(complete_server):
status, head, body = request(complete_server, path="/healthz")
assert status == 200
assert head["Content-Type"] == "application/json"
assert json.loads(body)["mode"] == "stub"
# ------------------------------------------------- safety rails on the manager
def vid(n):
return f"vid{n:08d}"
@pytest.fixture()
def manager_factory(tmp_path):
made = []
def build(**kwargs):
options = dict(max_pipelines=99, cache_bytes=10 ** 12, no_fetch=False,
growing=False, max_retries=0, max_starts=3,
starts_window=3600.0)
options.update(kwargs)
work = tmp_path / f"work{len(made)}"
work.mkdir()
class Recording(proxy.Manager):
"""Records starts instead of spawning a pipeline."""
def __init__(self, *args, **kw):
super().__init__(*args, **kw)
self.ran = []
def _run(self, session):
self.ran.append(session.video_id)
session.final.set() # completes instantly, never stays active
manager = Recording(str(work), **options)
made.append(manager)
return manager
return build
def test_cold_start_budget_is_enforced(manager_factory):
manager = manager_factory(max_starts=3)
results = [manager.get(vid(i)) for i in range(5)]
started = [r for r in results if r[0] is not None]
refused = [r for r in results if r[0] is None]
assert len(started) == 3
assert len(refused) == 2
assert all("cold-start budget" in r[1] for r in refused)
assert manager.counters["refused_ratelimit"] == 2
assert len(manager.ran) == 3
def test_cache_hits_are_never_rate_limited(manager_factory):
"""Re-watching an already-fetched video must keep working with the budget
spent, or a metadata refresh would break normal playback for an hour."""
manager = manager_factory(max_starts=3)
for i in range(5):
manager.get(vid(i))
session, refusal = manager.get(vid(0))
assert session is not None and refusal is None
assert manager.counters["reused"] >= 1
assert len(manager.ran) == 3
def test_start_budget_window_prunes(manager_factory):
manager = manager_factory(max_starts=2)
manager.get(vid(10))
manager.get(vid(11))
assert manager.get(vid(12))[0] is None
# Age the recorded starts past the window.
manager.start_log = type(manager.start_log)(t - 3601 for t in manager.start_log)
assert manager.get(vid(13))[0] is not None
assert len(manager.start_log) == 1
def test_concurrency_cap(tmp_path):
class Blocking(proxy.Manager):
"""Pipelines that never finish, so sessions stay active."""
def _run(self, session):
pass
work = tmp_path / "blocking"
work.mkdir()
manager = Blocking(str(work), max_pipelines=2, cache_bytes=10 ** 12,
no_fetch=False, growing=False, max_retries=0,
max_starts=99, starts_window=3600.0)
results = [manager.get(vid(20 + i)) for i in range(4)]
ok = [r for r in results if r[0] is not None]
refused = [r for r in results if r[0] is None]
assert len(ok) == 2
assert len(refused) == 2
assert all("pipeline cap" in r[1] for r in refused)
# A busy refusal must not consume start budget, or a burst of concurrent
# requests would exhaust the hourly allowance without fetching anything.
assert manager.counters["refused_busy"] == 2
assert len(manager.start_log) == 2
def test_no_fetch_mode_refuses_everything(manager_factory):
manager = manager_factory(no_fetch=True, max_starts=99)
session, refusal = manager.get(vid(30))
assert session is None
assert "no-fetch" in refusal
assert manager.ran == []
assert len(manager.start_log) == 0
tests/test_reap.py
+201
View File
@@ -0,0 +1,201 @@
"""Retention: the window, the min-keep floor, and the tombstones.
These are the tests that matter most in this suite. Retention is the only part of
ytstream that deletes things, and two of its rules exist because of measured
behaviour rather than taste:
* `min_keep_videos` exists because 52 of 117 real channels upload nothing in 30
days and would otherwise be empty, flickering Jellyfin series.
* the `aged_out` tombstone exists because without it the poller re-materialises
everything the sweep just deleted, forever.
"""
from __future__ import annotations
from datetime import timedelta
import pytest
from ytstream import discovery, reap, strm, util, videos
from conftest import add_channel, add_video
def _old(days: int) -> str:
return (util.today() - timedelta(days=days)).isoformat()
@pytest.fixture()
def materialised(conn, settings, media_root, channel):
"""Put videos on disk with a spread of ages. Returns their ids, newest first."""
counter = {"n": 0}
def build(ages: list[int], chan=None):
chan = chan or channel
ids = []
for age in ages:
video_id = f"vid{counter['n']:08d}"
counter["n"] += 1
add_video(conn, chan["id"], video_id, upload_date=_old(age))
strm.materialise(conn, settings, chan, videos.get(conn, video_id))
ids.append(video_id)
return ids
return build
def test_video_inside_the_window_is_kept(conn, settings, materialised):
materialised([1])
assert reap.candidates(conn, settings) == []
def test_video_outside_the_window_is_deleted(conn, settings, materialised, media_root):
settings.set("min_keep_videos", "0")
[video_id] = materialised([99])
path = media_root / videos.get(conn, video_id)["rel_path"]
assert path.exists()
result = reap.run(conn, settings)
assert result["aged_out"] == 1
assert not path.exists()
assert videos.get(conn, video_id)["state"] == videos.AGED_OUT
def test_deleting_removes_the_sidecars_too(conn, settings, materialised, media_root):
settings.set("min_keep_videos", "0")
[video_id] = materialised([99])
strm_path = media_root / videos.get(conn, video_id)["rel_path"]
stem = strm_path.name[: -len(".strm")]
nfo_path = strm_path.with_name(stem + ".nfo")
assert nfo_path.exists()
reap.run(conn, settings)
assert not nfo_path.exists()
assert not strm_path.exists()
def test_min_keep_videos_protects_the_newest_regardless_of_age(
conn, settings, materialised
):
"""The measured common case: a channel whose only videos are all ancient."""
settings.set("min_keep_videos", "5")
materialised([400, 500, 600, 700, 800])
assert reap.candidates(conn, settings) == []
assert reap.run(conn, settings)["aged_out"] == 0
def test_min_keep_videos_releases_once_enough_newer_ones_exist(
conn, settings, materialised
):
"""Six old videos with a floor of five: exactly the oldest one goes."""
settings.set("min_keep_videos", "5")
ids = materialised([100, 200, 300, 400, 500, 600])
oldest = ids[-1]
assert reap.run(conn, settings)["aged_out"] == 1
assert videos.get(conn, oldest)["state"] == videos.AGED_OUT
for kept in ids[:-1]:
assert videos.get(conn, kept)["state"] == videos.MATERIALISED
def test_min_keep_floor_counts_per_channel_not_globally(
conn, settings, media_root, channel, materialised
):
other = add_channel(conn, "UCzzzzzzzzzzzzzzzzzzzzzz", "Other", "Other")
settings.set("min_keep_videos", "2")
materialised([300, 400], chan=channel)
materialised([300, 400], chan=other)
# Two channels, two videos each, floor of two: nothing is eligible. If the
# floor were global, two of the four would be deleted.
assert reap.candidates(conn, settings) == []
def test_channel_retention_override_beats_the_global_setting(
conn, settings, media_root, channel, materialised
):
settings.set("min_keep_videos", "0")
settings.set("retention_days", "365")
materialised([90])
assert reap.candidates(conn, settings) == []
with conn:
conn.execute("UPDATE channel SET retention_days = 30 WHERE id = ?",
(channel["id"],))
assert len(reap.candidates(conn, settings)) == 1
def test_aged_out_row_is_a_tombstone_the_poller_will_not_revive(
conn, settings, media_root, channel, materialised
):
"""The failure this prevents: sweep deletes, poll re-adds, forever."""
settings.set("min_keep_videos", "0")
[video_id] = materialised([99])
reap.run(conn, settings)
assert videos.get(conn, video_id)["state"] == videos.AGED_OUT
# The video is still in the feed — YouTube has no idea we deleted it.
entry = {"video_id": video_id, "title": "Video", "published": util.today(),
"published_at": None}
outcome = discovery._record(
conn, channel, entry, videos.SOURCE_UULF, util.today() - timedelta(days=30)
)
assert outcome == "known"
assert videos.get(conn, video_id)["state"] == videos.AGED_OUT
def test_rescan_revives_skipped_old_but_never_aged_out(
conn, settings, media_root, channel, materialised
):
settings.set("min_keep_videos", "0")
[gone] = materialised([99])
reap.run(conn, settings)
add_video(conn, channel["id"], "skippedold1", upload_date=_old(40),
state=videos.SKIPPED_OLD)
settings.set("retention_days", "365")
revived = discovery.rescan_channel(conn, settings, channel)
assert revived == 1
assert videos.get(conn, "skippedold1")["state"] == videos.LISTED
assert videos.get(conn, gone)["state"] == videos.AGED_OUT
def test_empty_season_directory_is_pruned(conn, settings, materialised, media_root):
settings.set("min_keep_videos", "0")
[video_id] = materialised([99])
season_dir = (media_root / videos.get(conn, video_id)["rel_path"]).parent
assert season_dir.is_dir()
reap.run(conn, settings)
assert not season_dir.exists()
# The channel directory survives: it still holds tvshow.nfo and artwork, and
# an active subscription should not vanish from Jellyfin between uploads.
assert season_dir.parent.is_dir()
def test_prune_never_climbs_past_the_media_root(media_root):
nested = media_root / "Chan" / "Season 2026"
nested.mkdir(parents=True)
removed = util.prune_empty_dirs(nested, media_root)
assert removed == 2
assert media_root.is_dir()
def test_row_without_rel_path_still_gets_a_tombstone(conn, settings, channel):
"""A video whose files vanished underneath us must not be retried forever."""
settings.set("min_keep_videos", "0")
add_video(conn, channel["id"], "orphan0001", upload_date=_old(99),
state=videos.MATERIALISED)
row = videos.get(conn, "orphan0001")
assert row["rel_path"] is None
assert reap.delete_video(conn, row) is False
assert videos.get(conn, "orphan0001")["state"] == videos.AGED_OUT
tests/test_runner.py
+232
View File
@@ -0,0 +1,232 @@
"""Orchestration: ordering, the lock, and what turns the cron check red."""
from __future__ import annotations
import pytest
from ytstream import config, discovery, jellyfin, reap, runner, subsync, videos
from conftest import add_video
@pytest.fixture()
def quiet_jellyfin(monkeypatch):
"""Jellyfin refresh is best effort; record calls instead of making them."""
calls = []
monkeypatch.setattr(jellyfin.Jellyfin, "refresh", lambda self: calls.append(1))
return calls
@pytest.fixture()
def no_thumbs(monkeypatch):
from ytstream import strm
monkeypatch.setattr(strm, "fetch_thumbnail", lambda *a, **k: False)
# ----------------------------------------------------------------------- lock
def test_lock_is_exclusive(tmp_path):
path = tmp_path / "run.lock"
with runner.exclusive_lock(path):
with pytest.raises(runner.AlreadyRunning):
with runner.exclusive_lock(path):
pass # pragma: no cover
def test_lock_is_released_after_use(tmp_path):
path = tmp_path / "run.lock"
with runner.exclusive_lock(path):
pass
with runner.exclusive_lock(path):
pass
def test_lock_is_released_even_when_the_body_raises(tmp_path):
path = tmp_path / "run.lock"
with pytest.raises(ValueError):
with runner.exclusive_lock(path):
raise ValueError("boom")
with runner.exclusive_lock(path):
pass
# --------------------------------------------------------------- materialising
def test_materialise_all_writes_everything_listed(conn, settings, media_root,
channel, no_thumbs):
for index in range(3):
add_video(conn, channel["id"], f"vid{index:08d}")
stats = runner.materialise_all(conn, settings)
assert stats["materialised"] == 3
assert stats["errors"] == 0
assert videos.queue_depth(conn) == 0
def test_tvshow_nfo_is_written_once_per_channel(conn, settings, media_root,
channel, no_thumbs):
for index in range(3):
add_video(conn, channel["id"], f"vid{index:08d}")
stats = runner.materialise_all(conn, settings)
assert stats["shows"] == 1
assert (media_root / "clabretro" / "tvshow.nfo").exists()
def test_channel_with_nothing_in_the_window_creates_no_directory(
conn, settings, media_root, channel, no_thumbs
):
"""52 of 117 measured channels are in this state, and an empty series in
Jellyfin looks like a bug rather than a quiet channel."""
add_video(conn, channel["id"], "vid00000001", state=videos.SKIPPED_OLD)
runner.materialise_all(conn, settings)
assert not (media_root / "clabretro").exists()
def test_materialise_all_honours_the_limit(conn, settings, media_root, channel,
no_thumbs):
for index in range(5):
add_video(conn, channel["id"], f"vid{index:08d}")
stats = runner.materialise_all(conn, settings, limit=2)
assert stats["materialised"] == 2
def test_materialise_skips_a_video_whose_channel_vanished(conn, settings,
media_root, channel,
no_thumbs):
add_video(conn, channel["id"], "vid00000001")
# Simulate the channel being unsubscribed between claiming and writing, with
# foreign keys off so the row survives to be found.
conn.execute("PRAGMA foreign_keys = OFF")
with conn:
conn.execute("DELETE FROM channel WHERE id = ?", (channel["id"],))
stats = runner.materialise_all(conn, settings)
assert stats["materialised"] == 0
assert stats["errors"] == 0
# ------------------------------------------------------------------ full cycle
def test_run_order_is_sync_poll_materialise_reap(conn, settings, media_root,
channel, monkeypatch,
quiet_jellyfin):
order = []
monkeypatch.setattr(subsync, "sync_all",
lambda *a: order.append("sync") or {"added": 0, "queued": 0,
"removed": 0, "refused": 0,
"sources": 0,
"pending_removal": 0})
monkeypatch.setattr(discovery, "poll_all",
lambda *a: order.append("poll") or {"queued": 0})
monkeypatch.setattr(runner, "materialise_all",
lambda *a, **k: order.append("materialise") or
{"materialised": 0, "shows": 0, "errors": 0})
monkeypatch.setattr(reap, "run",
lambda *a: order.append("reap") or {"aged_out": 0})
runner.run(conn, settings)
assert order == ["sync", "poll", "materialise", "reap"]
def test_single_channel_run_skips_the_sync(conn, settings, channel, monkeypatch,
quiet_jellyfin):
"""A targeted run is an operator action, not a mirror pass."""
monkeypatch.setattr(subsync, "sync_all",
lambda *a: pytest.fail("sync should not run"))
monkeypatch.setattr(discovery, "poll_all", lambda *a: {"queued": 0})
monkeypatch.setattr(reap, "run", lambda *a: {"aged_out": 0})
result = runner.run(conn, settings, channel_pk=channel["id"])
assert "sync" not in result
def test_run_records_last_run_at(conn, settings, channel, monkeypatch,
quiet_jellyfin):
monkeypatch.setattr(subsync, "sync_all", lambda *a: {"added": 0, "queued": 0,
"removed": 0, "refused": 0,
"sources": 0,
"pending_removal": 0})
monkeypatch.setattr(discovery, "poll_all", lambda *a: {"queued": 0})
monkeypatch.setattr(reap, "run", lambda *a: {"aged_out": 0})
runner.run(conn, settings)
assert settings.raw("last_run_at")
def test_jellyfin_is_refreshed_only_when_the_tree_changed(
conn, settings, media_root, channel, monkeypatch, quiet_jellyfin, no_thumbs
):
monkeypatch.setattr(subsync, "sync_all", lambda *a: {"added": 0, "queued": 0,
"removed": 0, "refused": 0,
"sources": 0,
"pending_removal": 0})
monkeypatch.setattr(discovery, "poll_all", lambda *a: {"queued": 0})
runner.run(conn, settings)
assert quiet_jellyfin == []
add_video(conn, channel["id"], "vid00000001")
runner.run(conn, settings)
assert len(quiet_jellyfin) == 1
# ------------------------------------------------------------------ exit codes
def test_refused_sync_turns_the_check_red():
"""A silently-broken mirror is the worst outcome available: nothing looks
wrong until someone asks why a channel never appeared."""
assert runner.exit_code({"sync": {"refused": 1}}) == 1
def test_materialise_errors_turn_the_check_red():
assert runner.exit_code({"materialise": {"errors": 2}}) == 1
def test_poll_failures_alone_do_not_turn_the_check_red():
"""Two of 119 measured channels fail permanently — terminated or private."""
assert runner.exit_code({"poll": {"failed": 2}}) == 0
def test_clean_run_is_zero():
assert runner.exit_code({"sync": {"refused": 0}, "poll": {"failed": 0},
"materialise": {"errors": 0}}) == 0
# ------------------------------------------------------------------- summarise
def test_summarise_mentions_a_refusal():
text = runner.summarise({
"sync": {"added": 0, "queued": 0, "removed": 0, "refused": 1},
"poll": {"channels": 5, "queued": 0},
"materialise": {"materialised": 0},
"reap": {"aged_out": 0},
})
assert "SYNC_REFUSED" in text
def test_summarise_is_a_single_line():
text = runner.summarise({
"sync": {"added": 1, "queued": 2, "removed": 3, "refused": 0},
"poll": {"channels": 119, "queued": 4, "shorts": 5, "live": 6, "failed": 2},
"materialise": {"materialised": 7, "errors": 0},
"reap": {"aged_out": 8},
})
assert "\n" not in text
assert "channels=119" in text
assert "materialised=7" in text
assert "aged_out=8" in text
tests/test_settings.py
+125
View File
@@ -0,0 +1,125 @@
"""Typed settings accessors and form validation."""
import pytest
from ytstream import settings as settings_module
from ytstream.settings import DEFAULTS, Settings, validate, validate_all
class TestAccessors:
def test_missing_key_returns_the_default(self, settings):
assert settings.get_int("retention_days") == 30
assert settings.get_int("min_keep_videos") == 5
assert settings.get_str("proxy_base_url") == "http://127.0.0.1:8099"
def test_unknown_key_never_raises(self, settings):
assert settings.get_str("no_such_key") == ""
assert settings.get_int("no_such_key") == 0
def test_set_then_get(self, settings):
settings.set("retention_days", "30")
assert settings.get_int("retention_days") == 30
def test_set_overwrites(self, settings):
settings.set("retention_days", "30")
settings.set("retention_days", "45")
assert settings.get_int("retention_days") == 45
def test_corrupt_integer_falls_back_to_the_default(self, settings):
settings.set("retention_days", "not a number")
assert settings.get_int("retention_days") == 30
def test_reads_are_not_cached(self, conn, settings):
"""Rotating the API key must be nothing more than saving a new value —
no restart of the hourly job or the admin server."""
settings.set("youtube_api_key", "first")
assert settings.get_str("youtube_api_key") == "first"
# A second Settings object standing in for the other process.
other = Settings(conn)
other.set("youtube_api_key", "second")
assert settings.get_str("youtube_api_key") == "second"
def test_all_editable_covers_every_default(self, settings):
assert set(settings.all_editable()) == set(DEFAULTS)
def test_secrets_are_not_editable(self):
for key in settings_module.SECRET_KEYS:
assert key not in settings_module.EDITABLE
def test_both_api_keys_are_masked(self):
assert "youtube_api_key" in settings_module.MASKED_KEYS
assert "jellyfin_api_key" in settings_module.MASKED_KEYS
def test_download_era_settings_are_gone(self):
for gone in ("disk_cap_gb", "write_subs", "sub_langs", "sponsorblock_mark",
"max_attempts", "backfill_days"):
assert gone not in DEFAULTS
class TestValidation:
@pytest.mark.parametrize(
"key, value",
[
("retention_days", "30"),
("min_keep_videos", "0"),
("min_keep_videos", "5"),
("backfill_max_videos", "300"),
("max_height", "1080"),
("min_duration_seconds", "120"),
("subsync_max_new", "25"),
("subsync_missing_threshold", "3"),
("jellyfin_url", "http://127.0.0.1:8096"),
("pot_provider_url", "https://example.com:4416"),
("proxy_base_url", "http://127.0.0.1:8099"),
("youtube_api_key", ""),
],
)
def test_accepts_good_values(self, key, value):
ok, _ = validate(key, value)
assert ok
@pytest.mark.parametrize(
"key, value",
[
("retention_days", "abc"),
("retention_days", "0"),
("retention_days", "-5"),
("min_keep_videos", "-1"),
("max_height", "10"),
("min_duration_seconds", "-1"),
("jellyfin_url", "not-a-url"),
("jellyfin_url", "ftp://host/"),
("jellyfin_url", "http://"),
("proxy_base_url", "nonsense"),
],
)
def test_rejects_bad_values(self, key, value):
ok, message = validate(key, value)
assert not ok
assert message
def test_zero_missing_threshold_is_rejected(self):
"""Zero would unsubscribe on the first absent response, which is exactly
the failure the threshold exists to prevent."""
ok, message = validate("subsync_missing_threshold", "0")
assert not ok
assert message
def test_validate_all_reports_each_bad_field(self):
errors = validate_all(
{"retention_days": "abc", "max_height": "1080", "jellyfin_url": "nope"}
)
assert set(errors) == {"retention_days", "jellyfin_url"}
def test_validate_all_ignores_unknown_keys(self):
assert validate_all({"not_a_setting": "x"}) == {}
def test_empty_api_keys_are_acceptable(self):
assert validate("jellyfin_api_key", "")[0]
assert validate("youtube_api_key", "")[0]
def test_whitespace_is_tolerated(self):
ok, _ = validate("retention_days", " 21 ")
assert ok
tests/test_strm.py
+260
View File
@@ -0,0 +1,260 @@
"""Materialising: .strm contents, NFO sidecars, naming, and idempotency."""
from __future__ import annotations
from datetime import date
import pytest
from ytstream import config, strm, videos
from conftest import add_video
@pytest.fixture()
def video(conn, channel):
return add_video(conn, channel["id"], "dQw4w9WgXcQ",
title="A Video: With/Punctuation",
upload_date="2026-08-12", duration=1337)
@pytest.fixture()
def no_thumbs(monkeypatch):
"""Thumbnails are a network fetch; every test here runs without one."""
monkeypatch.setattr(strm, "fetch_thumbnail", lambda *a, **k: False)
def test_strm_contains_only_the_proxy_url(conn, settings, media_root, channel,
video, no_thumbs):
result = strm.materialise(conn, settings, channel, video)
path = media_root / result["rel_path"]
assert path.read_text() == "http://127.0.0.1:8099/watch/dQw4w9WgXcQ"
# No trailing newline, and nothing else in the file.
assert path.read_bytes() == b"http://127.0.0.1:8099/watch/dQw4w9WgXcQ"
def test_strm_url_follows_the_configured_base(conn, settings, media_root, channel,
video, no_thumbs):
settings.set("proxy_base_url", "http://127.0.0.1:9999/")
result = strm.materialise(conn, settings, channel, video)
assert (media_root / result["rel_path"]).read_text().startswith(
"http://127.0.0.1:9999/watch/"
)
def test_layout_matches_the_naming_scheme(conn, settings, media_root, channel,
video, no_thumbs):
result = strm.materialise(conn, settings, channel, video)
# Season = upload year, episode = MMDD*10 + ordinal.
assert result["season"] == 2026
assert result["episode"] == 8120
assert result["rel_path"] == (
"clabretro/Season 2026/"
"clabretro - S2026E8120 - A Video With Punctuation [dQw4w9WgXcQ].strm"
)
def test_nfo_is_written_alongside(conn, settings, media_root, channel, video,
no_thumbs):
result = strm.materialise(conn, settings, channel, video)
nfo_path = (media_root / result["rel_path"]).with_suffix(".nfo")
text = nfo_path.read_text()
assert "<season>2026</season>" in text
assert "<episode>8120</episode>" in text
assert "<aired>2026-08-12</aired>" in text
# durationinseconds is what stops a .strm episode showing a zero runtime
# before it has ever been played.
assert "<durationinseconds>1337</durationinseconds>" in text
assert 'type="youtube"' in text
def test_nfo_has_no_streamdetails(conn, settings, media_root, channel, video,
no_thumbs):
"""Pre-seeding them was measured to change nothing about Jellyfin probing."""
result = strm.materialise(conn, settings, channel, video)
text = (media_root / result["rel_path"]).with_suffix(".nfo").read_text()
assert "streamdetails" not in text
assert "fileinfo" not in text
def test_materialise_is_idempotent(conn, settings, media_root, channel, video,
no_thumbs):
first = strm.materialise(conn, settings, channel, video)
before = (media_root / first["rel_path"]).read_bytes()
second = strm.materialise(
conn, settings, channel, videos.get(conn, "dQw4w9WgXcQ")
)
assert second["rel_path"] == first["rel_path"]
assert (media_root / second["rel_path"]).read_bytes() == before
def test_row_is_marked_materialised(conn, settings, media_root, channel, video,
no_thumbs):
strm.materialise(conn, settings, channel, video)
row = videos.get(conn, "dQw4w9WgXcQ")
assert row["state"] == videos.MATERIALISED
assert row["rel_path"]
assert row["materialised_at"]
def test_episode_ordinals_increment_within_a_day(conn, settings, media_root,
channel, no_thumbs):
for index in range(3):
row = add_video(conn, channel["id"], f"vid{index:08d}",
upload_date="2026-08-12")
result = strm.materialise(conn, settings, channel, row)
assert result["episode"] == 8120 + index
def test_ordinals_are_stable_when_an_earlier_video_ages_out(
conn, settings, media_root, channel, no_thumbs
):
"""Aged-out rows keep their episode number, so later ordinals never shift."""
first = add_video(conn, channel["id"], "vid00000001", upload_date="2026-08-12")
strm.materialise(conn, settings, channel, first)
videos.mark_aged_out(conn, "vid00000001")
second = add_video(conn, channel["id"], "vid00000002", upload_date="2026-08-12")
result = strm.materialise(conn, settings, channel, second)
assert result["episode"] == 8121
def test_title_falls_back_to_the_video_id(conn, settings, media_root, channel,
no_thumbs):
"""Backfilled rows carry no title until the feed supplies one."""
row = add_video(conn, channel["id"], "vid00000001", title="",
upload_date="2026-08-12")
result = strm.materialise(conn, settings, channel, row)
assert "vid00000001" in result["rel_path"]
def test_write_show_creates_tvshow_nfo(media_root, channel):
strm.write_show(channel)
text = (media_root / "clabretro" / "tvshow.nfo").read_text()
assert "<title>clabretro</title>" in text
assert channel["channel_id"] in text
def test_show_nfo_uses_the_channel_title_not_a_video_title(conn, media_root, channel):
"""The channel/video join has `title` on both sides, and reading the wrong one
renamed every series after whichever episode happened to be first."""
add_video(conn, channel["id"], "vid00000001", title="Some Episode Title")
strm.write_show(channel)
text = (media_root / "clabretro" / "tvshow.nfo").read_text()
assert "Some Episode Title" not in text
def test_remove_deletes_strm_and_sidecars(conn, settings, media_root, channel,
video, no_thumbs):
result = strm.materialise(conn, settings, channel, video)
strm_path = media_root / result["rel_path"]
stem = strm_path.name[: -len(".strm")]
thumb = strm_path.with_name(stem + "-thumb.jpg")
thumb.write_bytes(b"x" * 2000)
removed = strm.remove(videos.get(conn, "dQw4w9WgXcQ"))
assert removed == 3 # .strm, .nfo, -thumb.jpg
assert not strm_path.exists()
assert not thumb.exists()
def test_remove_leaves_files_it_does_not_own(conn, settings, media_root, channel,
video, no_thumbs):
result = strm.materialise(conn, settings, channel, video)
season_dir = (media_root / result["rel_path"]).parent
stranger = season_dir / "someone-elses-file.txt"
stranger.write_text("not ours")
strm.remove(videos.get(conn, "dQw4w9WgXcQ"))
assert stranger.exists()
def test_remove_channel_tree(conn, media_root, channel):
tree = media_root / "clabretro"
(tree / "Season 2026").mkdir(parents=True)
(tree / "tvshow.nfo").write_text("<tvshow/>")
assert strm.remove_channel_tree(channel) is True
assert not tree.exists()
def test_remove_channel_tree_refuses_the_media_root(conn, media_root):
from conftest import add_channel
row = add_channel(conn, "UC" + "q" * 22, "Blank", "")
keep = media_root / "keep"
keep.mkdir()
assert strm.remove_channel_tree(row) is False
assert keep.exists()
def test_fetch_thumbnail_rejects_the_grey_placeholder(monkeypatch, tmp_path):
"""YouTube serves a tiny placeholder rather than a 404 for missing maxres."""
import urllib.request
class Response:
status = 200
def __init__(self, payload):
self.payload = payload
def read(self):
return self.payload
def __enter__(self):
return self
def __exit__(self, *exc):
return False
calls = []
def fake_open(request, timeout=None):
calls.append(request.full_url)
# maxres returns the placeholder; hq returns something real.
return Response(b"x" * 120 if "maxres" in request.full_url else b"y" * 5000)
monkeypatch.setattr(urllib.request, "urlopen", fake_open)
destination = tmp_path / "out-thumb.jpg"
assert strm.fetch_thumbnail("dQw4w9WgXcQ", destination) is True
assert destination.read_bytes() == b"y" * 5000
assert len(calls) == 2
def test_fetch_thumbnail_is_skipped_when_one_already_exists(monkeypatch, tmp_path):
import urllib.request
monkeypatch.setattr(urllib.request, "urlopen",
lambda *a, **k: pytest.fail("should not fetch"))
existing = tmp_path / "out-thumb.jpg"
existing.write_bytes(b"cached")
assert strm.fetch_thumbnail("dQw4w9WgXcQ", existing) is True
def test_untitled_video_does_not_get_its_id_written_back_as_a_title(
conn, settings, media_root, channel, no_thumbs
):
"""Writing the fallback back to the database makes the row look titled, which
permanently disables the title repair in discovery._record. That shipped once
and left five of twenty episodes named after their video ids."""
row = add_video(conn, channel["id"], "vid00000001", title="",
upload_date="2026-08-12")
result = strm.materialise(conn, settings, channel, row)
# The filename falls back to the id...
assert "vid00000001]" in result["rel_path"]
# ...but the row stays untitled, so a later feed poll can still repair it.
assert videos.get(conn, "vid00000001")["title"] == ""
tests/test_subsync.py
+366
View File
@@ -0,0 +1,366 @@
"""Subscription mirroring, and above all its refusals.
The sync is authoritative in both directions and the removal half deletes a
channel's whole tree, so most of what needs pinning down here is what it does
with *bad* data. Every one of these failure modes looks identical to "he
unsubscribed from everything" on the wire:
403 subscriptionForbidden he re-ticked the privacy box
network error susan's link dropped
200 with zero items could be true, could be a broken response
None of them may delete anything. The tests below are the reason that claim can
be made with a straight face.
"""
from __future__ import annotations
import pytest
from ytstream import api, subsync, videos
from conftest import CHANNEL_ID, FakeApi, add_channel, add_video, patch_api
BROTHER = "UCPcTWaLV8zwx4WP4QExHj4Q"
@pytest.fixture()
def source(conn):
key = subsync.add_source(conn, channel_id=BROTHER, label="C Flux")
with conn:
conn.execute("UPDATE source SET imported = 1 WHERE key = ?", (key,))
return subsync.get_source(conn, key)
@pytest.fixture()
def fresh_source(conn):
"""A source that has never imported — the first-sync path."""
key = subsync.add_source(conn, channel_id=BROTHER, label="C Flux")
return subsync.get_source(conn, key)
def sub(channel_id, title):
return {"channel_id": channel_id, "title": title}
def _fake(monkeypatch, **kwargs):
return patch_api(monkeypatch, subsync, FakeApi(**kwargs))
# ------------------------------------------------------------------- additions
def test_first_sync_queues_everything_and_adds_nothing(
conn, settings, fresh_source, monkeypatch
):
"""119 subscriptions would trip any cap, so day one is approval-only."""
_fake(monkeypatch, subs=[sub(f"UC{i:022d}", f"Chan {i}") for i in range(40)])
stats = subsync.sync_source(conn, settings, fresh_source)
assert stats["added"] == 0
assert stats["queued"] == 40
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 0
assert len(subsync.pending(conn)) == 40
# And it does not queue them again on the next pass.
assert subsync.get_source(conn, fresh_source["key"])["imported"] == 1
def test_second_sync_adds_within_the_cap(conn, settings, source, monkeypatch):
settings.set("subsync_max_new", "25")
_fake(monkeypatch, subs=[sub("UCaaaaaaaaaaaaaaaaaaaaaa", "Alpha")],
channel={"channel_id": "UCaaaaaaaaaaaaaaaaaaaaaa", "title": "Alpha",
"description": "", "handle": None, "avatar_url": None})
stats = subsync.sync_source(conn, settings, source)
assert stats["added"] == 1
row = conn.execute("SELECT * FROM channel").fetchone()
assert row["title"] == "Alpha"
assert row["source"] == "youtube"
def test_burst_over_the_cap_adds_nothing_and_queues_all(
conn, settings, source, monkeypatch
):
settings.set("subsync_max_new", "3")
_fake(monkeypatch, subs=[sub(f"UC{i:022d}", f"Chan {i}") for i in range(10)])
stats = subsync.sync_source(conn, settings, source)
assert stats["added"] == 0
assert stats["queued"] == 10
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 0
def test_rejected_channels_are_never_queued_again(
conn, settings, fresh_source, monkeypatch
):
entries = [sub("UCaaaaaaaaaaaaaaaaaaaaaa", "Alpha")]
_fake(monkeypatch, subs=entries)
subsync.sync_source(conn, settings, fresh_source)
queued = subsync.pending(conn)
assert len(queued) == 1
subsync.resolve(conn, [queued[0]["id"]], "rejected")
source = subsync.get_source(conn, fresh_source["key"])
stats = subsync.sync_source(conn, settings, source)
assert stats["queued"] == 0
assert stats["added"] == 0
assert subsync.pending(conn) == []
def test_approving_subscribes(conn, settings, fresh_source, monkeypatch):
_fake(monkeypatch, subs=[sub("UCaaaaaaaaaaaaaaaaaaaaaa", "Alpha")],
channel={"channel_id": "UCaaaaaaaaaaaaaaaaaaaaaa", "title": "Alpha",
"description": "Desc", "handle": "@alpha", "avatar_url": None})
subsync.sync_source(conn, settings, fresh_source)
queued = subsync.pending(conn)
stats = subsync.approve(conn, settings, [row["id"] for row in queued])
assert stats == {"added": 1, "failed": 0}
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 1
assert subsync.pending(conn) == []
# -------------------------------------------------------------------- refusals
def test_private_subscriptions_change_nothing(conn, settings, source, monkeypatch):
add_channel(conn, CHANNEL_ID, "Existing", "Existing")
_fake(monkeypatch, raises=api.SubscriptionsPrivate(
403, "subscriptionForbidden", "not allowed"))
stats = subsync.sync_source(conn, settings, source)
assert stats["refused"]
assert "private" in stats["refused"]
assert stats["removed"] == 0
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 1
# The channel's miss counter must not move either, or three consecutive
# outages would delete the library without a single healthy response.
assert conn.execute("SELECT missing_syncs FROM channel").fetchone()[0] == 0
def test_empty_response_is_treated_as_suspect(conn, settings, source, monkeypatch):
"""A genuinely empty list and a broken one are indistinguishable, so assume
the harmless reading."""
add_channel(conn, CHANNEL_ID, "Existing", "Existing")
_fake(monkeypatch, subs=[])
stats = subsync.sync_source(conn, settings, source)
assert stats["refused"]
assert stats["removed"] == 0
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 1
def test_api_not_configured_changes_nothing(conn, settings, source, monkeypatch):
add_channel(conn, CHANNEL_ID, "Existing", "Existing")
_fake(monkeypatch, raises=api.NotConfigured(403, "forbidden", "blocked"))
stats = subsync.sync_source(conn, settings, source)
assert stats["refused"]
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 1
def test_network_error_changes_nothing(conn, settings, source, monkeypatch):
add_channel(conn, CHANNEL_ID, "Existing", "Existing")
_fake(monkeypatch, raises=api.ApiError(0, "network", "connection reset"))
stats = subsync.sync_source(conn, settings, source)
assert stats["refused"]
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 1
def test_refusal_is_recorded_on_the_source(conn, settings, source, monkeypatch):
_fake(monkeypatch, raises=api.ApiError(0, "network", "boom"))
subsync.sync_source(conn, settings, source)
row = subsync.get_source(conn, source["key"])
assert row["last_sync_ok"] == 0
assert row["consecutive_failures"] == 1
assert "boom" in row["last_error"]
def test_three_outages_in_a_row_still_delete_nothing(
conn, settings, source, monkeypatch
):
"""The threshold counts absences from healthy responses, not failures."""
add_channel(conn, CHANNEL_ID, "Existing", "Existing")
_fake(monkeypatch, raises=api.ApiError(0, "network", "down"))
for _ in range(5):
subsync.sync_source(conn, settings, subsync.get_source(conn, source["key"]))
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 1
# -------------------------------------------------------------------- removals
def test_absence_counts_up_but_does_not_delete_below_the_threshold(
conn, settings, source, monkeypatch
):
settings.set("subsync_missing_threshold", "3")
add_channel(conn, CHANNEL_ID, "Doomed", "Doomed")
_fake(monkeypatch, subs=[sub("UCaaaaaaaaaaaaaaaaaaaaaa", "Alpha")])
settings.set("subsync_max_new", "0") # keep the addition path out of this
for expected in (1, 2):
stats = subsync.sync_source(
conn, settings, subsync.get_source(conn, source["key"])
)
assert stats["removed"] == 0
assert stats["pending_removal"] == 1
assert conn.execute(
"SELECT missing_syncs FROM channel WHERE dir_name = 'Doomed'"
).fetchone()[0] == expected
def test_deletion_happens_on_the_threshold_sync(
conn, settings, source, media_root, monkeypatch
):
settings.set("subsync_missing_threshold", "3")
settings.set("subsync_max_new", "0")
doomed = add_channel(conn, CHANNEL_ID, "Doomed", "Doomed")
tree = media_root / "Doomed"
(tree / "Season 2026").mkdir(parents=True)
(tree / "tvshow.nfo").write_text("<tvshow/>")
add_video(conn, doomed["id"], "vid00000001")
_fake(monkeypatch, subs=[sub("UCaaaaaaaaaaaaaaaaaaaaaa", "Alpha")])
for _ in range(3):
stats = subsync.sync_source(
conn, settings, subsync.get_source(conn, source["key"])
)
assert stats["removed"] == 1
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 0
assert not tree.exists()
# The videos went with it, via ON DELETE CASCADE.
assert conn.execute("SELECT COUNT(*) FROM video").fetchone()[0] == 0
def test_reappearing_resets_the_counter(conn, settings, source, monkeypatch):
settings.set("subsync_missing_threshold", "3")
settings.set("subsync_max_new", "0")
add_channel(conn, CHANNEL_ID, "Flaky", "Flaky")
_fake(monkeypatch, subs=[sub("UCaaaaaaaaaaaaaaaaaaaaaa", "Alpha")])
subsync.sync_source(conn, settings, subsync.get_source(conn, source["key"]))
assert conn.execute("SELECT missing_syncs FROM channel").fetchone()[0] == 1
_fake(monkeypatch, subs=[sub(CHANNEL_ID, "Flaky")])
subsync.sync_source(conn, settings, subsync.get_source(conn, source["key"]))
assert conn.execute("SELECT missing_syncs FROM channel").fetchone()[0] == 0
def test_manual_channels_are_never_removed(
conn, settings, source, media_root, monkeypatch
):
settings.set("subsync_missing_threshold", "1")
settings.set("subsync_max_new", "0")
add_channel(conn, CHANNEL_ID, "Pinned", "Pinned", source="manual")
_fake(monkeypatch, subs=[sub("UCaaaaaaaaaaaaaaaaaaaaaa", "Alpha")])
for _ in range(3):
stats = subsync.sync_source(
conn, settings, subsync.get_source(conn, source["key"])
)
assert stats["removed"] == 0
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 1
assert conn.execute("SELECT missing_syncs FROM channel").fetchone()[0] == 0
def test_threshold_of_one_deletes_on_the_first_absence(
conn, settings, source, monkeypatch
):
"""Configurable so tests need not loop; the default stays at 3."""
settings.set("subsync_missing_threshold", "1")
settings.set("subsync_max_new", "0")
add_channel(conn, CHANNEL_ID, "Doomed", "Doomed")
_fake(monkeypatch, subs=[sub("UCaaaaaaaaaaaaaaaaaaaaaa", "Alpha")])
stats = subsync.sync_source(conn, settings, source)
assert stats["removed"] == 1
def test_zero_threshold_is_clamped_to_one(conn, settings, source, monkeypatch):
"""A stored 0 must not mean "delete before any absence is confirmed"."""
settings.set("subsync_missing_threshold", "0")
settings.set("subsync_max_new", "0")
add_channel(conn, CHANNEL_ID, "Doomed", "Doomed")
_fake(monkeypatch, subs=[sub("UCaaaaaaaaaaaaaaaaaaaaaa", "Alpha")])
stats = subsync.sync_source(conn, settings, source)
# Clamped to 1, so the first absence is enough — but it took one absence,
# not zero, and the row was actually observed missing.
assert stats["removed"] == 1
assert stats["seen"] == 1
# --------------------------------------------------------------------- general
def test_unsafe_channel_dir_is_not_deleted(conn, settings, media_root):
"""A blank dir_name must never resolve the delete to the media root."""
row = add_channel(conn, CHANNEL_ID, "Bad", " ")
(media_root / "keepme").mkdir()
from ytstream import strm
assert strm.remove_channel_tree(row) is False
assert (media_root / "keepme").exists()
def test_one_unresolvable_channel_does_not_abort_the_sync(
conn, settings, source, monkeypatch
):
class Exploding(FakeApi):
def channel(self, channel_id):
if channel_id.endswith("bad"):
raise api.ApiError(500, "backendError", "boom")
return {"channel_id": channel_id, "title": "Fine", "description": "",
"handle": None, "avatar_url": None}
monkeypatch.setattr(
subsync.channels, "subscribe_from_sync",
lambda conn, settings, cid, title: (_ for _ in ()).throw(RuntimeError("no"))
if cid.endswith("bad") else add_channel(conn, cid, title, title),
)
patch_api(monkeypatch, subsync,
FakeApi(subs=[sub("UC" + "a" * 19 + "bad", "Bad"),
sub("UC" + "b" * 22, "Good")]))
stats = subsync.sync_source(conn, settings, source)
assert stats["added"] == 1
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 1
def test_sync_all_aggregates_and_counts_refusals(conn, settings, source, monkeypatch):
_fake(monkeypatch, raises=api.ApiError(0, "network", "boom"))
totals = subsync.sync_all(conn, settings)
assert totals["sources"] == 1
assert totals["refused"] == 1
def test_disabled_source_is_skipped(conn, settings, source, monkeypatch):
with conn:
conn.execute("UPDATE source SET enabled = 0")
fake = _fake(monkeypatch, subs=[sub("UCaaaaaaaaaaaaaaaaaaaaaa", "Alpha")])
totals = subsync.sync_all(conn, settings)
assert totals["sources"] == 0
assert fake.subscription_calls == 0
tests/test_videos.py
+209
View File
@@ -0,0 +1,209 @@
"""The video state machine and episode numbering."""
from __future__ import annotations
from datetime import date
import pytest
from ytstream import videos
from conftest import add_channel, add_video
def test_insert_and_get(conn, channel):
row = add_video(conn, channel["id"], "vid00000001")
assert row["state"] == videos.LISTED
assert videos.exists(conn, "vid00000001")
assert not videos.exists(conn, "nosuchvideo")
def test_insert_is_idempotent(conn, channel):
add_video(conn, channel["id"], "vid00000001", title="First")
add_video(conn, channel["id"], "vid00000001", title="Second")
assert conn.execute("SELECT COUNT(*) FROM video").fetchone()[0] == 1
assert videos.get(conn, "vid00000001")["title"] == "First"
def test_aged_out_is_terminal_and_never_revived():
assert videos.AGED_OUT in videos.TERMINAL
assert videos.AGED_OUT in videos.NEVER_REVIVE
# skipped_old is terminal but IS revivable by rescan; conflating the two
# would resurrect months of deleted episodes as new.
assert videos.SKIPPED_OLD in videos.TERMINAL
assert videos.SKIPPED_OLD not in videos.NEVER_REVIVE
def test_no_download_era_states_survive():
"""Materialising a text file cannot fail, so there is no retry ladder."""
for gone in ("pending", "downloading", "downloaded", "failed", "deferred"):
assert gone not in (
videos.LISTED, videos.MATERIALISED, videos.SKIPPED_SHORT,
videos.SKIPPED_LIVE, videos.SKIPPED_OLD, videos.AGED_OUT,
)
def test_mark_aged_out_clears_rel_path_but_keeps_the_row(conn, channel):
add_video(conn, channel["id"], "vid00000001", state=videos.MATERIALISED)
with conn:
conn.execute("UPDATE video SET rel_path = 'a/b.strm' WHERE video_id = ?",
("vid00000001",))
videos.mark_aged_out(conn, "vid00000001")
row = videos.get(conn, "vid00000001")
assert row is not None
assert row["state"] == videos.AGED_OUT
assert row["rel_path"] is None
assert row["deleted_at"]
def test_mark_materialised_records_everything(conn, channel):
add_video(conn, channel["id"], "vid00000001")
videos.mark_materialised(conn, "vid00000001", rel_path="a/b.strm", season=2026,
episode=8120, upload_date="2026-08-12", duration=99,
title="T")
row = videos.get(conn, "vid00000001")
assert (row["state"], row["season"], row["episode"], row["duration"]) == (
videos.MATERIALISED, 2026, 8120, 99
)
def test_set_duration_alone(conn, channel):
add_video(conn, channel["id"], "vid00000001", duration=None)
videos.set_duration(conn, "vid00000001", 1234)
assert videos.get(conn, "vid00000001")["duration"] == 1234
# ------------------------------------------------------------ episode numbers
def test_first_episode_of_a_day(conn, channel):
add_video(conn, channel["id"], "vid00000001")
assert videos.next_episode(
conn, channel["id"], date(2026, 8, 12), "vid00000001"
) == (2026, 8120)
def test_ordinal_counts_existing_rows_for_that_day(conn, channel):
add_video(conn, channel["id"], "vid00000001")
with conn:
conn.execute("UPDATE video SET season = 2026, episode = 8120 "
"WHERE video_id = ?", ("vid00000001",))
add_video(conn, channel["id"], "vid00000002")
assert videos.next_episode(
conn, channel["id"], date(2026, 8, 12), "vid00000002"
) == (2026, 8121)
def test_ordinal_ignores_other_days(conn, channel):
add_video(conn, channel["id"], "vid00000001")
with conn:
conn.execute("UPDATE video SET season = 2026, episode = 8110 "
"WHERE video_id = ?", ("vid00000001",))
add_video(conn, channel["id"], "vid00000002")
assert videos.next_episode(
conn, channel["id"], date(2026, 8, 12), "vid00000002"
) == (2026, 8120)
def test_ordinal_ignores_other_channels(conn, channel):
other = add_channel(conn, "UC" + "z" * 22, "Other", "Other")
add_video(conn, other["id"], "vid00000001")
with conn:
conn.execute("UPDATE video SET season = 2026, episode = 8120 "
"WHERE video_id = ?", ("vid00000001",))
add_video(conn, channel["id"], "vid00000002")
assert videos.next_episode(
conn, channel["id"], date(2026, 8, 12), "vid00000002"
) == (2026, 8120)
def test_ordinal_clamps_at_ten_uploads_a_day(conn, channel):
for index in range(10):
video_id = f"vid{index:08d}"
add_video(conn, channel["id"], video_id)
with conn:
conn.execute("UPDATE video SET season = 2026, episode = ? "
"WHERE video_id = ?", (8120 + index, video_id))
add_video(conn, channel["id"], "vid00000099")
season, episode = videos.next_episode(
conn, channel["id"], date(2026, 8, 12), "vid00000099"
)
assert (season, episode) == (2026, 8129)
def test_a_videos_own_row_does_not_bump_its_ordinal(conn, channel):
"""Re-materialising must produce the same number, not the next one."""
add_video(conn, channel["id"], "vid00000001")
with conn:
conn.execute("UPDATE video SET season = 2026, episode = 8120 "
"WHERE video_id = ?", ("vid00000001",))
assert videos.next_episode(
conn, channel["id"], date(2026, 8, 12), "vid00000001"
) == (2026, 8120)
# --------------------------------------------------------------------- queues
def test_claim_listed_returns_only_listed_rows(conn, channel):
add_video(conn, channel["id"], "listed00001", state=videos.LISTED)
add_video(conn, channel["id"], "done0000001", state=videos.MATERIALISED)
add_video(conn, channel["id"], "short000001", state=videos.SKIPPED_SHORT)
add_video(conn, channel["id"], "aged0000001", state=videos.AGED_OUT)
rows = videos.claim_listed(conn)
assert [row["video_id"] for row in rows] == ["listed00001"]
def test_claim_listed_is_oldest_upload_first(conn, channel):
add_video(conn, channel["id"], "newer000001", upload_date="2026-08-10")
add_video(conn, channel["id"], "older000001", upload_date="2026-01-01")
rows = videos.claim_listed(conn)
assert [row["video_id"] for row in rows] == ["older000001", "newer000001"]
def test_claim_listed_honours_the_limit(conn, channel):
for index in range(5):
add_video(conn, channel["id"], f"vid{index:08d}")
assert len(videos.claim_listed(conn, limit=2)) == 2
def test_materialised_for_channel_is_newest_first(conn, channel):
"""The retention sweep counts down from the newest to honour min_keep_videos."""
add_video(conn, channel["id"], "older000001", upload_date="2026-01-01",
state=videos.MATERIALISED)
add_video(conn, channel["id"], "newer000001", upload_date="2026-08-10",
state=videos.MATERIALISED)
rows = videos.materialised_for_channel(conn, channel["id"])
assert [row["video_id"] for row in rows] == ["newer000001", "older000001"]
def test_queue_depth_and_counts(conn, channel):
add_video(conn, channel["id"], "listed00001", state=videos.LISTED)
add_video(conn, channel["id"], "listed00002", state=videos.LISTED)
add_video(conn, channel["id"], "done0000001", state=videos.MATERIALISED)
assert videos.queue_depth(conn) == 2
counts = videos.counts_by_state(conn)
assert counts[videos.LISTED] == 2
assert counts[videos.MATERIALISED] == 1
def test_deleting_a_channel_cascades_to_its_videos(conn, channel):
add_video(conn, channel["id"], "vid00000001")
with conn:
conn.execute("DELETE FROM channel WHERE id = ?", (channel["id"],))
assert conn.execute("SELECT COUNT(*) FROM video").fetchone()[0] == 0
tests/test_web.py
+146
View File
@@ -0,0 +1,146 @@
"""The admin UI: rendering, auth gating and CSRF.
The templates are f-strings over a dict, so a renamed key is a KeyError at render
time rather than a type error at import time — which is exactly the kind of
breakage that only shows up when somebody opens the page. These tests render every
page for real.
"""
from __future__ import annotations
import pytest
from ytstream import videos
from ytstream.web import auth, templates
from conftest import add_channel, add_video
@pytest.fixture()
def channel_row(conn, settings, channel):
"""The dict shape server.py builds for the channel table."""
add_video(conn, channel["id"], "vid00000001", state=videos.MATERIALISED)
return {
"id": channel["id"],
"title": channel["title"],
"handle": channel["handle"],
"channel_id": channel["channel_id"],
"retention_days": None,
"global_retention": 30,
"last_polled_at": "2026-08-12T14:00:00+00:00",
"last_poll_ok": 1,
"consecutive_poll_failures": 0,
"episodes": 1,
"source": "youtube",
"missing_syncs": 0,
"missing_threshold": 3,
"latest": "2026-08-12",
}
def test_login_page_renders(settings):
html = templates.login_page().decode()
assert "ytstream" in html
assert "youtube-automate" not in html
assert 'type="password"' in html
def test_login_page_shows_an_error():
assert "Wrong" in templates.login_page("Wrong password").decode()
def test_index_page_renders(channel_row):
html = templates.index_page(
channels=[channel_row], settings_values={"retention_days": "30"},
settings_errors={}, csrf="tok", add_error=None, queue_depth=0,
).decode()
assert "clabretro" in html
assert "ytstream" in html
# The provenance column replaced the meaningless size column.
assert "youtube" in html
assert "Size" not in html
def test_index_page_renders_with_no_channels():
html = templates.index_page(
channels=[], settings_values={}, settings_errors={}, csrf="tok",
add_error=None, queue_depth=0,
).decode()
assert "0 channel(s)" in html
def test_absence_badge_appears_before_the_channel_disappears(channel_row):
"""A channel counting towards removal must be visible while it still exists."""
channel_row["missing_syncs"] = 2
html = templates.index_page(
channels=[channel_row], settings_values={}, settings_errors={}, csrf="tok",
add_error=None, queue_depth=0,
).decode()
assert "absent 2/3 syncs" in html
def test_no_absence_badge_when_healthy(channel_row):
html = templates.index_page(
channels=[channel_row], settings_values={}, settings_errors={}, csrf="tok",
add_error=None, queue_depth=0,
).decode()
assert "absent" not in html
def test_hostile_channel_title_is_escaped(channel_row):
channel_row["title"] = '<script>alert("x")</script>'
html = templates.index_page(
channels=[channel_row], settings_values={}, settings_errors={}, csrf="tok",
add_error=None, queue_depth=0,
).decode()
assert "<script>alert" not in html
assert "&lt;script&gt;" in html
def test_settings_errors_are_rendered_inline(channel_row):
html = templates.index_page(
channels=[channel_row], settings_values={"retention_days": "abc"},
settings_errors={"retention_days": "must be a whole number"},
csrf="tok", add_error=None, queue_depth=0,
).decode()
assert "must be a whole number" in html
def test_masked_settings_are_not_rendered_in_plaintext(channel_row):
"""Both API keys are secrets; neither belongs in the HTML."""
html = templates.index_page(
channels=[channel_row],
settings_values={"youtube_api_key": "AIzaSECRETVALUE",
"jellyfin_api_key": "JELLYSECRET"},
settings_errors={}, csrf="tok", add_error=None, queue_depth=0,
).decode()
assert "AIzaSECRETVALUE" not in html
assert "JELLYSECRET" not in html
# ------------------------------------------------------------------------ csrf
def test_csrf_token_round_trips():
secret = auth.new_secret()
session = "session-token"
token = auth.csrf_token(secret, session)
assert auth.verify_csrf(secret, session, token)
def test_csrf_token_is_bound_to_the_session():
secret = auth.new_secret()
token = auth.csrf_token(secret, "session-a")
assert not auth.verify_csrf(secret, "session-b", token)
def test_csrf_token_rejects_tampering():
secret = auth.new_secret()
token = auth.csrf_token(secret, "s")
assert not auth.verify_csrf(secret, "s", token[:-1] + "x")
def test_csrf_rejects_an_empty_token():
secret = auth.new_secret()
assert not auth.verify_csrf(secret, "s", "")
ytstream/__init__.py
+7
View File
@@ -0,0 +1,7 @@
"""ytstream — a just-in-time YouTube library for Jellyfin.
Writes .strm files and metadata; a companion proxy materialises video on demand
when Jellyfin asks for it. No video bytes are ever stored.
"""
__version__ = "1.0.0"
ytstream/__main__.py
+4
View File
@@ -0,0 +1,4 @@
from .cli import main
if __name__ == "__main__":
raise SystemExit(main())
ytstream/api.py
+327
View File
@@ -0,0 +1,327 @@
"""YouTube Data API v3 client.
This is the metadata path. yt-dlp is not involved in cataloguing at all — it is
invoked only by the proxy, for one video, because somebody pressed play. See
plan.md §3 for why that boundary is worth having.
Everything here is a `list` method costing 1 quota unit per call and returning up
to 50 items, against a 10,000/day budget. Measured steady state is ~75 units/day,
so there is no caching layer and no need for one.
`search.list` is never called. It has its own 100-call/day bucket and we have no
use for it.
"""
from __future__ import annotations
import json
import logging
import re
import urllib.error
import urllib.parse
import urllib.request
from datetime import date, datetime, timezone
log = logging.getLogger(__name__)
BASE = "https://www.googleapis.com/youtube/v3"
# Pages are capped at 50 by the API itself.
PAGE_SIZE = 50
# A refusal to page forever if the API ever stops returning a stable
# nextPageToken. 200 pages is 10,000 items, far past anything we ask for.
MAX_PAGES = 200
_DURATION = re.compile(
r"^P(?:(?P<w>\d+)W)?(?:(?P<d>\d+)D)?"
r"(?:T(?:(?P<h>\d+)H)?(?:(?P<m>\d+)M)?(?:(?P<s>\d+)S)?)?$"
)
class ApiError(RuntimeError):
"""Any non-2xx from the API, with the reason Google actually gave."""
def __init__(self, status: int, reason: str, message: str):
super().__init__(f"HTTP {status} {reason}: {message}")
self.status = status
self.reason = reason
self.message = message
class NotConfigured(ApiError):
"""The key is missing, invalid, restricted, or the API is not enabled.
Separate from ApiError because it is an operator problem, not a transient
one: retrying will not help and the caller must not interpret it as "the
remote data changed". See plan.md §16 for the three flavours of this and why
they are hard to tell apart.
"""
class SubscriptionsPrivate(ApiError):
"""403 subscriptionForbidden — the channel's subscriptions are not public.
Its own type because the sync must never read this as "he unsubscribed from
everything" (plan.md §4.4).
"""
def parse_duration(text: str) -> int | None:
"""ISO-8601 duration to seconds. PT1H2M3S -> 3723. None if unparseable.
Live and upcoming videos report PT0S, which parses to 0 rather than None —
the caller distinguishes those by liveStreamingDetails, not by duration.
"""
match = _DURATION.match((text or "").strip())
if not match:
return None
parts = {key: int(value or 0) for key, value in match.groupdict().items()}
return (parts["w"] * 604800 + parts["d"] * 86400
+ parts["h"] * 3600 + parts["m"] * 60 + parts["s"])
def parse_published(text: str) -> tuple[date | None, str | None]:
"""RFC-3339 to (date, original string).
The date is what naming uses; the original is kept because it is exact and
approximate_date is not (plan.md §3).
"""
raw = (text or "").strip()
if not raw:
return None, None
try:
stamp = datetime.fromisoformat(raw.replace("Z", "+00:00"))
except ValueError:
return None, raw
return stamp.astimezone(timezone.utc).date(), raw
def uploads_playlist_id(channel_id: str, kind: str = "UULF") -> str:
"""UULF (long-form only) or UU (everything) for a UC… channel id.
UULF is undocumented but verified to work through playlistItems.list on
2026-08-12, and excludes 54% of a measured catalogue — all Shorts and
livestreams. UU is the documented fallback if YouTube ever retires it.
"""
if not channel_id.startswith("UC"):
raise ValueError(f"not a channel id: {channel_id!r}")
return kind + channel_id[2:]
class Api:
"""Thin client. One instance per run; reads the key at construction."""
def __init__(self, key: str, *, timeout: float = 30.0):
self.key = (key or "").strip()
self.timeout = timeout
self.calls = 0 # quota units spent, for logging and the doctor
# ---------------------------------------------------------------- transport
def _get(self, endpoint: str, **params) -> dict:
if not self.key:
raise NotConfigured(0, "noKey", "no YouTube API key configured")
params["key"] = self.key
url = f"{BASE}/{endpoint}?" + urllib.parse.urlencode(params)
request = urllib.request.Request(url, headers={"Accept": "application/json"})
self.calls += 1
try:
with urllib.request.urlopen(request, timeout=self.timeout) as response:
return json.load(response)
except urllib.error.HTTPError as exc:
raise self._classify(exc) from exc
except OSError as exc:
# Network-level: transient, and explicitly not NotConfigured.
raise ApiError(0, "network", str(exc)) from exc
def _classify(self, exc: urllib.error.HTTPError) -> ApiError:
"""Turn Google's 403s into something a caller can branch on.
The useful signal lives in error.details[].reason, not
error.errors[].reason — both SERVICE_DISABLED and
API_KEY_SERVICE_BLOCKED surface as a bare `forbidden` in the latter.
"""
raw = exc.read().decode("utf8", "replace")
reason = message = ""
details: set[str] = set()
try:
error = json.loads(raw).get("error", {})
message = error.get("message", "")
errors = error.get("errors", [])
reason = errors[0].get("reason", "") if errors else ""
for detail in error.get("details", []):
if detail.get("reason"):
details.add(detail["reason"])
except ValueError:
message = raw[:200]
if reason == "subscriptionForbidden":
return SubscriptionsPrivate(exc.code, reason, message)
if (details & {"SERVICE_DISABLED", "API_KEY_SERVICE_BLOCKED"}
or reason in ("accessNotConfigured", "keyInvalid", "forbidden")
or "has not been used in project" in message
or "are blocked" in message):
return NotConfigured(exc.code, reason or next(iter(details), ""), message)
return ApiError(exc.code, reason, message)
def _paged(self, endpoint: str, **params):
"""Yield items across all pages, stopping when the token runs out."""
token = None
for _ in range(MAX_PAGES):
if token:
params["pageToken"] = token
page = self._get(endpoint, **params)
for item in page.get("items") or []:
yield item
token = page.get("nextPageToken")
if not token:
return
log.warning("%s: stopped paging at %d pages", endpoint, MAX_PAGES)
# ------------------------------------------------------------ subscriptions
def subscriptions(self, channel_id: str) -> list[dict]:
"""A channel's public subscriptions as [{channel_id, title}].
Requires only an API key, provided the account has not ticked "Keep all
my subscriptions private" — verified 2026-08-12.
Deliberately returns the *fetched* list rather than anything derived from
pageInfo.totalResults, which overcounts: it reported 127 against 119
actually returned, because terminated and private channels still count.
"""
out, seen = [], set()
for item in self._paged("subscriptions", part="snippet",
channelId=channel_id, maxResults=PAGE_SIZE):
snippet = item.get("snippet") or {}
resource = snippet.get("resourceId") or {}
sub_id = resource.get("channelId")
if not sub_id or sub_id in seen:
continue
seen.add(sub_id)
out.append({"channel_id": sub_id,
"title": (snippet.get("title") or "").strip() or sub_id})
return out
# ----------------------------------------------------------------- channels
def channel(self, channel_id: str) -> dict | None:
"""Title, description and artwork for one channel."""
items = self._get("channels", part="snippet,contentDetails",
id=channel_id, maxResults=1).get("items") or []
if not items:
return None
item = items[0]
snippet = item.get("snippet") or {}
thumbs = snippet.get("thumbnails") or {}
best = max(thumbs.values(), key=lambda t: t.get("width") or 0, default={})
return {
"channel_id": item.get("id") or channel_id,
"title": (snippet.get("title") or "").strip(),
"description": snippet.get("description") or "",
"handle": (snippet.get("customUrl") or "") or None,
"avatar_url": best.get("url"),
}
def resolve_handle(self, handle: str) -> dict | None:
"""@handle -> channel. Only needed when subscribing by hand."""
handle = handle.strip()
if not handle.startswith("@"):
handle = "@" + handle
items = self._get("channels", part="snippet",
forHandle=handle, maxResults=1).get("items") or []
return self.channel(items[0]["id"]) if items else None
# ------------------------------------------------------------ uploads walk
def uploads(self, channel_id: str, *, kind: str = "UULF",
since: date | None = None, limit: int | None = None,
page_token: str | None = None):
"""Walk a channel's uploads newest-first, yielding (entry, next_token).
`entry` carries the exact publish timestamp, which is the whole reason
this exists rather than `--flat-playlist`: that reports no timestamp at
all, and `approximate_date` is wrong by up to two days, which would put
episodes in the wrong day.
Stops at `since` or `limit`, whichever comes first. Yielding the token
alongside each entry is what makes a bounded backfill resumable — the
caller commits it per page.
"""
playlist_id = uploads_playlist_id(channel_id, kind)
token = page_token
produced = 0
for _ in range(MAX_PAGES):
# Both parts, because they carry different halves of what we need and
# cost the same single unit together as either does alone:
# contentDetails.videoPublishedAt -> when the VIDEO was published
# snippet.title -> the title
# Do not be tempted by snippet.publishedAt: that is when the video was
# added to the playlist, which is not the same thing.
#
# Asking for the title here rather than letting the RSS poll supply it
# is not an optimisation. RSS returns 15 entries, which for a channel
# publishing under one long-form video a day reaches back only ~23
# days — less than the 30-day window — so the oldest few episodes of
# every backfill would otherwise be named after their video id.
params = dict(part="snippet,contentDetails", playlistId=playlist_id,
maxResults=PAGE_SIZE)
if token:
params["pageToken"] = token
page = self._get("playlistItems", **params)
next_token = page.get("nextPageToken")
for item in page.get("items") or []:
details = item.get("contentDetails") or {}
snippet = item.get("snippet") or {}
video_id = details.get("videoId")
if not video_id:
continue
published, exact = parse_published(details.get("videoPublishedAt"))
if published is None:
# A private or deleted video still occupies a playlist slot
# and has no publish date. Skip it rather than stopping.
continue
if since is not None and published < since:
return
title = (snippet.get("title") or "").strip()
# YouTube uses these placeholders for videos that have gone away
# but still hold a playlist slot. Treated as no title at all.
if title in ("Private video", "Deleted video"):
title = ""
yield {"video_id": video_id,
"published": published,
"published_at": exact,
"title": title}, next_token
produced += 1
if limit is not None and produced >= limit:
return
token = next_token
if not token:
return
# ---------------------------------------------------------------- durations
def durations(self, video_ids: list[str]) -> dict[str, dict]:
"""{video_id: {duration, is_live}} for up to any number of ids.
Batched 50 per call, so 441 videos costs 9 units. `is_live` comes from the
presence of liveStreamingDetails rather than from the duration, because
live and upcoming items both report PT0S.
"""
out: dict[str, dict] = {}
for start in range(0, len(video_ids), PAGE_SIZE):
batch = video_ids[start:start + PAGE_SIZE]
page = self._get("videos", part="contentDetails,liveStreamingDetails",
id=",".join(batch), maxResults=PAGE_SIZE)
for item in page.get("items") or []:
details = item.get("contentDetails") or {}
out[item["id"]] = {
"duration": parse_duration(details.get("duration", "")),
"is_live": bool(item.get("liveStreamingDetails")),
}
return out
ytstream/channels.py
+301
View File
@@ -0,0 +1,301 @@
"""Channel resolution, subscribe and unsubscribe."""
from __future__ import annotations
import logging
import re
import shutil
import sqlite3
import subprocess
import tempfile
import urllib.request
from pathlib import Path
from . import config, naming, nfo, util, ytdlp
from .settings import Settings
log = logging.getLogger(__name__)
_CHANNEL_ID = re.compile(r"^UC[A-Za-z0-9_-]{22}$")
_HANDLE = re.compile(r"^@[A-Za-z0-9._-]+$")
# Artwork we try to pull at subscribe time. Best effort — a channel without them
# still works, it just looks plainer in Jellyfin.
_ARTWORK = (("avatar_uncropped", "poster.jpg"), ("banner_uncropped", "fanart.jpg"))
class ResolutionError(RuntimeError):
pass
def normalise_url(text: str) -> str:
"""Turn any accepted channel reference into a URL yt-dlp understands."""
text = (text or "").strip()
if not text:
raise ResolutionError("no channel given")
if text.startswith(("http://", "https://")):
return text
if _CHANNEL_ID.match(text):
return f"https://www.youtube.com/channel/{text}"
if _HANDLE.match(text):
return f"https://www.youtube.com/{text}"
if text.startswith("www.youtube.com") or text.startswith("youtube.com"):
return "https://" + text
# Bare word: assume it's a handle without the @.
if re.match(r"^[A-Za-z0-9._-]+$", text):
return f"https://www.youtube.com/@{text}"
raise ResolutionError(f"could not interpret {text!r} as a channel")
def resolve(settings: Settings, text: str) -> dict:
"""Fetch channel metadata without enumerating the uploads."""
url = normalise_url(text)
args = [
"--flat-playlist",
"--playlist-items",
"0",
"-J",
"--no-warnings",
"--ignore-config",
*ytdlp.extractor_args(settings.get_str("pot_provider_url")),
url,
]
try:
data = ytdlp.run_json(args, timeout=180)
except ytdlp.YtdlpError as exc:
raise ResolutionError(str(exc)) from exc
channel_id = data.get("channel_id") or data.get("id") or ""
if not _CHANNEL_ID.match(channel_id):
raise ResolutionError(f"no channel id found for {url}")
handle = data.get("uploader_id") or ""
if handle and not handle.startswith("@"):
handle = ""
return {
"channel_id": channel_id,
"title": (data.get("channel") or data.get("title") or channel_id).strip(),
"description": data.get("description") or "",
"handle": handle,
"thumbnails": data.get("thumbnails") or [],
}
def uulf_playlist_id(channel_id: str) -> str:
"""Long-form-only uploads playlist for a channel (specs.md §4)."""
return "UULF" + channel_id[2:]
# --------------------------------------------------------------------------
# artwork
def _pick_thumbnail(thumbnails: list[dict], wanted_id: str) -> str | None:
for thumb in thumbnails:
if str(thumb.get("id", "")) == wanted_id and thumb.get("url"):
return thumb["url"]
return None
def _download_image(url: str, destination: Path) -> bool:
"""Fetch an image and normalise it to JPEG via ffmpeg.
YouTube serves avatars as webp as often as jpeg, and naming a webp file
.jpg would be a lie some clients notice.
"""
try:
request = urllib.request.Request(
url, headers={"User-Agent": config.USER_AGENT}
)
with urllib.request.urlopen(request, timeout=60) as response:
payload = response.read()
except OSError as exc:
log.warning("artwork download failed (%s): %s", url, exc)
return False
with tempfile.NamedTemporaryFile(suffix=".img", delete=True) as raw:
raw.write(payload)
raw.flush()
result = subprocess.run(
["ffmpeg", "-y", "-loglevel", "error", "-i", raw.name, str(destination)],
capture_output=True,
text=True,
timeout=120,
)
if result.returncode != 0:
log.warning("artwork conversion failed: %s", result.stderr.strip()[:200])
return False
return True
def write_channel_metadata(channel_dir: Path, info: dict) -> None:
"""Write tvshow.nfo and best-effort artwork into the channel directory."""
channel_dir.mkdir(parents=True, exist_ok=True)
nfo.write(
channel_dir / "tvshow.nfo",
nfo.tvshow_nfo(info["title"], info.get("description"), info["channel_id"]),
)
for thumb_id, filename in _ARTWORK:
url = _pick_thumbnail(info.get("thumbnails") or [], thumb_id)
if url:
_download_image(url, channel_dir / filename)
# --------------------------------------------------------------------------
# subscribe / unsubscribe
def get(conn: sqlite3.Connection, pk: int) -> sqlite3.Row | None:
return conn.execute("SELECT * FROM channel WHERE id = ?", (pk,)).fetchone()
def get_by_channel_id(conn: sqlite3.Connection, channel_id: str) -> sqlite3.Row | None:
return conn.execute(
"SELECT * FROM channel WHERE channel_id = ?", (channel_id,)
).fetchone()
def all_channels(conn: sqlite3.Connection) -> list[sqlite3.Row]:
return conn.execute("SELECT * FROM channel ORDER BY title COLLATE NOCASE").fetchall()
def _unique_dir_name(conn: sqlite3.Connection, base: str) -> str:
"""dir_name is UNIQUE; two channels can legitimately share a title."""
candidate = base
suffix = 2
while conn.execute(
"SELECT 1 FROM channel WHERE dir_name = ?", (candidate,)
).fetchone():
candidate = f"{base} ({suffix})"
suffix += 1
return candidate
def subscribe(conn: sqlite3.Connection, settings: Settings, text: str) -> sqlite3.Row:
"""Resolve, insert and lay down on-disk metadata. Raises ResolutionError."""
info = resolve(settings, text)
existing = get_by_channel_id(conn, info["channel_id"])
if existing:
raise ResolutionError(f"already subscribed to {existing['title']}")
dir_name = _unique_dir_name(
conn, naming.channel_dir_name(info["title"], info["channel_id"])
)
with conn:
cursor = conn.execute(
"INSERT INTO channel (channel_id, handle, title, description, dir_name, added_at) "
"VALUES (?, ?, ?, ?, ?, ?)",
(
info["channel_id"],
info["handle"],
info["title"],
info["description"],
dir_name,
util.utcnow_iso(),
),
)
pk = cursor.lastrowid
write_channel_metadata(config.MEDIA_ROOT / dir_name, info)
log.info("subscribed to %s (%s)", info["title"], info["channel_id"])
return get(conn, pk)
def subscribe_from_sync(
conn: sqlite3.Connection, settings: Settings, channel_id: str, title: str
) -> sqlite3.Row:
"""Subscribe a channel discovered by the subscription mirror.
Distinct from `subscribe()` because it must not touch yt-dlp: cataloguing runs
entirely on the Data API and RSS, and 119 channels' worth of yt-dlp channel
resolution is exactly the residential-IP request burst the design avoids
(plan.md §3). The API already gave us the id and title; one `channels.list`
call fills in description and artwork.
"""
from . import api as ytapi
existing = get_by_channel_id(conn, channel_id)
if existing:
return existing
info = {"channel_id": channel_id, "title": title.strip() or channel_id,
"description": "", "handle": None, "avatar_url": None}
try:
fetched = ytapi.Api(settings.get_str("youtube_api_key")).channel(channel_id)
if fetched and fetched.get("title"):
info = fetched
except ytapi.ApiError as exc:
# A channel we cannot describe is still a channel we can mirror.
log.warning("could not fetch metadata for %s: %s", channel_id, exc)
dir_name = _unique_dir_name(
conn, naming.channel_dir_name(info["title"], channel_id)
)
with conn:
cursor = conn.execute(
"INSERT INTO channel (channel_id, handle, title, description, dir_name, "
" added_at, source) VALUES (?, ?, ?, ?, ?, ?, 'youtube')",
(channel_id, info.get("handle"), info["title"],
info.get("description") or "", dir_name, util.utcnow_iso()),
)
pk = cursor.lastrowid
row = get(conn, pk)
# tvshow.nfo only. The channel directory is created lazily by the first
# episode, so a channel with nothing inside the retention window does not
# leave an empty series in Jellyfin (plan.md §5).
if info.get("avatar_url"):
_write_show_metadata(config.MEDIA_ROOT / dir_name, info)
log.info("subscribed to %s (%s) via sync", info["title"], channel_id)
return row
def _write_show_metadata(channel_dir: Path, info: dict) -> None:
"""tvshow.nfo plus a poster, for a channel resolved through the API."""
channel_dir.mkdir(parents=True, exist_ok=True)
nfo.write(
channel_dir / "tvshow.nfo",
nfo.tvshow_nfo(info["title"], info.get("description"), info["channel_id"]),
)
if info.get("avatar_url"):
_download_image(info["avatar_url"], channel_dir / "poster.jpg")
def refresh_metadata(conn: sqlite3.Connection, settings: Settings, pk: int) -> None:
"""Re-resolve a channel and rewrite tvshow.nfo if the title changed.
dir_name is deliberately never recomputed — channels rename themselves and
we do not want orphaned directories.
"""
row = get(conn, pk)
if row is None:
return
info = resolve(settings, row["channel_id"])
if info["title"] != row["title"] or info["description"] != (row["description"] or ""):
with conn:
conn.execute(
"UPDATE channel SET title = ?, description = ? WHERE id = ?",
(info["title"], info["description"], pk),
)
write_channel_metadata(config.MEDIA_ROOT / row["dir_name"], info)
def unsubscribe(conn: sqlite3.Connection, pk: int) -> str:
"""Hard delete: remove the directory tree, then the rows. Irreversible."""
row = get(conn, pk)
if row is None:
raise LookupError(f"no channel with id {pk}")
title = row["title"]
channel_dir = config.MEDIA_ROOT / row["dir_name"]
if channel_dir.is_dir():
shutil.rmtree(channel_dir, ignore_errors=True)
with conn:
conn.execute("DELETE FROM channel WHERE id = ?", (pk,))
log.info("unsubscribed from %s and removed %s", title, channel_dir)
return title
ytstream/cli.py
+556
View File
@@ -0,0 +1,556 @@
"""Command line entry point."""
from __future__ import annotations
import argparse
import getpass
import sys
from . import (
api,
channels,
config,
db,
discovery,
doctor,
jellyfin,
reap,
runner,
strm,
subsync,
util,
videos,
)
from .settings import Settings
from .web import auth
def _open():
conn = db.connect()
return conn, Settings(conn)
# --------------------------------------------------------------------------
# setup
def cmd_doctor(args) -> int:
conn, settings = _open()
try:
text, code = doctor.report(doctor.run_checks(settings))
print(text)
return code
finally:
conn.close()
def cmd_set_password(args) -> int:
conn, settings = _open()
try:
password = getpass.getpass("New admin password: ")
if len(password) < 8:
print("Password must be at least 8 characters.", file=sys.stderr)
return 1
if password != getpass.getpass("Repeat: "):
print("Passwords did not match.", file=sys.stderr)
return 1
settings.set("admin_password_hash", auth.hash_password(password))
if not settings.raw("session_secret"):
settings.set("session_secret", auth.new_secret())
print("Admin password set.")
return 0
finally:
conn.close()
def cmd_set_jellyfin_key(args) -> int:
conn, settings = _open()
try:
key = (args.key or getpass.getpass("Jellyfin API key: ")).strip()
if not key:
print("No key given.", file=sys.stderr)
return 1
client = jellyfin.Jellyfin(settings.get_str("jellyfin_url"), key)
try:
client.virtual_folders()
except jellyfin.JellyfinError as exc:
print(f"Key rejected by Jellyfin: {exc}", file=sys.stderr)
return 1
settings.set("jellyfin_api_key", key)
print("Jellyfin API key stored and verified.")
return 0
finally:
conn.close()
def cmd_set_youtube_key(args) -> int:
"""Store the Data API key, verifying it first.
Verified rather than trusted because there are three distinct ways for a
fresh Google Cloud project to be wrong and they all present as HTTP 403
`forbidden` (plan.md §16). Storing a key that cannot work would turn an
operator mistake into a mystery an hour later in the cron log.
"""
conn, settings = _open()
try:
key = (args.key or getpass.getpass("YouTube Data API key: ")).strip()
if not key:
print("No key given.", file=sys.stderr)
return 1
client = api.Api(key)
try:
client.durations(["dQw4w9WgXcQ"])
except api.NotConfigured as exc:
print(f"Key not usable: {exc.message or exc}", file=sys.stderr)
print("\nThis is normally one of:", file=sys.stderr)
print(" * YouTube Data API v3 not enabled on the project", file=sys.stderr)
print(" * the key's API restrictions exclude it", file=sys.stderr)
print(" * a console change that has not propagated yet (~2 min)",
file=sys.stderr)
return 1
except api.ApiError as exc:
print(f"Key check failed: {exc}", file=sys.stderr)
return 1
settings.set("youtube_api_key", key)
print("YouTube API key stored and verified.")
return 0
finally:
conn.close()
def cmd_setup_jellyfin_library(args) -> int:
conn, settings = _open()
try:
client = jellyfin.from_settings(settings)
if not client.configured:
print("Set jellyfin_url and jellyfin_api_key first.", file=sys.stderr)
return 1
existing = client.find_library(config.MEDIA_ROOT)
if existing:
print(f"Library already exists: {existing.get('Name')}")
return 0
client.create_library(config.MEDIA_ROOT)
print(f"Created Jellyfin library for {config.MEDIA_ROOT}.")
return 0
finally:
conn.close()
# --------------------------------------------------------------------------
# subscription sources
def cmd_add_source(args) -> int:
"""Register a YouTube account whose subscriptions we mirror."""
conn, settings = _open()
try:
client = api.Api(settings.get_str("youtube_api_key"))
reference = args.channel.strip()
try:
info = (client.channel(reference) if reference.startswith("UC")
else client.resolve_handle(reference))
except api.ApiError as exc:
print(f"Could not resolve {reference}: {exc}", file=sys.stderr)
return 1
if not info:
print(f"No such channel: {reference}", file=sys.stderr)
return 1
# Prove the subscriptions are actually readable before storing anything.
# A source that 403s is worse than no source: it looks configured.
try:
found = client.subscriptions(info["channel_id"])
except api.SubscriptionsPrivate:
print(f"{info['title']}'s subscriptions are private.", file=sys.stderr)
print('Fix: YouTube -> Settings -> Privacy -> uncheck "Keep all my '
'subscriptions private".', file=sys.stderr)
return 1
except api.ApiError as exc:
print(f"Could not read subscriptions: {exc}", file=sys.stderr)
return 1
key = subsync.add_source(
conn, channel_id=info["channel_id"], label=info["title"]
)
print(f"Added source {key} ({info['title']}) — {len(found)} subscriptions.")
print("Run `ytstream sync` to queue them for approval.")
return 0
finally:
conn.close()
def cmd_sources(args) -> int:
conn, _ = _open()
try:
rows = subsync.all_sources(conn)
if not rows:
print("No subscription sources. Add one with `ytstream add-source`.")
return 0
for row in rows:
state = "ok" if row["last_sync_ok"] else "FAILING"
if row["last_sync_ok"] is None:
state = "never synced"
print(f"{row['key']} {row['label']} [{state}]")
print(f" last sync: {row['last_sync_at'] or 'never'} "
f"failures: {row['consecutive_failures']} "
f"imported: {'yes' if row['imported'] else 'no'}")
if row["last_error"]:
print(f" last error: {row['last_error'][:120]}")
return 0
finally:
conn.close()
def cmd_sync(args) -> int:
conn, settings = _open()
try:
totals = subsync.sync_all(conn, settings)
print(f"sources={totals['sources']} added={totals['added']} "
f"queued={totals['queued']} removed={totals['removed']} "
f"pending_removal={totals['pending_removal']} "
f"refused={totals['refused']}")
return 1 if totals["refused"] else 0
finally:
conn.close()
def cmd_pending(args) -> int:
conn, _ = _open()
try:
rows = subsync.pending(conn)
if not rows:
print("Nothing awaiting approval.")
return 0
print(f"{len(rows)} channel(s) awaiting approval:")
for row in rows:
print(f"{row['id']:>5} {row['title'][:50]:<50} {row['channel_id']}")
print("\nApprove with: ytstream approve <id> [<id>...] | --all")
return 0
finally:
conn.close()
def cmd_approve(args) -> int:
conn, settings = _open()
try:
if args.all:
ids = [row["id"] for row in subsync.pending(conn)]
else:
ids = args.ids
if not ids:
print("Nothing to approve.")
return 0
stats = subsync.approve(conn, settings, ids)
print(f"added={stats['added']} failed={stats['failed']}")
return 1 if stats["failed"] else 0
finally:
conn.close()
def cmd_reject(args) -> int:
conn, _ = _open()
try:
ids = ([row["id"] for row in subsync.pending(conn)] if args.all else args.ids)
count = subsync.resolve(conn, ids, "rejected")
print(f"rejected {count} channel(s); they will not be queued again")
return 0
finally:
conn.close()
# --------------------------------------------------------------------------
# channels and catalogue
def cmd_subscribe(args) -> int:
"""Subscribe by hand. Marked `manual`, so the sync never removes it."""
conn, settings = _open()
try:
try:
row = channels.subscribe(conn, settings, args.url)
except channels.ResolutionError as exc:
print(str(exc), file=sys.stderr)
return 1
with conn:
conn.execute("UPDATE channel SET source = 'manual' WHERE id = ?",
(row["id"],))
print(f"Subscribed to {row['title']} (id {row['id']}, pinned as manual).")
return 0
finally:
conn.close()
def cmd_unsubscribe(args) -> int:
conn, settings = _open()
try:
row = channels.get(conn, args.id)
if row is None:
print(f"No channel with id {args.id}.", file=sys.stderr)
return 1
if not args.yes:
answer = input(f"Delete {row['title']} and its whole tree? [y/N] ")
if answer.strip().lower() not in ("y", "yes"):
print("Cancelled.")
return 0
title = channels.unsubscribe(conn, args.id)
jellyfin.from_settings(settings).refresh()
print(f"Unsubscribed from {title}.")
return 0
finally:
conn.close()
def cmd_channels(args) -> int:
conn, _ = _open()
try:
rows = channels.all_channels(conn)
if not rows:
print("No channels subscribed.")
return 0
print(f"{'id':>4} {'title':<34} {'src':<7} {'eps':>4} {'miss':>4} last poll")
for row in rows:
count = conn.execute(
"SELECT COUNT(*) FROM video WHERE channel_pk = ? AND state = ?",
(row["id"], videos.MATERIALISED),
).fetchone()[0]
print(f"{row['id']:>4} {row['title'][:34]:<34} {row['source']:<7} "
f"{count:>4} {row['missing_syncs']:>4} "
f"{row['last_polled_at'] or 'never'}")
return 0
finally:
conn.close()
def cmd_poll(args) -> int:
conn, settings = _open()
try:
if args.rescan:
rows = ([channels.get(conn, args.channel)] if args.channel
else channels.all_channels(conn))
total = sum(discovery.rescan_channel(conn, settings, row)
for row in rows if row)
print(f"re-queued {total} previously-skipped video(s)")
totals = discovery.poll_all(conn, settings, args.channel)
print(f"channels={totals['channels']} queued={totals['queued']} "
f"known={totals['known']} outside-window={totals['old']} "
f"repaired={totals['repaired']} titled={totals['titled']} "
f"shorts={totals['shorts']} live={totals['live']} "
f"failures={totals['failed']}")
return 0
finally:
conn.close()
def cmd_materialise(args) -> int:
conn, settings = _open()
try:
if args.all:
# Rebuild the tree from the database. This is the supported recovery
# from a Jellyfin "replace all metadata" accident — never a refresh
# against YouTube.
with conn:
conn.execute("UPDATE video SET state = ? WHERE state = ?",
(videos.LISTED, videos.MATERIALISED))
stats = runner.materialise_all(conn, settings, args.limit)
print(f"materialised={stats['materialised']} shows={stats['shows']} "
f"errors={stats['errors']}")
return 1 if stats["errors"] else 0
finally:
conn.close()
def cmd_reap(args) -> int:
conn, settings = _open()
try:
result = reap.run(conn, settings)
print(f"aged_out={result['aged_out']}")
return 0
finally:
conn.close()
def cmd_run(args) -> int:
try:
with runner.exclusive_lock():
conn, settings = _open()
try:
result = runner.run(conn, settings, args.channel)
print(runner.summarise(result))
return runner.exit_code(result)
finally:
conn.close()
except runner.AlreadyRunning:
# Expected when a long first import outlasts the hourly cron tick.
return 0
def cmd_serve(args) -> int:
from .web import server
server.serve(args.host, args.port, secure_cookies=not args.insecure_cookies)
return 0
def cmd_set_retention(args) -> int:
conn, _ = _open()
try:
row = channels.get(conn, args.id)
if row is None:
print(f"No channel with id {args.id}.", file=sys.stderr)
return 1
if args.days.lower() in ("default", "none", "clear"):
value = None
else:
try:
value = int(args.days)
except ValueError:
print("days must be a whole number or 'default'.", file=sys.stderr)
return 1
if value < 1:
print("days must be at least 1.", file=sys.stderr)
return 1
with conn:
conn.execute("UPDATE channel SET retention_days = ? WHERE id = ?",
(value, args.id))
shown = "the global default" if value is None else f"{value} days"
print(f"{row['title']} retention set to {shown}.")
return 0
finally:
conn.close()
def cmd_status(args) -> int:
conn, settings = _open()
try:
counts = videos.counts_by_state(conn)
total = conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0]
print(f"channels: {total}")
for state in (videos.MATERIALISED, videos.LISTED, videos.SKIPPED_SHORT,
videos.SKIPPED_LIVE, videos.SKIPPED_OLD, videos.AGED_OUT):
print(f" {state:<14} {counts.get(state, 0)}")
print(f"last run: {settings.raw('last_run_at') or 'never'}")
awaiting = len(subsync.pending(conn))
if awaiting:
print(f"awaiting approval: {awaiting} (see `ytstream pending`)")
return 0
finally:
conn.close()
# --------------------------------------------------------------------------
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
prog="ytstream",
description="A just-in-time YouTube library for Jellyfin. Stores no video.",
)
parser.add_argument("-v", "--verbose", action="store_true", help="debug logging")
sub = parser.add_subparsers(dest="command", required=True)
sub.add_parser("doctor", help="check the installation").set_defaults(
func=cmd_doctor)
sub.add_parser("status", help="counts by state").set_defaults(func=cmd_status)
sub.add_parser("set-password", help="set the admin UI password").set_defaults(
func=cmd_set_password)
jkey = sub.add_parser("set-jellyfin-key", help="store and verify the Jellyfin key")
jkey.add_argument("key", nargs="?", help="omit to be prompted (preferred)")
jkey.set_defaults(func=cmd_set_jellyfin_key)
ykey = sub.add_parser("set-youtube-key",
help="store and verify the YouTube Data API key")
ykey.add_argument("key", nargs="?", help="omit to be prompted (preferred)")
ykey.set_defaults(func=cmd_set_youtube_key)
sub.add_parser("setup-jellyfin-library",
help="create the Shows library for the media root").set_defaults(
func=cmd_setup_jellyfin_library)
source = sub.add_parser("add-source",
help="mirror a YouTube account's subscriptions")
source.add_argument("channel", help="UC... id or @handle")
source.set_defaults(func=cmd_add_source)
sub.add_parser("sources", help="list subscription sources").set_defaults(
func=cmd_sources)
sub.add_parser("sync", help="pull subscriptions from every source").set_defaults(
func=cmd_sync)
sub.add_parser("pending", help="channels awaiting approval").set_defaults(
func=cmd_pending)
approve = sub.add_parser("approve", help="subscribe approved channels")
approve.add_argument("ids", type=int, nargs="*")
approve.add_argument("--all", action="store_true", help="approve everything queued")
approve.set_defaults(func=cmd_approve)
reject = sub.add_parser("reject", help="never queue these channels again")
reject.add_argument("ids", type=int, nargs="*")
reject.add_argument("--all", action="store_true")
reject.set_defaults(func=cmd_reject)
subscribe = sub.add_parser("subscribe",
help="subscribe by hand (pinned, never auto-removed)")
subscribe.add_argument("url", help="channel URL, @handle, or UC... id")
subscribe.set_defaults(func=cmd_subscribe)
unsubscribe = sub.add_parser("unsubscribe",
help="remove a channel and its whole tree")
unsubscribe.add_argument("id", type=int, help="channel id from `channels`")
unsubscribe.add_argument("--yes", action="store_true", help="skip confirmation")
unsubscribe.set_defaults(func=cmd_unsubscribe)
sub.add_parser("channels", help="list subscribed channels").set_defaults(
func=cmd_channels)
poll = sub.add_parser("poll", help="discover new videos")
poll.add_argument("--channel", type=int, help="restrict to one channel id")
poll.add_argument("--rescan", action="store_true",
help="also re-queue videos skipped as too old that the "
"current window now covers (never revives aged-out ones)")
poll.set_defaults(func=cmd_poll)
mat = sub.add_parser("materialise", help="write .strm/.nfo for listed videos")
mat.add_argument("--limit", type=int)
mat.add_argument("--all", action="store_true",
help="rewrite every episode from the database — the recovery "
"path after a Jellyfin metadata wipe")
mat.set_defaults(func=cmd_materialise)
sub.add_parser("reap", help="delete videos past the retention window").set_defaults(
func=cmd_reap)
run_cmd = sub.add_parser("run", help="sync, poll, materialise, reap (what cron calls)")
run_cmd.add_argument("--channel", type=int, help="restrict to one channel")
run_cmd.set_defaults(func=cmd_run)
serve = sub.add_parser("serve", help="run the admin web server")
serve.add_argument("--host", default="127.0.0.1")
serve.add_argument("--port", type=int, default=8086)
serve.add_argument("--insecure-cookies", action="store_true",
help="omit the Secure cookie flag (local http testing only)")
serve.set_defaults(func=cmd_serve)
retention = sub.add_parser("set-retention",
help="set or clear a channel's retention override")
retention.add_argument("id", type=int, help="channel id")
retention.add_argument("days", help="number of days, or 'default' to clear")
retention.set_defaults(func=cmd_set_retention)
return parser
def main(argv: list[str] | None = None) -> int:
parser = build_parser()
args = parser.parse_args(argv)
util.setup_logging(args.verbose)
util.apply_umask()
return args.func(args)
ytstream/config.py
+36
View File
@@ -0,0 +1,36 @@
"""Filesystem paths and process-level constants.
Every path is overridable through the environment so the test suite can point the
whole application at a tmpdir without touching the real media tree.
"""
from __future__ import annotations
import os
from pathlib import Path
def _path(env: str, default: str) -> Path:
return Path(os.environ.get(env, default))
STATE_DIR = _path("YTS_STATE_DIR", "/var/lib/ytstream")
MEDIA_ROOT = _path("YTS_MEDIA_ROOT", "/disks/Plex/_ytstream")
DB_PATH = _path("YTS_DB_PATH", str(STATE_DIR / "ytstream.db"))
LOCK_PATH = _path("YTS_LOCK_PATH", str(STATE_DIR / "run.lock"))
VENV_BIN = _path("YTS_VENV_BIN", str(STATE_DIR / "venv" / "bin"))
# Files created by the service must stay group-readable by `mediaserver`, which is
# how Jellyfin reaches the tree. See plan.md §9.
UMASK = 0o002
# Sidecars we own and may therefore delete when a video ages out. The `.strm` is
# the media file itself and is handled separately, so it is deliberately absent.
SIDECAR_SUFFIXES = (".nfo", "-thumb.jpg")
USER_AGENT = "ytstream/1.0"
# There is deliberately no `.work/` dir under the media root. The proxy's scratch
# space is tmpfs, outside the library entirely, so ytstream needs neither the
# dot-prefix nor the `.ignore` file youtube-automate relies on to hide it.
ytstream/db.py
+149
View File
@@ -0,0 +1,149 @@
"""SQLite access and schema migrations.
WAL is mandatory: the hourly cron job and the long-running admin server both write.
This is schema version 1 of a new database, not a migration of
youtube-automate's. The two differ enough — no download bookkeeping, a
subscription source, an approval queue — that carrying the old rows across would
be more code than the two channels and eighteen rows in it are worth (plan.md §7).
"""
from __future__ import annotations
import sqlite3
from pathlib import Path
from . import config
SCHEMA_VERSION = 1
_SCHEMA_V1 = """
CREATE TABLE IF NOT EXISTS channel (
id INTEGER PRIMARY KEY,
channel_id TEXT NOT NULL UNIQUE,
handle TEXT,
title TEXT NOT NULL,
description TEXT,
dir_name TEXT NOT NULL UNIQUE,
added_at TEXT NOT NULL,
backfilled INTEGER NOT NULL DEFAULT 0,
-- Which uploads playlist won for this channel: 'UULF' (long-form only) or
-- 'UU' (everything, needs duration filtering). Verified 2026-08-12 that the
-- API accepts UULF, so this is normally 'UULF'.
uploads_playlist TEXT NOT NULL DEFAULT 'UULF',
-- Resume point for a bounded backfill: the pageToken of the next page to
-- fetch, or NULL when the backfill is complete.
backfill_cursor TEXT,
-- Per-channel override of the global retention window. NULL = use the setting.
retention_days INTEGER,
-- Provenance. 'youtube' = came from the mirrored subscription list and may be
-- removed by a sync. 'manual' = pinned by hand and never auto-removed.
source TEXT NOT NULL DEFAULT 'youtube',
-- Consecutive syncs in which this channel was absent from an otherwise-healthy
-- subscription response. Only at the threshold is it unsubscribed (plan.md §4.4).
missing_syncs INTEGER NOT NULL DEFAULT 0,
last_polled_at TEXT,
last_poll_ok INTEGER,
consecutive_poll_failures INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE IF NOT EXISTS video (
id INTEGER PRIMARY KEY,
video_id TEXT NOT NULL UNIQUE,
channel_pk INTEGER NOT NULL REFERENCES channel(id) ON DELETE CASCADE,
title TEXT,
-- Date-only, what naming derives season/episode from.
upload_date TEXT,
-- Exact RFC-3339 from playlistItems.list. Kept because approximate_date is
-- wrong by up to 2 days and episode numbers depend on the date (plan.md §3).
published_at TEXT,
duration INTEGER,
season INTEGER,
episode INTEGER,
state TEXT NOT NULL,
discovery_source TEXT NOT NULL,
-- Path of the .strm relative to the media root, NULL once aged out.
rel_path TEXT,
discovered_at TEXT NOT NULL,
materialised_at TEXT,
deleted_at TEXT
);
CREATE INDEX IF NOT EXISTS idx_video_state ON video(state);
CREATE INDEX IF NOT EXISTS idx_video_upload_date ON video(upload_date);
CREATE INDEX IF NOT EXISTS idx_video_channel ON video(channel_pk);
-- One row per mirrored YouTube account. In practice exactly one, but a table
-- rather than six setting keys because these are fields of one thing that change
-- together (plan.md §7).
CREATE TABLE IF NOT EXISTS source (
key TEXT PRIMARY KEY,
label TEXT NOT NULL,
channel_id TEXT NOT NULL,
enabled INTEGER NOT NULL DEFAULT 1,
imported INTEGER NOT NULL DEFAULT 0,
last_sync_at TEXT,
last_sync_ok INTEGER,
consecutive_failures INTEGER NOT NULL DEFAULT 0,
last_error TEXT
);
-- Channels awaiting a human decision: the whole list on a source's first sync,
-- and anything over subsync_max_new thereafter.
CREATE TABLE IF NOT EXISTS pending_approval (
id INTEGER PRIMARY KEY,
source TEXT NOT NULL,
channel_id TEXT NOT NULL,
title TEXT NOT NULL,
seen_at TEXT NOT NULL,
resolved_at TEXT,
resolution TEXT,
UNIQUE (source, channel_id)
);
CREATE INDEX IF NOT EXISTS idx_pending_unresolved
ON pending_approval(resolved_at);
CREATE TABLE IF NOT EXISTS setting (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
"""
def connect(path: Path | None = None) -> sqlite3.Connection:
"""Open the database, applying migrations if needed."""
path = Path(path) if path is not None else config.DB_PATH
path.parent.mkdir(parents=True, exist_ok=True)
conn = sqlite3.connect(path, timeout=30.0, isolation_level=None)
conn.row_factory = sqlite3.Row
conn.execute("PRAGMA journal_mode = WAL")
conn.execute("PRAGMA foreign_keys = ON")
conn.execute("PRAGMA busy_timeout = 30000")
migrate(conn)
return conn
def migrate(conn: sqlite3.Connection) -> int:
"""Bring the schema up to SCHEMA_VERSION. Idempotent."""
current = conn.execute("PRAGMA user_version").fetchone()[0]
if current >= SCHEMA_VERSION:
return current
with conn:
if current < 1:
conn.executescript(_SCHEMA_V1)
# Future migrations append here, each guarded by `if current < N`.
conn.execute(f"PRAGMA user_version = {SCHEMA_VERSION}")
return SCHEMA_VERSION
ytstream/discovery.py
+422
View File
@@ -0,0 +1,422 @@
"""Discovery: RSS polling for what's new, the Data API for the backfill.
Two sources, chosen on cost:
* **RSS** (`feeds/videos.xml?playlist_id=UULF…`) is free, unauthenticated and
carries exact publish timestamps. It is the steady-state poller. Measured
2026-08-12: all 119 subscribed channels poll in 7.9 s, and for a channel
publishing under one long-form video a day the 15-entry feed spans ~23 days —
so RSS alone very nearly covers a 30-day retention window.
* **The Data API** is for the backfill on subscribe, where RSS does not reach far
enough, and for durations, which RSS does not carry at all.
The UULF playlist (`UU` with `LF` spliced in) is undocumented but excludes Shorts
and livestreams at the cheapest possible point — verified both through RSS and,
on 2026-08-12, through `playlistItems.list`. On a measured channel it excludes 54%
of the catalogue. The `channel_id` feed is the fallback, and rows discovered that
way carry `discovery_source='uc_feed'` so the duration pass knows to apply the
Shorts and livestream filters itself.
"""
from __future__ import annotations
import logging
import sqlite3
import urllib.error
import urllib.request
import xml.etree.ElementTree as ET
from datetime import date, timedelta
from . import api, channels, config, strm, util, videos
from .settings import Settings
log = logging.getLogger(__name__)
NS = {
"atom": "http://www.w3.org/2005/Atom",
"yt": "http://www.youtube.com/xml/schemas/2015",
"media": "http://search.yahoo.com/mrss/",
}
FEED_BASE = "https://www.youtube.com/feeds/videos.xml"
class FeedUnavailable(Exception):
"""The feed could not be fetched at all (network/5xx). Not the same as 404."""
def uulf_feed_url(channel_id: str) -> str:
return f"{FEED_BASE}?playlist_id={channels.uulf_playlist_id(channel_id)}"
def uc_feed_url(channel_id: str) -> str:
return f"{FEED_BASE}?channel_id={channel_id}"
def fetch_feed(url: str, timeout: float = 30.0) -> bytes | None:
"""Return the feed body, or None if YouTube says it doesn't exist.
A 404 on UULF means "no such playlist", i.e. the channel has no long-form
videos at all — that is not an error. Two of 119 measured channels return
hard errors here because they were terminated or made private while still
appearing in the subscription list.
"""
request = urllib.request.Request(url, headers={"User-Agent": config.USER_AGENT})
try:
with urllib.request.urlopen(request, timeout=timeout) as response:
return response.read()
except urllib.error.HTTPError as exc:
if exc.code == 404:
return None
raise FeedUnavailable(f"HTTP {exc.code}") from exc
except OSError as exc:
raise FeedUnavailable(str(exc)) from exc
def parse_entries(payload: bytes) -> list[dict]:
"""Parse an Atom feed into video dicts. The feed carries no duration.
Entries are read from `atom:entry` elements only. The feed also has a
top-level `published` (the playlist's own creation date, which can be years
old); scraping timestamps with a regex instead picks that up and produces
nonsense upload rates.
"""
try:
root = ET.fromstring(payload)
except ET.ParseError as exc:
raise FeedUnavailable(f"unparseable feed: {exc}") from exc
entries = []
for entry in root.findall("atom:entry", NS):
video_id = entry.findtext("yt:videoId", "", NS)
if not video_id:
continue
published = entry.findtext("atom:published", "", NS)
try:
published_date = date.fromisoformat(published[:10])
except ValueError:
continue
entries.append(
{
"video_id": video_id,
"title": (entry.findtext("atom:title", "", NS) or "").strip(),
"published": published_date,
"published_at": published,
}
)
return entries
def effective_retention_days(settings: Settings, channel: sqlite3.Row) -> int:
override = channel["retention_days"] if "retention_days" in channel.keys() else None
if override:
return int(override)
return settings.get_int("retention_days")
# --------------------------------------------------------------------------
# recording
def _record(
conn: sqlite3.Connection,
channel: sqlite3.Row,
entry: dict,
source: str,
cutoff: date,
) -> str:
"""Insert or repair one discovered video. Returns what happened."""
existing = videos.get(conn, entry["video_id"])
if existing is not None:
# An aged-out row is a tombstone. Touching it here is what would make the
# retention sweep and the poller fight each other forever.
if existing["state"] in videos.NEVER_REVIVE:
return "known"
# The only repair we perform: a video the fallback path rejected as too
# short, later confirmed long-form by the authoritative UULF feed.
if (
source == videos.SOURCE_UULF
and existing["state"] == videos.SKIPPED_SHORT
and existing["discovery_source"] == videos.SOURCE_UC
):
videos.set_state(conn, entry["video_id"], videos.LISTED)
with conn:
conn.execute(
"UPDATE video SET discovery_source = ? WHERE video_id = ?",
(videos.SOURCE_UULF, entry["video_id"]),
)
log.info("re-queued %s: UULF confirms it is long-form", entry["video_id"])
return "repaired"
# A row can still arrive here untitled — a video that was private when the
# backfill saw it, for instance. If a later feed supplies the title, take
# it, and if the episode is already on disk under its video id, remove the
# files and re-queue so it is rewritten under the real name.
if not (existing["title"] or "").strip() and entry["title"]:
with conn:
conn.execute(
"UPDATE video SET title = ? WHERE video_id = ?",
(entry["title"], entry["video_id"]),
)
if existing["state"] == videos.MATERIALISED:
strm.remove(existing)
videos.set_state(conn, entry["video_id"], videos.LISTED)
log.info("%s: title arrived late, re-materialising",
entry["video_id"])
return "titled"
return "known"
state = videos.LISTED if entry["published"] >= cutoff else videos.SKIPPED_OLD
videos.insert(
conn,
channel_pk=channel["id"],
video_id=entry["video_id"],
title=entry["title"],
upload_date=entry["published"].isoformat(),
published_at=entry.get("published_at"),
state=state,
discovery_source=source,
)
return "queued" if state == videos.LISTED else "old"
def _feed_entries(channel: sqlite3.Row) -> tuple[list[dict], str]:
"""UULF if it has anything, else the channel feed. Raises FeedUnavailable."""
payload = fetch_feed(uulf_feed_url(channel["channel_id"]))
entries = parse_entries(payload) if payload else []
if entries:
return entries, videos.SOURCE_UULF
log.warning("UULF feed empty for %s, falling back to channel_id feed",
channel["title"])
payload = fetch_feed(uc_feed_url(channel["channel_id"]))
return (parse_entries(payload) if payload else []), videos.SOURCE_UC
def poll_channel(
conn: sqlite3.Connection, settings: Settings, channel: sqlite3.Row
) -> dict:
"""Poll one channel. Never raises for feed problems — records them instead."""
stats = {"queued": 0, "old": 0, "known": 0, "repaired": 0, "titled": 0,
"source": None}
try:
entries, source = _feed_entries(channel)
except FeedUnavailable as exc:
_record_poll_failure(conn, channel, str(exc))
stats["error"] = str(exc)
return stats
cutoff = util.today() - timedelta(
days=effective_retention_days(settings, channel)
)
fresh: list[str] = []
for entry in entries:
outcome = _record(conn, channel, entry, source, cutoff)
stats[outcome] += 1
if outcome == "queued":
fresh.append(entry["video_id"])
# Only the fallback path needs filtering — UULF is pre-filtered by YouTube —
# but every new video needs a duration for its NFO either way.
if fresh:
stats.update(enrich_durations(conn, settings, fresh))
stats["source"] = source
_record_poll_success(conn, channel)
return stats
def _record_poll_success(conn: sqlite3.Connection, channel: sqlite3.Row) -> None:
with conn:
conn.execute(
"UPDATE channel SET last_polled_at = ?, last_poll_ok = 1, "
"consecutive_poll_failures = 0 WHERE id = ?",
(util.utcnow_iso(), channel["id"]),
)
def _record_poll_failure(conn: sqlite3.Connection, channel: sqlite3.Row, error: str) -> None:
log.error("poll failed for %s: %s", channel["title"], error)
with conn:
conn.execute(
"UPDATE channel SET last_polled_at = ?, last_poll_ok = 0, "
"consecutive_poll_failures = consecutive_poll_failures + 1 WHERE id = ?",
(util.utcnow_iso(), channel["id"]),
)
# --------------------------------------------------------------------------
# durations
def enrich_durations(
conn: sqlite3.Connection, settings: Settings, video_ids: list[str]
) -> dict:
"""Fill in durations from the API, and filter Shorts and livestreams.
RSS carries no duration, so without this every episode would show a runtime
of zero until it had been played once. Batched 50 per call, so the whole
441-video library costs 9 quota units.
"""
stats = {"resolved": 0, "shorts": 0, "live": 0}
if not video_ids:
return stats
minimum = settings.get_int("min_duration_seconds")
client = api.Api(settings.get_str("youtube_api_key"))
try:
found = client.durations(video_ids)
except api.ApiError as exc:
# Durations are an enrichment, not a gate: a NULL duration costs a runtime
# display, not a working library.
log.warning("duration lookup failed for %d video(s): %s", len(video_ids), exc)
return stats
for video_id, info in found.items():
videos.set_duration(conn, video_id, info["duration"])
stats["resolved"] += 1
if info["is_live"]:
videos.set_state(conn, video_id, videos.SKIPPED_LIVE)
stats["live"] += 1
elif info["duration"] is not None and info["duration"] < minimum:
videos.set_state(conn, video_id, videos.SKIPPED_SHORT)
stats["shorts"] += 1
return stats
# --------------------------------------------------------------------------
# backfill
def _set_cursor(conn: sqlite3.Connection, channel_pk: int, token: str | None) -> None:
with conn:
conn.execute(
"UPDATE channel SET backfill_cursor = ? WHERE id = ?", (token, channel_pk)
)
def backfill_channel(
conn: sqlite3.Connection, settings: Settings, channel: sqlite3.Row
) -> dict:
"""Walk a channel's uploads back to the retention window, via the Data API.
This replaces youtube-automate's yt-dlp backfill for one reason above all:
`--flat-playlist` reports no timestamp, and `approximate_date` is wrong by up
to two days. Season and episode derive from the upload date and Jellyfin
caches episode numbers, so a date wrong by a day is a mistake that can only be
fixed by wiping metadata. `playlistItems.list` returns the exact publish time.
Resumable: the page token is committed per page, so a crash mid-backfill
resumes rather than restarting the channel.
"""
stats = {"queued": 0, "pages": 0, "resolved": 0, "shorts": 0, "live": 0}
cutoff = util.today() - timedelta(
days=effective_retention_days(settings, channel)
)
limit = settings.get_int("backfill_max_videos") or None
client = api.Api(settings.get_str("youtube_api_key"))
kind = channel["uploads_playlist"] or "UULF"
fresh: list[str] = []
try:
for entry, next_token in client.uploads(
channel["channel_id"], kind=kind, since=cutoff, limit=limit,
page_token=channel["backfill_cursor"],
):
if not videos.exists(conn, entry["video_id"]):
videos.insert(
conn,
channel_pk=channel["id"],
video_id=entry["video_id"],
# The title comes from the same call as the date. Relying on
# the RSS poll instead left the oldest ~5 of every 20-episode
# backfill named after their video id, because RSS reaches
# back only ~23 days against a 30-day window.
title=entry.get("title") or "",
upload_date=entry["published"].isoformat(),
published_at=entry["published_at"],
state=videos.LISTED,
discovery_source=videos.SOURCE_BACKFILL,
)
fresh.append(entry["video_id"])
stats["queued"] += 1
_set_cursor(conn, channel["id"], next_token)
stats["pages"] += 1
except api.NotConfigured as exc:
# No key yet. Leave `backfilled` unset so it retries once configured.
log.error("backfill skipped for %s: %s", channel["title"], exc)
stats["error"] = str(exc)
return stats
except api.ApiError as exc:
log.error("backfill failed for %s: %s", channel["title"], exc)
stats["error"] = str(exc)
return stats
stats.update(enrich_durations(conn, settings, fresh))
_set_cursor(conn, channel["id"], None)
with conn:
conn.execute("UPDATE channel SET backfilled = 1 WHERE id = ?",
(channel["id"],))
return stats
def rescan_channel(
conn: sqlite3.Connection, settings: Settings, channel: sqlite3.Row
) -> int:
"""Re-queue `skipped_old` rows that a widened retention window now covers.
`skipped_old` is judged against whatever window was in force at discovery
time and is otherwise terminal. Without this, raising retention_days would
appear to do nothing for an infrequent uploader.
`aged_out` rows are deliberately NOT revived. They were on disk once and were
deleted; bringing them back would present months of old episodes to Jellyfin
as new, and the whole point of the tombstone is that this cannot happen.
"""
cutoff = util.today() - timedelta(
days=effective_retention_days(settings, channel)
)
with conn:
cursor = conn.execute(
"UPDATE video SET state = ? "
"WHERE channel_pk = ? AND state = ? AND upload_date >= ?",
(videos.LISTED, channel["id"], videos.SKIPPED_OLD, cutoff.isoformat()),
)
if cursor.rowcount:
log.info("%s: re-queued %d video(s) now inside the %s window",
channel["title"], cursor.rowcount, cutoff.isoformat())
return cursor.rowcount
def poll_all(
conn: sqlite3.Connection, settings: Settings, channel_pk: int | None = None
) -> dict:
"""Backfill anything new, then poll everything. Returns aggregate counts."""
if channel_pk is not None:
rows = [row for row in [channels.get(conn, channel_pk)] if row is not None]
else:
rows = channels.all_channels(conn)
totals = {"channels": 0, "queued": 0, "old": 0, "known": 0, "repaired": 0,
"titled": 0, "shorts": 0, "live": 0, "failed": 0}
for channel in rows:
totals["channels"] += 1
if not channel["backfilled"]:
stats = backfill_channel(conn, settings, channel)
# Poll straight after a backfill: it is what supplies the titles the
# contentDetails pages do not carry.
if "error" not in stats:
poll_stats = poll_channel(conn, settings, channel)
for key in ("titled", "known"):
stats[key] = stats.get(key, 0) + poll_stats.get(key, 0)
else:
stats = poll_channel(conn, settings, channel)
if "error" in stats:
totals["failed"] += 1
for key in ("queued", "old", "known", "repaired", "titled", "shorts", "live"):
totals[key] += stats.get(key, 0)
log.info("%s: %s", channel["title"], stats)
return totals
ytstream/doctor.py
+211
View File
@@ -0,0 +1,211 @@
"""Preflight checks.
`doctor` is the first acceptance criterion and the thing to run when something
breaks. Every check returns a row rather than raising, so one failure doesn't
hide the others.
"""
from __future__ import annotations
import grp
import os
import stat
import subprocess
from dataclasses import dataclass
from pathlib import Path
from . import config, jellyfin, ytdlp
from .settings import Settings
MEDIA_GROUP = "mediaserver"
@dataclass
class Check:
name: str
ok: bool
detail: str
fatal: bool = True
def _deno() -> Check:
binary = config.VENV_BIN / "deno"
if not binary.exists():
return Check(
"js runtime",
False,
f"deno not found at {binary} — yt-dlp cannot solve n challenges (specs.md §3)",
)
try:
result = subprocess.run(
[str(binary), "--version"], capture_output=True, text=True, timeout=30
)
except OSError as exc:
return Check("js runtime", False, f"deno unusable: {exc}")
if result.returncode != 0:
return Check("js runtime", False, "deno --version failed")
return Check("js runtime", True, result.stdout.splitlines()[0])
def _ytdlp() -> Check:
try:
return Check("yt-dlp", True, ytdlp.version())
except (OSError, ytdlp.YtdlpError) as exc:
return Check("yt-dlp", False, str(exc))
def _ejs() -> Check:
try:
from importlib.metadata import version as pkg_version
return Check("yt-dlp-ejs", True, pkg_version("yt-dlp-ejs"))
except Exception:
return Check(
"yt-dlp-ejs",
False,
"not installed — reinstall with the yt-dlp[default] extra (specs.md §3)",
)
def _pot(settings: Settings) -> Check:
url = settings.get_str("pot_provider_url")
try:
info = ytdlp.pot_provider_ping(url)
except Exception as exc:
return Check("pot provider", False, f"{url}/ping unreachable: {exc}")
server_version = str(info.get("version", "?"))
installed = ytdlp.plugin_version()
if installed and installed != server_version:
return Check(
"pot provider",
False,
f"version skew: server {server_version} vs plugin {installed}",
)
return Check("pot provider", True, f"up, version {server_version}")
def _database() -> Check:
try:
from . import db
conn = db.connect()
version = conn.execute("PRAGMA user_version").fetchone()[0]
mode = conn.execute("PRAGMA journal_mode").fetchone()[0]
conn.close()
except Exception as exc:
return Check("database", False, str(exc))
if str(mode).lower() != "wal":
return Check("database", False, f"journal_mode is {mode}, expected wal")
return Check("database", True, f"{config.DB_PATH} (schema v{version}, {mode})")
def _media_root() -> Check:
root = config.MEDIA_ROOT
if not root.is_dir():
return Check("media root", False, f"{root} does not exist")
if not os.access(root, os.W_OK | os.X_OK):
return Check("media root", False, f"{root} is not writable")
info = root.stat()
problems = []
try:
group = grp.getgrgid(info.st_gid).gr_name
except KeyError:
group = str(info.st_gid)
if group != MEDIA_GROUP:
problems.append(f"group is {group}, expected {MEDIA_GROUP}")
if not info.st_mode & stat.S_ISGID:
problems.append("setgid bit not set (new dirs won't inherit the group)")
if problems:
return Check("media root", False, f"{root}: " + "; ".join(problems))
return Check("media root", True, f"{root} ({group}, setgid)")
def _proxy() -> Check:
"""The proxy is what turns a .strm into video. Without it nothing plays.
Replaces youtube-automate's work-dir check: ytstream has no work dir, because
the proxy's scratch space is tmpfs outside the media tree entirely.
"""
import json
import urllib.error
import urllib.request
url = "http://127.0.0.1:8099/healthz"
try:
with urllib.request.urlopen(url, timeout=5) as response:
payload = json.load(response)
except (urllib.error.URLError, OSError, ValueError) as exc:
return Check("proxy", False, f"{url} unreachable ({exc}) — nothing will play")
mode = payload.get("mode", "?")
used = payload.get("cache_used_gb", 0)
return Check("proxy", True, f"{url} ok (mode={mode}, cache={used:.2f} GB)")
def _jellyfin(settings: Settings) -> Check:
client = jellyfin.from_settings(settings)
if not client.base_url:
return Check("jellyfin", False, "jellyfin_url not set", fatal=False)
try:
info = client.public_info()
except jellyfin.JellyfinError as exc:
return Check("jellyfin", False, str(exc))
label = f"{info.get('ServerName', '?')} {info.get('Version', '?')}"
if not client.api_key:
return Check(
"jellyfin",
False,
f"{label} reachable but no API key set (run set-jellyfin-key)",
fatal=False,
)
try:
library = client.find_library(config.MEDIA_ROOT)
except jellyfin.JellyfinError as exc:
return Check("jellyfin", False, f"API key rejected: {exc}")
if library is None:
return Check(
"jellyfin",
False,
f"{label}, no library for {config.MEDIA_ROOT} (run setup-jellyfin-library)",
fatal=False,
)
return Check("jellyfin", True, f"{label}, library '{library.get('Name')}'")
def run_checks(settings: Settings) -> list[Check]:
return [
_ytdlp(),
_ejs(),
_deno(),
_pot(settings),
_database(),
_media_root(),
_proxy(),
_jellyfin(settings),
]
def report(checks: list[Check]) -> tuple[str, int]:
"""Render the checks and return (text, exit_code)."""
lines = []
failed_fatal = 0
for check in checks:
if check.ok:
mark = "ok "
elif check.fatal:
mark = "FAIL"
failed_fatal += 1
else:
mark = "warn"
lines.append(f" [{mark}] {check.name:<14} {check.detail}")
if failed_fatal:
lines.append(f"\n{failed_fatal} fatal problem(s).")
else:
lines.append("\nAll fatal checks passed.")
return "\n".join(lines), (1 if failed_fatal else 0)
ytstream/jellyfin.py
+145
View File
@@ -0,0 +1,145 @@
"""Minimal Jellyfin API client.
Only three things are needed: check the server is alive, create the Shows library
with internet metadata providers switched off, and trigger a refresh after we
change the tree.
"""
from __future__ import annotations
import json
import logging
import urllib.error
import urllib.parse
import urllib.request
from pathlib import Path
from . import config
log = logging.getLogger(__name__)
LIBRARY_NAME = "YouTube (stream)"
COLLECTION_TYPE = "tvshows"
# Metadata is supplied entirely by our own NFO sidecars, so every fetcher is
# disabled for all three item types a Shows library resolves.
_ITEM_TYPES = ("Series", "Season", "Episode")
class JellyfinError(RuntimeError):
pass
class Jellyfin:
def __init__(self, base_url: str, api_key: str = "", timeout: float = 30.0):
self.base_url = (base_url or "").rstrip("/")
self.api_key = api_key or ""
self.timeout = timeout
@property
def configured(self) -> bool:
return bool(self.base_url and self.api_key)
def _request(
self,
method: str,
path: str,
params: dict | None = None,
body: dict | None = None,
):
url = self.base_url + path
if params:
url += "?" + urllib.parse.urlencode(params)
data = None
headers = {"User-Agent": config.USER_AGENT, "Accept": "application/json"}
if self.api_key:
headers["X-Emby-Token"] = self.api_key
if body is not None:
data = json.dumps(body).encode("utf-8")
headers["Content-Type"] = "application/json"
request = urllib.request.Request(url, data=data, headers=headers, method=method)
try:
with urllib.request.urlopen(request, timeout=self.timeout) as response:
payload = response.read()
except urllib.error.HTTPError as exc:
raise JellyfinError(f"{method} {path} -> HTTP {exc.code}") from exc
except OSError as exc:
raise JellyfinError(f"{method} {path} -> {exc}") from exc
if not payload:
return None
try:
return json.loads(payload)
except json.JSONDecodeError:
return None
def public_info(self) -> dict:
"""Unauthenticated liveness check."""
return self._request("GET", "/System/Info/Public") or {}
def virtual_folders(self) -> list[dict]:
return self._request("GET", "/Library/VirtualFolders") or []
def find_library(self, path: Path | str) -> dict | None:
target = str(path).rstrip("/")
for folder in self.virtual_folders():
for location in folder.get("Locations") or []:
if str(location).rstrip("/") == target:
return folder
return None
def create_library(self, path: Path | str, name: str = LIBRARY_NAME) -> None:
"""Create the Shows library with all internet providers disabled."""
options = {
"EnableInternetProviders": False,
"SaveLocalMetadata": True,
"EnableRealtimeMonitor": False,
"EnableChapterImageExtraction": False,
"PathInfos": [{"Path": str(path)}],
"TypeOptions": [
{
"Type": item_type,
"MetadataFetchers": [],
"MetadataFetcherOrder": [],
"ImageFetchers": [],
"ImageFetcherOrder": [],
}
for item_type in _ITEM_TYPES
],
}
self._request(
"POST",
"/Library/VirtualFolders",
params={
"name": name,
"collectionType": COLLECTION_TYPE,
"paths": str(path),
"refreshLibrary": "false",
},
body={"LibraryOptions": options},
)
def refresh(self) -> None:
"""Trigger a library scan. Best effort — never fatal to the caller.
Deliberately a plain `/Library/Refresh` with no query parameters. A normal
scan was measured to make **zero** media probes, which is the single fact
this whole design rests on: a scan of a .strm library must not cause the
proxy to fetch every video nobody is watching.
`metadataRefreshMode=FullRefresh` / `replaceAllMetadata=true` *does* probe.
Never add either here, whatever a future caller seems to want — the
correct way to rebuild metadata is to re-materialise the tree from the
database, which is `ytstream materialise --all`.
"""
try:
self._request("POST", "/Library/Refresh")
except JellyfinError as exc:
log.warning("jellyfin refresh failed: %s", exc)
def from_settings(settings) -> Jellyfin:
return Jellyfin(
settings.get_str("jellyfin_url"), settings.get_str("jellyfin_api_key")
)
ytstream/naming.py
+104
View File
@@ -0,0 +1,104 @@
"""Filename sanitisation and season/episode numbering.
Season is the upload year; episode is ``MMDD * 10 + ordinal_within_day``. That
scheme sorts correctly across a whole year (1 Jan is 1010, 31 Dec is 12310) and
leaves room for ten uploads per channel per day.
"""
from __future__ import annotations
import logging
import re
from datetime import date
log = logging.getLogger(__name__)
# Characters that are illegal or awkward in filenames on the platforms Jellyfin
# clients run on. Replaced with a space rather than deleted so that "A/B" reads
# as "A B" instead of collapsing into "AB".
FORBIDDEN = '/\\:*?"<>|'
MAX_TITLE_LEN = 120
MAX_ORDINAL = 9
_CONTROL = re.compile(r"[\x00-\x1f\x7f]")
_WHITESPACE = re.compile(r"\s+")
def sanitize_component(text: str, max_len: int = MAX_TITLE_LEN) -> str:
"""Make one path component safe, collapsing whitespace and truncating."""
text = _CONTROL.sub(" ", text or "")
text = "".join(" " if char in FORBIDDEN else char for char in text)
text = _WHITESPACE.sub(" ", text).strip()
text = truncate_on_word_boundary(text, max_len)
# A component may not begin or end with a dot or space: leading dots hide the
# file from Jellyfin, trailing ones confuse some clients.
text = text.strip(" .")
return text
def truncate_on_word_boundary(text: str, max_len: int) -> str:
if len(text) <= max_len:
return text
cut = text[:max_len]
space = cut.rfind(" ")
# Only honour the word boundary if it doesn't throw away most of the name.
if space > max_len * 0.6:
cut = cut[:space]
return cut.rstrip()
def channel_dir_name(title: str, channel_id: str) -> str:
"""Directory name for a channel. Stored once and never recomputed."""
name = sanitize_component(title)
return name or channel_id
def parse_upload_date(value: str | date) -> date:
"""Accept yt-dlp's YYYYMMDD, ISO YYYY-MM-DD, or a date."""
if isinstance(value, date):
return value
text = str(value).strip()
if len(text) == 8 and text.isdigit():
return date(int(text[:4]), int(text[4:6]), int(text[6:8]))
return date.fromisoformat(text[:10])
def season_for(upload_date: date) -> int:
return upload_date.year
def episode_base(upload_date: date) -> int:
"""First episode number available on this date."""
return (upload_date.month * 100 + upload_date.day) * 10
def episode_number(upload_date: date, ordinal: int) -> int:
"""Episode number for the nth upload on a given date (n starting at 0)."""
if ordinal > MAX_ORDINAL:
log.warning(
"more than %d uploads on %s; clamping ordinal %d",
MAX_ORDINAL + 1,
upload_date.isoformat(),
ordinal,
)
return episode_base(upload_date) + min(max(ordinal, 0), MAX_ORDINAL)
def episode_range(upload_date: date) -> tuple[int, int]:
"""Inclusive (low, high) episode numbers belonging to this date."""
base = episode_base(upload_date)
return base, base + MAX_ORDINAL
def season_dir_name(season: int) -> str:
return f"Season {season}"
def basename(channel_dir: str, season: int, episode: int, title: str, video_id: str) -> str:
"""Filename stem shared by the media file and every sidecar.
The [video_id] suffix guarantees uniqueness regardless of title collisions.
"""
safe_title = sanitize_component(title) or video_id
return f"{channel_dir} - S{season}E{episode} - {safe_title} [{video_id}]"
ytstream/nfo.py
+86
View File
@@ -0,0 +1,86 @@
"""Kodi-style NFO sidecars.
Video descriptions are hostile input — they contain ampersands, angle brackets,
emoji, ASCII art and control characters — so these are always built with
ElementTree's serialiser and never by string formatting.
"""
from __future__ import annotations
import re
import xml.etree.ElementTree as ET
from pathlib import Path
# XML 1.0 forbids most control characters outright; ElementTree will happily
# serialise them and produce a document no parser will read back. Written as a
# raw string so `re` interprets the escapes, not Python.
_ILLEGAL_XML = re.compile(
r"[^\x09\x0a\x0d\x20-퟿-�\U00010000-\U0010ffff]"
)
def clean_text(value: str | None) -> str:
return _ILLEGAL_XML.sub("", value or "")
def _child(parent: ET.Element, tag: str, text: str | None) -> ET.Element:
element = ET.SubElement(parent, tag)
element.text = clean_text(text)
return element
def _serialise(root: ET.Element) -> bytes:
ET.indent(root, space=" ")
return ET.tostring(root, encoding="utf-8", xml_declaration=True)
def tvshow_nfo(title: str, plot: str | None, channel_id: str) -> bytes:
root = ET.Element("tvshow")
_child(root, "title", title)
_child(root, "plot", plot)
_child(root, "studio", "YouTube")
unique = _child(root, "uniqueid", channel_id)
unique.set("type", "youtube")
unique.set("default", "true")
return _serialise(root)
def episode_nfo(
*,
title: str,
show_title: str,
season: int,
episode: int,
plot: str | None,
aired: str,
duration_seconds: int | None,
video_id: str,
) -> bytes:
root = ET.Element("episodedetails")
_child(root, "title", title)
_child(root, "showtitle", show_title)
_child(root, "season", str(season))
_child(root, "episode", str(episode))
_child(root, "plot", plot)
_child(root, "aired", aired)
if duration_seconds:
# Kodi/Jellyfin expect <runtime> in whole minutes.
_child(root, "runtime", str(max(1, round(duration_seconds / 60))))
# Jellyfin prefers this when present, and it is what stops a .strm episode
# showing a runtime of zero before it has ever been played. Deliberately
# NOT wrapped in <fileinfo><streamdetails>: pre-seeding those was measured
# to change nothing about whether Jellyfin probes the media (FINDINGS §6).
_child(root, "durationinseconds", str(int(duration_seconds)))
_child(root, "studio", "YouTube")
unique = _child(root, "uniqueid", video_id)
unique.set("type", "youtube")
unique.set("default", "true")
return _serialise(root)
def write(path: Path, payload: bytes) -> None:
"""Write atomically so a crash never leaves Jellyfin a half-written NFO."""
path.parent.mkdir(parents=True, exist_ok=True)
temporary = path.with_name(path.name + ".tmp")
temporary.write_bytes(payload)
temporary.replace(path)
ytstream/reap.py
+89
View File
@@ -0,0 +1,89 @@
"""Retention: delete videos that have aged out of the window.
Two rules, and the second is what makes the first usable:
1. A video older than `retention_days` is deleted from disk and tombstoned.
2. **Except** the `min_keep_videos` most recent videos of each channel, which are
always kept however old they are.
Rule 2 exists because of a measurement, not a preference. Of 117 subscribed
channels, 52 uploaded nothing long-form in a 30-day window and the median channel
uploaded once. Under rule 1 alone, most channels would be empty Jellyfin series,
and a channel uploading every six weeks would flicker in and out of existence as
its single video crossed the line. Keeping the last few videos regardless of age
costs ~260 episodes across the whole library and removes the flicker entirely,
because a video can only leave once `min_keep_videos` newer ones exist.
Deletion leaves the row behind as a tombstone in state `aged_out`. That is
load-bearing twice over: without it the next poll finds the video in the feed and
re-materialises it forever, and episode ordinals for a given day would shift as
videos disappear.
"""
from __future__ import annotations
import logging
import sqlite3
from datetime import timedelta
from . import jellyfin, strm, util, videos
from .settings import Settings
log = logging.getLogger(__name__)
def effective_retention(settings: Settings, channel_override: int | None) -> int:
return int(channel_override) if channel_override else settings.get_int("retention_days")
def candidates(conn: sqlite3.Connection, settings: Settings) -> list[sqlite3.Row]:
"""Materialised videos past their window, excluding the protected newest N.
Evaluated per channel because both bounds are per channel: the window may be
overridden on the channel row, and `min_keep_videos` counts within it.
"""
keep = max(0, settings.get_int("min_keep_videos"))
today = util.today()
due: list[sqlite3.Row] = []
channels = conn.execute(
"SELECT id, title, retention_days FROM channel"
).fetchall()
for channel in channels:
days = effective_retention(settings, channel["retention_days"])
cutoff = (today - timedelta(days=days)).isoformat()
# Newest first, so everything from index `keep` onwards is unprotected.
on_disk = videos.materialised_for_channel(conn, channel["id"])
for video in on_disk[keep:]:
if video["upload_date"] and video["upload_date"] < cutoff:
due.append(video)
return due
def delete_video(conn: sqlite3.Connection, video: sqlite3.Row) -> bool:
"""Delete one video's files and leave a tombstone row."""
if not video["rel_path"]:
# Already gone from disk; still needs the tombstone.
videos.mark_aged_out(conn, video["video_id"])
return False
removed = strm.remove(video)
videos.mark_aged_out(conn, video["video_id"])
log.info("aged out %s (uploaded %s): %d file(s)",
video["video_id"], video["upload_date"], removed)
return True
def run(conn: sqlite3.Connection, settings: Settings) -> dict:
"""One retention pass."""
deleted = 0
for video in candidates(conn, settings):
if delete_video(conn, video):
deleted += 1
if deleted:
# Without this, Jellyfin shows ghost episodes until its own scheduled scan.
jellyfin.from_settings(settings).refresh()
return {"aged_out": deleted}
ytstream/runner.py
+149
View File
@@ -0,0 +1,149 @@
"""`run` orchestration: sync, then poll, then materialise, then reap — under a lock.
Order matters. The subscription sync runs *first* so that a channel added on
YouTube at 14:00 has its catalogue built in the same pass rather than an hour
later. Retention runs *last* so a video discovered and materialised in this run is
judged against the window once, not twice.
"""
from __future__ import annotations
import contextlib
import fcntl
import logging
import sqlite3
from pathlib import Path
from . import config, discovery, jellyfin, reap, strm, subsync, util, videos
from .settings import Settings
log = logging.getLogger(__name__)
class AlreadyRunning(Exception):
pass
@contextlib.contextmanager
def exclusive_lock(path: Path | None = None):
"""Non-blocking flock. Raises AlreadyRunning if another run holds it.
The cron schedule is hourly and a first-run import of 119 channels can outlast
that, so overlapping runs are expected and must be a silent no-op rather than
two workers fighting over the same queue.
"""
path = path or config.LOCK_PATH
path.parent.mkdir(parents=True, exist_ok=True)
handle = path.open("w")
try:
try:
fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError as exc:
raise AlreadyRunning(f"another run holds {path}") from exc
yield
finally:
with contextlib.suppress(OSError):
fcntl.flock(handle, fcntl.LOCK_UN)
handle.close()
def materialise_all(
conn: sqlite3.Connection, settings: Settings, limit: int | None = None
) -> dict:
"""Write a `.strm` + `.nfo` + thumbnail for everything in `listed`.
There is no retry ladder and no failure state. Writing a 50-byte text file
either works or the filesystem is broken, and in the latter case the run
should stop rather than mark 400 videos as failed.
"""
stats = {"materialised": 0, "shows": 0, "errors": 0}
seen_channels: set[int] = set()
for video in videos.claim_listed(conn, limit):
channel = conn.execute(
"SELECT * FROM channel WHERE id = ?", (video["channel_pk"],)
).fetchone()
if channel is None: # channel deleted mid-run
continue
if channel["id"] not in seen_channels:
# Lazily, so a channel with nothing inside the window never creates an
# empty series in Jellyfin (plan.md §5).
strm.write_show(channel)
seen_channels.add(channel["id"])
stats["shows"] += 1
try:
strm.materialise(conn, settings, channel, video)
stats["materialised"] += 1
except OSError as exc:
log.error("could not materialise %s: %s", video["video_id"], exc)
stats["errors"] += 1
return stats
def run(
conn: sqlite3.Connection, settings: Settings, channel_pk: int | None = None
) -> dict:
"""One full cycle. Assumes the caller holds the lock."""
result: dict = {}
# Sync first: a channel added on YouTube should get its catalogue this run.
# Skipped for a single-channel run, which is a targeted operation.
if channel_pk is None:
result["sync"] = subsync.sync_all(conn, settings)
result["poll"] = discovery.poll_all(conn, settings, channel_pk)
result["materialise"] = materialise_all(conn, settings)
result["reap"] = reap.run(conn, settings)
if result["materialise"]["materialised"] or result["reap"]["aged_out"]:
jellyfin.from_settings(settings).refresh()
settings.set("last_run_at", util.utcnow_iso())
return result
def summarise(result: dict) -> str:
sync = result.get("sync", {})
poll = result.get("poll", {})
made = result.get("materialise", {})
reaped = result.get("reap", {})
parts = [
f"channels={poll.get('channels', 0)}",
f"discovered={poll.get('queued', 0)}",
f"materialised={made.get('materialised', 0)}",
f"aged_out={reaped.get('aged_out', 0)}",
f"shorts={poll.get('shorts', 0)}",
f"live={poll.get('live', 0)}",
f"poll_failures={poll.get('failed', 0)}",
]
if sync:
parts[1:1] = [
f"subs_added={sync.get('added', 0)}",
f"subs_queued={sync.get('queued', 0)}",
f"subs_removed={sync.get('removed', 0)}",
]
if sync.get("refused"):
parts.append(f"SYNC_REFUSED={sync['refused']}")
if made.get("errors"):
parts.append(f"errors={made['errors']}")
return " ".join(parts)
def exit_code(result: dict) -> int:
"""Non-zero when something needs a human.
The cron entry runs under `runitor`, so this is what turns the healthchecks
check red. A silently-broken subscription mirror is the worst outcome
available — nothing looks wrong until someone asks why a channel never
appeared — so a refused sync must be loud.
"""
sync = result.get("sync", {})
if sync.get("refused"):
return 1
if result.get("materialise", {}).get("errors"):
return 1
if result.get("poll", {}).get("failed"):
# Two of 119 measured channels fail permanently (terminated or private),
# so a poll failure is a warning, not a red check.
log.warning("%d channel(s) failed to poll", result["poll"]["failed"])
return 0
ytstream/settings.py
+160
View File
@@ -0,0 +1,160 @@
"""Typed settings accessors backed by the `setting` key/value table.
A missing key must never crash anything, so every read falls back to the default
and every malformed stored value falls back to the default too.
Reads go to the database every time rather than being cached in the process. That
is deliberate: rotating the YouTube API key has to be nothing more than saving a
new value in the admin UI, with no restart of the hourly job or the admin server
(plan.md §11).
"""
from __future__ import annotations
import sqlite3
from urllib.parse import urlparse
DEFAULTS: dict[str, str] = {
# Retention. One number governs both how far back a new channel is backfilled
# and when a video is deleted, so the library cannot grow (plan.md §5).
"retention_days": "30",
# ...except that 52 of 117 measured channels upload nothing in 30 days, which
# would leave them as empty Jellyfin series flickering in and out. So the real
# rule is max(retention_days, min_keep_videos most recent).
"min_keep_videos": "5",
# Runaway guard only: a channel that turns out to upload 50 times a day.
"backfill_max_videos": "300",
"max_height": "1080",
"min_duration_seconds": "120",
"jellyfin_url": "http://127.0.0.1:8096",
"jellyfin_api_key": "",
"pot_provider_url": "http://127.0.0.1:4416",
"youtube_api_key": "",
"proxy_base_url": "http://127.0.0.1:8099",
# Subscription mirroring (plan.md §4.4). max_new is derived from the measured
# 119 subscriptions as max(10, ceil(119 * 0.2)).
"subsync_max_new": "25",
"subsync_missing_threshold": "3",
}
# Editable through the settings form. Everything else in the table is internal.
EDITABLE = tuple(DEFAULTS)
# Never rendered, never settable through the web form.
SECRET_KEYS = ("admin_password_hash", "session_secret")
# Shown as a masked value rather than plaintext.
MASKED_KEYS = ("jellyfin_api_key", "youtube_api_key")
_INT_KEYS = (
"retention_days",
"min_keep_videos",
"backfill_max_videos",
"max_height",
"min_duration_seconds",
"subsync_max_new",
"subsync_missing_threshold",
)
_BOOL_KEYS = ()
_URL_KEYS = ("jellyfin_url", "pot_provider_url", "proxy_base_url")
_TRUE = {"1", "true", "yes", "on"}
_FALSE = {"0", "false", "no", "off", ""}
class Settings:
def __init__(self, conn: sqlite3.Connection):
self.conn = conn
def raw(self, key: str) -> str:
row = self.conn.execute(
"SELECT value FROM setting WHERE key = ?", (key,)
).fetchone()
if row is None:
return DEFAULTS.get(key, "")
return row["value"]
def get_str(self, key: str) -> str:
return self.raw(key)
def get_int(self, key: str) -> int:
try:
return int(self.raw(key))
except (TypeError, ValueError):
return int(DEFAULTS.get(key, "0") or 0)
def get_bool(self, key: str) -> bool:
value = self.raw(key).strip().lower()
if value in _TRUE:
return True
if value in _FALSE:
return False
return DEFAULTS.get(key, "false").lower() in _TRUE
def set(self, key: str, value: str) -> None:
with self.conn:
self.conn.execute(
"INSERT INTO setting (key, value) VALUES (?, ?) "
"ON CONFLICT(key) DO UPDATE SET value = excluded.value",
(key, str(value)),
)
def all_editable(self) -> dict[str, str]:
return {key: self.raw(key) for key in EDITABLE}
def validate(key: str, value: str) -> tuple[bool, str]:
"""Validate one submitted setting.
Returns (ok, message). On failure the caller re-renders the form with the
message inline rather than raising.
"""
value = value.strip()
if key in _INT_KEYS:
try:
number = int(value)
except ValueError:
return False, "must be a whole number"
if number < 0:
return False, "must be zero or greater"
if key == "retention_days" and number < 1:
return False, "must be at least 1 day"
if key == "max_height" and number < 144:
return False, "must be at least 144"
if key == "subsync_missing_threshold" and number < 1:
# Zero would unsubscribe on the first absent response, which is
# exactly the failure mode the threshold exists to prevent.
return False, "must be at least 1 sync"
return True, ""
if key in _BOOL_KEYS:
if value.lower() not in _TRUE | _FALSE:
return False, "must be true or false"
return True, ""
if key in _URL_KEYS:
parsed = urlparse(value)
if parsed.scheme not in ("http", "https") or not parsed.netloc:
return False, "must be a http:// or https:// URL"
return True, ""
if key in ("jellyfin_api_key", "youtube_api_key"):
return True, ""
return key in DEFAULTS, "unknown setting"
def validate_all(submitted: dict[str, str]) -> dict[str, str]:
"""Return {key: error} for everything that failed validation."""
errors: dict[str, str] = {}
for key, value in submitted.items():
if key not in EDITABLE:
continue
ok, message = validate(key, value)
if not ok:
errors[key] = message
return errors
ytstream/strm.py
+207
View File
@@ -0,0 +1,207 @@
"""Materialising a video: write a `.strm`, an `.nfo` and a thumbnail.
This is what replaced youtube-automate's 330-line `download.py`. There is no
subprocess, no format selection, no progress parsing and no retry ladder, because
the whole job is writing a URL into a text file. Everything expensive was moved to
the moment somebody presses play, which is the point of the design.
Thumbnails come from `i.ytimg.com`, which needs no API key and no quota.
"""
from __future__ import annotations
import logging
import shutil
import sqlite3
import urllib.error
import urllib.request
from pathlib import Path
from . import config, naming, nfo, util, videos
from .settings import Settings
log = logging.getLogger(__name__)
THUMB_SUFFIX = "-thumb.jpg"
# maxres does not exist for every video; hq always does.
_THUMB_URLS = (
"https://i.ytimg.com/vi/{vid}/maxresdefault.jpg",
"https://i.ytimg.com/vi/{vid}/hqdefault.jpg",
)
def watch_url(settings: Settings, video_id: str) -> str:
base = settings.get_str("proxy_base_url").rstrip("/")
return f"{base}/watch/{video_id}"
def channel_dir(channel: sqlite3.Row) -> Path:
return config.MEDIA_ROOT / channel["dir_name"]
def episode_paths(channel: sqlite3.Row, season: int, episode: int,
title: str, video_id: str) -> tuple[Path, str]:
"""(.strm path, path relative to the media root)."""
stem = naming.basename(channel["dir_name"], season, episode, title, video_id)
relative = Path(channel["dir_name"]) / naming.season_dir_name(season) / (stem + ".strm")
return config.MEDIA_ROOT / relative, str(relative)
def write_show(channel: sqlite3.Row) -> None:
"""tvshow.nfo for a channel.
Takes the row's show fields explicitly rather than trusting a joined row: the
channel/video join has `title` on both sides, and reading the wrong one
renamed every series after whichever episode happened to be first. That
shipped once already.
"""
directory = channel_dir(channel)
directory.mkdir(parents=True, exist_ok=True)
nfo.write(
directory / "tvshow.nfo",
nfo.tvshow_nfo(
title=channel["title"],
plot=channel["description"],
channel_id=channel["channel_id"],
),
)
def fetch_thumbnail(video_id: str, destination: Path, *, timeout: float = 20.0) -> bool:
"""Best effort. A missing thumbnail is cosmetic, never a failure."""
if destination.exists():
return True
for template in _THUMB_URLS:
url = template.format(vid=video_id)
request = urllib.request.Request(url, headers={"User-Agent": config.USER_AGENT})
try:
with urllib.request.urlopen(request, timeout=timeout) as response:
if response.status != 200:
continue
payload = response.read()
except (urllib.error.HTTPError, urllib.error.URLError, OSError):
continue
if len(payload) < 1024:
# YouTube serves a 120-byte grey placeholder rather than a 404.
continue
destination.parent.mkdir(parents=True, exist_ok=True)
temporary = destination.with_name(destination.name + ".tmp")
temporary.write_bytes(payload)
temporary.replace(destination)
return True
log.debug("no thumbnail available for %s", video_id)
return False
def materialise(
conn: sqlite3.Connection,
settings: Settings,
channel: sqlite3.Row,
video: sqlite3.Row,
) -> dict:
"""Write one episode's `.strm`, `.nfo` and thumbnail, and record it.
Idempotent: re-running over an already-materialised video rewrites the same
bytes to the same paths.
"""
upload_date = naming.parse_upload_date(video["upload_date"])
season, episode = videos.next_episode(
conn, channel["id"], upload_date, video["video_id"]
)
# `title` is for the filename; `stored_title` is what goes back to the
# database. They differ when we have no title yet: writing the video id back
# would make the row look titled, permanently disabling the repair path in
# discovery._record that fills titles in from a later feed poll. That bug
# shipped once and renamed five of twenty episodes after their video ids.
stored_title = video["title"] or ""
title = stored_title or video["video_id"]
strm_path, relative = episode_paths(
channel, season, episode, title, video["video_id"]
)
strm_path.parent.mkdir(parents=True, exist_ok=True)
# No trailing newline: some Jellyfin versions have historically been fussy
# about trailing whitespace in .strm files, and there is nothing to gain.
temporary = strm_path.with_name(strm_path.name + ".tmp")
temporary.write_text(watch_url(settings, video["video_id"]), encoding="utf-8")
temporary.replace(strm_path)
nfo.write(
strm_path.with_suffix(".nfo"),
nfo.episode_nfo(
title=title,
show_title=channel["title"],
season=season,
episode=episode,
plot=None,
aired=upload_date.isoformat(),
duration_seconds=video["duration"],
video_id=video["video_id"],
),
)
stem = strm_path.name[: -len(".strm")]
fetch_thumbnail(video["video_id"], strm_path.with_name(stem + THUMB_SUFFIX))
videos.mark_materialised(
conn,
video["video_id"],
rel_path=relative,
season=season,
episode=episode,
upload_date=upload_date.isoformat(),
duration=video["duration"],
title=stored_title,
)
return {"video_id": video["video_id"], "rel_path": relative,
"season": season, "episode": episode}
def remove(video: sqlite3.Row) -> int:
"""Delete an episode's files. Returns how many were removed.
Only touches suffixes we know we wrote. A stray file someone else put in the
season directory is not ours to delete.
"""
if not video["rel_path"]:
return 0
strm_path = config.MEDIA_ROOT / video["rel_path"]
stem = strm_path.name[: -len(".strm")]
removed = 0
for path in [strm_path] + [
strm_path.with_name(stem + suffix) for suffix in config.SIDECAR_SUFFIXES
]:
try:
path.unlink()
removed += 1
except FileNotFoundError:
continue
except OSError as exc:
log.warning("could not remove %s: %s", path, exc)
# Prune empty season directories, but stop at the channel directory — it
# holds tvshow.nfo and the artwork, and an active subscription must not
# vanish from Jellyfin just because it published nothing this year. Passing
# MEDIA_ROOT as the boundary instead would delete the channel directory on
# any channel whose tvshow.nfo happened to be missing.
channel_dir = config.MEDIA_ROOT / Path(video["rel_path"]).parts[0]
util.prune_empty_dirs(strm_path.parent, channel_dir)
return removed
def remove_channel_tree(channel: sqlite3.Row) -> bool:
"""Delete a whole channel directory, on unsubscribe.
Guarded against deleting the media root itself, which a channel whose
dir_name somehow ended up empty would otherwise do.
"""
directory = channel_dir(channel)
if directory == config.MEDIA_ROOT or not str(channel["dir_name"]).strip():
log.error("refusing to remove %s: unsafe channel directory", directory)
return False
if not directory.exists():
return False
shutil.rmtree(directory)
log.info("removed %s", directory)
return True
ytstream/subsync.py
+305
View File
@@ -0,0 +1,305 @@
"""Mirror one YouTube account's public subscriptions.
The mirrored account is the source of truth: what it follows is what exists in
Jellyfin. That makes the sync authoritative in both directions, and the removal
half is destructive, so most of this module is about refusing to act on bad data.
The rules, and why each exists (plan.md §4.4):
* **A 403, a network error, or a zero-item response is never a removal.** All
three look identical to "he unsubscribed from everything". A genuinely empty
list is indistinguishable from a broken one in *consequence*, so we take the
harmless reading and change nothing.
* **A channel must be absent from `subsync_missing_threshold` consecutive healthy
responses** before it is unsubscribed. One bad page does not delete a library.
* **The first sync imports nothing automatically.** With 119 subscriptions the cap
would trip on day one whatever it is set to, which trains everyone to ignore it.
So the first sync queues the whole list for approval instead.
* **After that, more than `subsync_max_new` new channels in one run is refused**
and queued, on the assumption that an order-of-magnitude jump is a bug.
Everything here treats `source = 'manual'` channels as untouchable.
"""
from __future__ import annotations
import logging
import sqlite3
from . import api, channels, strm, util, videos
from .settings import Settings
log = logging.getLogger(__name__)
MANUAL = "manual"
YOUTUBE = "youtube"
class SyncRefused(RuntimeError):
"""The response was not trustworthy enough to act on.
Raised rather than returned so a caller cannot accidentally treat a refusal
as an empty subscription list.
"""
# --------------------------------------------------------------------- sources
def get_source(conn: sqlite3.Connection, key: str) -> sqlite3.Row | None:
return conn.execute("SELECT * FROM source WHERE key = ?", (key,)).fetchone()
def all_sources(conn: sqlite3.Connection, *, enabled_only: bool = False) -> list[sqlite3.Row]:
sql = "SELECT * FROM source"
if enabled_only:
sql += " WHERE enabled = 1"
return conn.execute(sql + " ORDER BY key").fetchall()
def add_source(conn: sqlite3.Connection, *, channel_id: str, label: str) -> str:
key = f"youtube:{channel_id}"
with conn:
conn.execute(
"INSERT INTO source (key, label, channel_id) VALUES (?, ?, ?) "
"ON CONFLICT(key) DO UPDATE SET label = excluded.label",
(key, label, channel_id),
)
return key
def _record_success(conn: sqlite3.Connection, key: str) -> None:
with conn:
conn.execute(
"UPDATE source SET last_sync_at = ?, last_sync_ok = 1, "
"consecutive_failures = 0, last_error = NULL WHERE key = ?",
(util.utcnow_iso(), key),
)
def _record_failure(conn: sqlite3.Connection, key: str, error: str) -> None:
with conn:
conn.execute(
"UPDATE source SET last_sync_at = ?, last_sync_ok = 0, "
"consecutive_failures = consecutive_failures + 1, last_error = ? "
"WHERE key = ?",
(util.utcnow_iso(), error[:500], key),
)
# ------------------------------------------------------------ approval queue
def queue_approval(conn: sqlite3.Connection, source: str, entries: list[dict]) -> int:
now = util.utcnow_iso()
with conn:
for entry in entries:
conn.execute(
"INSERT INTO pending_approval (source, channel_id, title, seen_at) "
"VALUES (?, ?, ?, ?) ON CONFLICT(source, channel_id) DO NOTHING",
(source, entry["channel_id"], entry["title"], now),
)
return len(entries)
def pending(conn: sqlite3.Connection, source: str | None = None) -> list[sqlite3.Row]:
sql = "SELECT * FROM pending_approval WHERE resolved_at IS NULL"
params: list = []
if source:
sql += " AND source = ?"
params.append(source)
return conn.execute(sql + " ORDER BY title COLLATE NOCASE", params).fetchall()
def resolve(conn: sqlite3.Connection, ids: list[int], resolution: str) -> int:
if not ids:
return 0
marks = ",".join("?" * len(ids))
with conn:
cursor = conn.execute(
f"UPDATE pending_approval SET resolved_at = ?, resolution = ? "
f"WHERE id IN ({marks}) AND resolved_at IS NULL",
[util.utcnow_iso(), resolution, *ids],
)
return cursor.rowcount
# ------------------------------------------------------------------- the sync
def _fetch(settings: Settings, source: sqlite3.Row) -> list[dict]:
"""Fetch the subscription list, or raise SyncRefused.
Every failure mode collapses to a refusal here so that no caller can mistake
one for an empty list.
"""
client = api.Api(settings.get_str("youtube_api_key"))
try:
remote = client.subscriptions(source["channel_id"])
except api.SubscriptionsPrivate as exc:
raise SyncRefused(
f"subscriptions are private ({exc.reason}) — nothing changed. "
'Fix: YouTube → Settings → Privacy → uncheck "Keep all my '
'subscriptions private".'
) from exc
except api.NotConfigured as exc:
raise SyncRefused(f"API not usable: {exc}") from exc
except api.ApiError as exc:
raise SyncRefused(f"API error: {exc}") from exc
if not remote:
# Could be true, could be a broken response. The consequences differ by
# everything, so assume the harmless one.
raise SyncRefused(
"subscription list came back empty; treating as suspect and "
"changing nothing"
)
return remote
def sync_source(
conn: sqlite3.Connection, settings: Settings, source: sqlite3.Row
) -> dict:
"""One source, one pass. Never raises for remote problems."""
stats = {"source": source["key"], "seen": 0, "added": 0, "queued": 0,
"removed": 0, "pending_removal": 0, "refused": None}
try:
remote = _fetch(settings, source)
except SyncRefused as exc:
log.error("%s: %s", source["key"], exc)
_record_failure(conn, source["key"], str(exc))
stats["refused"] = str(exc)
return stats
stats["seen"] = len(remote)
remote_ids = {entry["channel_id"] for entry in remote}
known = {
row["channel_id"]: row
for row in conn.execute("SELECT * FROM channel").fetchall()
}
# ---------------------------------------------------------------- additions
new = [entry for entry in remote if entry["channel_id"] not in known]
already_queued = {row["channel_id"] for row in pending(conn, source["key"])}
resolved = {
row["channel_id"]
for row in conn.execute(
"SELECT channel_id FROM pending_approval "
"WHERE source = ? AND resolution = 'rejected'",
(source["key"],),
).fetchall()
}
new = [entry for entry in new
if entry["channel_id"] not in already_queued
and entry["channel_id"] not in resolved]
max_new = settings.get_int("subsync_max_new")
if not source["imported"]:
# First sync: approve-everything-by-hand, never auto-subscribe.
if new:
stats["queued"] = queue_approval(conn, source["key"], new)
log.warning(
"%s: first sync — %d channel(s) queued for approval, none added",
source["key"], stats["queued"],
)
with conn:
conn.execute("UPDATE source SET imported = 1 WHERE key = ?",
(source["key"],))
elif len(new) > max_new:
stats["queued"] = queue_approval(conn, source["key"], new)
log.warning(
"%s: %d new channels exceeds subsync_max_new=%d — none added, all "
"queued for approval", source["key"], len(new), max_new,
)
else:
for entry in new:
try:
channels.subscribe_from_sync(
conn, settings, entry["channel_id"], entry["title"]
)
stats["added"] += 1
except Exception as exc: # noqa: BLE001
# One unresolvable channel must not abort the whole sync.
log.error("%s: could not add %s (%s): %s", source["key"],
entry["title"], entry["channel_id"], exc)
# ----------------------------------------------------------------- removals
threshold = max(1, settings.get_int("subsync_missing_threshold"))
for channel_id, row in known.items():
if row["source"] == MANUAL:
continue
if channel_id in remote_ids:
if row["missing_syncs"]:
with conn:
conn.execute(
"UPDATE channel SET missing_syncs = 0 WHERE id = ?",
(row["id"],),
)
continue
misses = row["missing_syncs"] + 1
with conn:
conn.execute("UPDATE channel SET missing_syncs = ? WHERE id = ?",
(misses, row["id"]))
if misses < threshold:
stats["pending_removal"] += 1
log.info("%s: %s absent for %d/%d syncs", source["key"],
row["title"], misses, threshold)
continue
log.warning("%s: %s absent for %d syncs — unsubscribing and deleting",
source["key"], row["title"], misses)
unsubscribe(conn, row)
stats["removed"] += 1
_record_success(conn, source["key"])
return stats
def unsubscribe(conn: sqlite3.Connection, channel: sqlite3.Row) -> None:
"""Delete a channel: its tree, then its rows.
Tree first. If it fails we still have the rows and can retry; the reverse
would orphan a directory Jellyfin keeps showing with nothing to explain it.
"""
strm.remove_channel_tree(channel)
with conn:
conn.execute("DELETE FROM channel WHERE id = ?", (channel["id"],))
log.info("unsubscribed %s (%s)", channel["title"], channel["channel_id"])
def approve(conn: sqlite3.Connection, settings: Settings, ids: list[int]) -> dict:
"""Subscribe the approved rows from the queue."""
stats = {"added": 0, "failed": 0}
rows = conn.execute(
f"SELECT * FROM pending_approval WHERE id IN ({','.join('?' * len(ids))})",
ids,
).fetchall() if ids else []
for row in rows:
try:
channels.subscribe_from_sync(
conn, settings, row["channel_id"], row["title"]
)
stats["added"] += 1
except Exception as exc: # noqa: BLE001
log.error("could not add %s: %s", row["title"], exc)
stats["failed"] += 1
continue
resolve(conn, [row["id"]], "approved")
return stats
def sync_all(conn: sqlite3.Connection, settings: Settings) -> dict:
totals = {"sources": 0, "added": 0, "queued": 0, "removed": 0,
"pending_removal": 0, "refused": 0}
for source in all_sources(conn, enabled_only=True):
stats = sync_source(conn, settings, source)
totals["sources"] += 1
if stats["refused"]:
totals["refused"] += 1
for key in ("added", "queued", "removed", "pending_removal"):
totals[key] += stats[key]
log.info("%s: %s", source["key"], stats)
return totals
ytstream/util.py
+70
View File
@@ -0,0 +1,70 @@
"""Small shared helpers."""
from __future__ import annotations
import logging
import os
from datetime import date, datetime, timezone
from . import config
def utcnow() -> datetime:
return datetime.now(timezone.utc)
def utcnow_iso() -> str:
return utcnow().replace(microsecond=0).isoformat()
def today() -> date:
return utcnow().date()
def apply_umask() -> None:
"""Ensure files land group-writable so Jellyfin's group can read them."""
os.umask(config.UMASK)
def setup_logging(verbose: bool = False) -> None:
logging.basicConfig(
level=logging.DEBUG if verbose else logging.INFO,
format="%(asctime)s %(levelname)-7s %(name)s: %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
)
# yt-dlp and urllib are noisy at debug level and we drive them deliberately.
logging.getLogger("urllib3").setLevel(logging.WARNING)
def prune_empty_dirs(start, stop) -> int:
"""Remove `start` and its empty parents, never passing `stop`.
With a rolling retention window, whole season directories empty out as a year
rolls over. Leaving them behind gives Jellyfin empty seasons to display, and
they accumulate one per channel per year.
`stop` itself is never removed, and the walk halts at the first non-empty
directory.
"""
from pathlib import Path
start, stop = Path(start), Path(stop)
removed = 0
current = start
while current != stop and stop in current.parents:
try:
current.rmdir() # only succeeds when empty
except OSError:
break
removed += 1
current = current.parent
return removed
def human_bytes(value: int | None) -> str:
size = float(value or 0)
for unit in ("B", "KB", "MB", "GB", "TB"):
if size < 1024 or unit == "TB":
return f"{size:.0f} {unit}" if unit == "B" else f"{size:.1f} {unit}"
size /= 1024
return f"{size:.1f} TB"
ytstream/videos.py
+214
View File
@@ -0,0 +1,214 @@
"""Video row helpers and the state machine.
States (plan.md §7):
listed known and inside the window; a .strm should exist for it
materialised .strm + .nfo + thumbnail written, rel_path set
skipped_short below min_duration_seconds; repaired if later seen in UULF
skipped_live livestream, never retried
skipped_old already outside the window when discovered; revivable by rescan
aged_out was materialised, then deleted by the retention sweep
There is no `pending`/`downloading`/`failed` triad. Writing a 50-byte text file
cannot meaningfully fail, so a video is either listed or it is on disk, and the
retry machinery youtube-automate needed for 500 MB downloads has no analogue here.
`skipped_old` and `aged_out` are both terminal but are NOT interchangeable:
raising retention_days should revive the first and must never revive the second,
or months of previously-deleted episodes would reappear in Jellyfin as new.
"""
from __future__ import annotations
import sqlite3
from datetime import date
from . import naming, util
LISTED = "listed"
MATERIALISED = "materialised"
SKIPPED_SHORT = "skipped_short"
SKIPPED_LIVE = "skipped_live"
SKIPPED_OLD = "skipped_old"
AGED_OUT = "aged_out"
# We have made a final negative decision about these; discovery must not requeue
# them. `aged_out` is the tombstone that stops the retention sweep and the poller
# fighting each other forever.
TERMINAL = (SKIPPED_LIVE, SKIPPED_OLD, AGED_OUT)
# Never revived by `rescan`, however the retention window changes.
NEVER_REVIVE = (AGED_OUT,)
SOURCE_UULF = "uulf_feed"
SOURCE_UC = "uc_feed"
SOURCE_BACKFILL = "backfill"
def get(conn: sqlite3.Connection, video_id: str) -> sqlite3.Row | None:
return conn.execute(
"SELECT * FROM video WHERE video_id = ?", (video_id,)
).fetchone()
def exists(conn: sqlite3.Connection, video_id: str) -> bool:
return get(conn, video_id) is not None
def insert(
conn: sqlite3.Connection,
*,
channel_pk: int,
video_id: str,
title: str,
upload_date: str | None,
state: str,
discovery_source: str,
duration: int | None = None,
published_at: str | None = None,
) -> None:
with conn:
conn.execute(
"INSERT OR IGNORE INTO video "
"(video_id, channel_pk, title, upload_date, published_at, duration, "
" state, discovery_source, discovered_at) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
(
video_id,
channel_pk,
title,
upload_date,
published_at,
duration,
state,
discovery_source,
util.utcnow_iso(),
),
)
def set_state(conn: sqlite3.Connection, video_id: str, state: str) -> None:
with conn:
conn.execute(
"UPDATE video SET state = ? WHERE video_id = ?", (state, video_id)
)
def set_duration(conn: sqlite3.Connection, video_id: str, duration: int | None) -> None:
with conn:
conn.execute(
"UPDATE video SET duration = ? WHERE video_id = ?", (duration, video_id)
)
def mark_materialised(
conn: sqlite3.Connection,
video_id: str,
*,
rel_path: str,
season: int,
episode: int,
upload_date: str,
duration: int | None,
title: str,
) -> None:
with conn:
conn.execute(
"UPDATE video SET state = ?, rel_path = ?, season = ?, episode = ?, "
"upload_date = ?, duration = ?, title = ?, materialised_at = ? "
"WHERE video_id = ?",
(
MATERIALISED,
rel_path,
season,
episode,
upload_date,
duration,
title,
util.utcnow_iso(),
video_id,
),
)
def mark_aged_out(conn: sqlite3.Connection, video_id: str) -> None:
"""Keep the row — it is the tombstone that prevents re-materialising."""
with conn:
conn.execute(
"UPDATE video SET state = ?, rel_path = NULL, deleted_at = ? "
"WHERE video_id = ?",
(AGED_OUT, util.utcnow_iso(), video_id),
)
def next_episode(
conn: sqlite3.Connection, channel_pk: int, upload_date: date, video_id: str
) -> tuple[int, int]:
"""Assign (season, episode) for a video.
The ordinal is computed against what is already in the database for this
channel and date — never against the current batch — so it stays stable
across runs and across crashes mid-batch. Aged-out rows keep their season and
episode precisely so that this stays stable as videos are deleted too.
"""
season = naming.season_for(upload_date)
low, high = naming.episode_range(upload_date)
row = conn.execute(
"SELECT MAX(episode) AS top FROM video "
"WHERE channel_pk = ? AND season = ? AND episode BETWEEN ? AND ? "
"AND video_id != ?",
(channel_pk, season, low, high, video_id),
).fetchone()
top = row["top"] if row and row["top"] is not None else None
if top is None:
return season, low
if top >= high:
# More than ten uploads in a day; naming.episode_number logs the clamp.
return season, high
return season, top + 1
def claim_listed(
conn: sqlite3.Connection, limit: int | None = None
) -> list[sqlite3.Row]:
"""Everything waiting to be materialised, oldest upload first."""
sql = (
"SELECT v.*, c.dir_name, c.title AS channel_title, c.channel_id "
"FROM video v JOIN channel c ON c.id = v.channel_pk "
"WHERE v.state = ? "
"ORDER BY v.upload_date ASC, v.discovered_at ASC"
)
params: list = [LISTED]
if limit:
sql += " LIMIT ?"
params.append(limit)
return conn.execute(sql, params).fetchall()
def materialised_for_channel(
conn: sqlite3.Connection, channel_pk: int
) -> list[sqlite3.Row]:
"""On-disk videos for one channel, newest upload first.
Newest-first because the retention sweep needs to count down from the most
recent to honour min_keep_videos.
"""
return conn.execute(
"SELECT * FROM video WHERE channel_pk = ? AND state = ? "
"ORDER BY upload_date DESC, episode DESC",
(channel_pk, MATERIALISED),
).fetchall()
def queue_depth(conn: sqlite3.Connection) -> int:
return conn.execute(
"SELECT COUNT(*) FROM video WHERE state = ?", (LISTED,)
).fetchone()[0]
def counts_by_state(conn: sqlite3.Connection) -> dict[str, int]:
rows = conn.execute(
"SELECT state, COUNT(*) AS n FROM video GROUP BY state"
).fetchall()
return {row["state"]: row["n"] for row in rows}
ytstream/web/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Admin web UI."""
ytstream/web/auth.py
+191
View File
@@ -0,0 +1,191 @@
"""Password hashing, session cookies and CSRF tokens.
The admin UI is publicly reachable over HTTPS, so this has to be real. The design
goal from specs.md §11 is "log in once per device, effectively never again",
which means a long-lived signed cookie rather than HTTP basic auth.
"""
from __future__ import annotations
import base64
import hashlib
import hmac
import json
import secrets
import time
SCRYPT_N = 2**14
SCRYPT_R = 8
SCRYPT_P = 1
DKLEN = 32
SESSION_MAX_AGE = 365 * 24 * 3600 # one year
COOKIE_NAME = "yta_session"
# Login throttling: after this many consecutive failures from one address, refuse
# for LOCKOUT_SECONDS regardless of whether the password is right.
MAX_FAILURES = 5
LOCKOUT_SECONDS = 60
def _b64(raw: bytes) -> str:
return base64.urlsafe_b64encode(raw).decode("ascii").rstrip("=")
def _unb64(text: str) -> bytes:
padding = "=" * (-len(text) % 4)
return base64.urlsafe_b64decode(text + padding)
# --------------------------------------------------------------------------
# passwords
def hash_password(password: str, *, salt: bytes | None = None) -> str:
salt = salt if salt is not None else secrets.token_bytes(16)
derived = hashlib.scrypt(
password.encode("utf-8"),
salt=salt,
n=SCRYPT_N,
r=SCRYPT_R,
p=SCRYPT_P,
dklen=DKLEN,
)
return f"scrypt${SCRYPT_N}${SCRYPT_R}${SCRYPT_P}${_b64(salt)}${_b64(derived)}"
def verify_password(stored: str, password: str) -> bool:
if not stored:
return False
try:
scheme, n, r, p, salt_b64, hash_b64 = stored.split("$")
if scheme != "scrypt":
return False
derived = hashlib.scrypt(
password.encode("utf-8"),
salt=_unb64(salt_b64),
n=int(n),
r=int(r),
p=int(p),
dklen=len(_unb64(hash_b64)),
)
except (ValueError, TypeError):
return False
return hmac.compare_digest(derived, _unb64(hash_b64))
# --------------------------------------------------------------------------
# sessions
def new_secret() -> str:
return _b64(secrets.token_bytes(32))
def _sign(secret: str, payload: bytes) -> str:
return _b64(hmac.new(secret.encode("utf-8"), payload, hashlib.sha256).digest())
def issue_session(secret: str, *, issued_at: float | None = None) -> str:
payload = json.dumps(
{"iat": int(issued_at if issued_at is not None else time.time())},
separators=(",", ":"),
).encode("utf-8")
return f"{_b64(payload)}.{_sign(secret, payload)}"
def verify_session(secret: str, token: str, *, now: float | None = None) -> bool:
if not token or not secret:
return False
try:
payload_b64, signature = token.split(".", 1)
payload = _unb64(payload_b64)
except (ValueError, TypeError):
return False
if not hmac.compare_digest(_sign(secret, payload), signature):
return False
try:
issued_at = int(json.loads(payload)["iat"])
except (ValueError, KeyError, TypeError):
return False
age = (now if now is not None else time.time()) - issued_at
return 0 <= age <= SESSION_MAX_AGE
def cookie_header(token: str, *, secure: bool = True) -> str:
parts = [
f"{COOKIE_NAME}={token}",
"Path=/",
"HttpOnly",
"SameSite=Lax",
f"Max-Age={SESSION_MAX_AGE}",
]
if secure:
parts.insert(2, "Secure")
return "; ".join(parts)
def clear_cookie_header() -> str:
return f"{COOKIE_NAME}=; Path=/; HttpOnly; SameSite=Lax; Max-Age=0"
def cookie_value(header: str, name: str = COOKIE_NAME) -> str:
"""Pull one cookie out of a request's Cookie header.
Deliberately hand-rolled rather than using http.cookies.SimpleCookie. That
parser silently discards the remainder of the header the moment it meets a
value it considers illegal — a JSON-ish value such as `prefs={"a":1}` is
enough — so any cookie appearing after it becomes invisible. A browser sends
us every cookie on the domain, including ones set by unrelated services, so
one stray value would otherwise make a perfectly valid session vanish and
bounce the user back to the login page with no error shown.
"""
for part in (header or "").split(";"):
candidate, separator, value = part.strip().partition("=")
if separator and candidate.strip() == name:
return value.strip().strip('"')
return ""
# --------------------------------------------------------------------------
# CSRF
def csrf_token(secret: str, session_token: str) -> str:
return _sign(secret, b"csrf:" + session_token.encode("utf-8"))
def verify_csrf(secret: str, session_token: str, submitted: str) -> bool:
if not submitted:
return False
return hmac.compare_digest(csrf_token(secret, session_token), submitted)
# --------------------------------------------------------------------------
# throttling
class LoginThrottle:
"""In-memory consecutive-failure tracker keyed by remote address."""
def __init__(self, max_failures: int = MAX_FAILURES, lockout: int = LOCKOUT_SECONDS):
self.max_failures = max_failures
self.lockout = lockout
self._state: dict[str, tuple[int, float]] = {}
def locked(self, key: str, *, now: float | None = None) -> bool:
failures, last = self._state.get(key, (0, 0.0))
if failures < self.max_failures:
return False
elapsed = (now if now is not None else time.time()) - last
if elapsed >= self.lockout:
self._state.pop(key, None)
return False
return True
def record_failure(self, key: str, *, now: float | None = None) -> None:
failures, _ = self._state.get(key, (0, 0.0))
self._state[key] = (failures + 1, now if now is not None else time.time())
def record_success(self, key: str) -> None:
self._state.pop(key, None)
ytstream/web/server.py
+399
View File
@@ -0,0 +1,399 @@
"""The admin HTTP server.
Stdlib only. Binds to localhost; nginx terminates TLS in front of it at
tube.jihakuz.xyz. Because that hostname is public, this carries real
authentication, CSRF tokens on every state-changing request, and login
throttling.
"""
from __future__ import annotations
import json
import logging
import subprocess
import sys
import threading
import urllib.parse
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from .. import channels, config, db, discovery, jellyfin, util, videos, ytdlp
from ..settings import EDITABLE, MASKED_KEYS, Settings, validate_all
from . import auth, templates
log = logging.getLogger(__name__)
MAX_BODY = 64 * 1024
class AdminServer(ThreadingHTTPServer):
daemon_threads = True
allow_reuse_address = True
def __init__(self, address, handler, *, secure_cookies: bool = True):
super().__init__(address, handler)
self.throttle = auth.LoginThrottle()
self.secure_cookies = secure_cookies
self.db_lock = threading.Lock()
class Handler(BaseHTTPRequestHandler):
server_version = "ytstream"
protocol_version = "HTTP/1.1"
# ---------------------------------------------------------------- utils
def log_message(self, fmt, *args): # noqa: A003 - stdlib signature
log.debug("%s - %s", self.client_address[0], fmt % args)
def _client_key(self) -> str:
"""Real client address, since we always sit behind nginx."""
forwarded = self.headers.get("X-Forwarded-For", "")
if forwarded:
return forwarded.split(",")[0].strip()
return self.client_address[0]
def _send(self, status: int, body: bytes, headers: dict | None = None) -> None:
self.send_response(status)
self.send_header("Content-Type", "text/html; charset=utf-8")
self.send_header("Content-Length", str(len(body)))
self.send_header("X-Content-Type-Options", "nosniff")
self.send_header("Referrer-Policy", "same-origin")
self.send_header("X-Frame-Options", "DENY")
for key, value in (headers or {}).items():
self.send_header(key, value)
self.end_headers()
self.wfile.write(body)
def _redirect(self, location: str, headers: dict | None = None) -> None:
combined = {"Location": location}
combined.update(headers or {})
self._send(303, b"", combined)
def _json(self, status: int, payload: dict) -> None:
body = json.dumps(payload, indent=2).encode("utf-8")
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def _form(self) -> dict[str, str]:
length = int(self.headers.get("Content-Length") or 0)
if length <= 0 or length > MAX_BODY:
return {}
raw = self.rfile.read(length).decode("utf-8", "replace")
return {
key: values[-1]
for key, values in urllib.parse.parse_qs(raw, keep_blank_values=True).items()
}
def _cookie_token(self) -> str:
return auth.cookie_value(self.headers.get("Cookie") or "")
# ------------------------------------------------------------- session
def _open(self):
conn = db.connect()
return conn, Settings(conn)
def _secret(self, settings: Settings) -> str:
secret = settings.raw("session_secret")
if not secret:
secret = auth.new_secret()
settings.set("session_secret", secret)
return secret
def _authenticated(self, settings: Settings) -> str | None:
"""Return the session token if the request is signed in, else None."""
token = self._cookie_token()
if token and auth.verify_session(self._secret(settings), token):
return token
return None
def _check_csrf(self, settings: Settings, token: str, form: dict) -> bool:
return auth.verify_csrf(self._secret(settings), token, form.get("csrf", ""))
# ---------------------------------------------------------------- GET
def do_GET(self) -> None: # noqa: N802 - stdlib signature
path = urllib.parse.urlparse(self.path).path.rstrip("/") or "/"
conn, settings = self._open()
try:
token = self._authenticated(settings)
# /health was specced as unauthenticated back when the UI was going
# to be tailnet-only. On a public hostname it is gratuitous
# fingerprinting surface (yt-dlp version, queue depth), so it needs
# a session like everything else.
if path == "/health":
if not token:
return self._json(401, {"error": "authentication required"})
return self._health(conn, settings)
if path == "/login":
if token:
return self._redirect("/")
return self._send(200, templates.login_page(self._login_hint(settings)))
if not token:
return self._redirect("/login")
if path == "/":
return self._send(200, self._render_index(conn, settings, token))
return self._send(404, templates.page("Not found", "<h1>Not found</h1>"))
finally:
conn.close()
def _login_hint(self, settings: Settings) -> str | None:
if not settings.raw("admin_password_hash"):
return "No password is set yet. Run `ytstream set-password` on susan."
return None
def _health(self, conn, settings: Settings) -> None:
try:
ytdlp_version = ytdlp.version()
except Exception as exc: # noqa: BLE001
ytdlp_version = f"error: {exc}"
try:
ytdlp.pot_provider_ping(settings.get_str("pot_provider_url"))
pot_up = True
except Exception: # noqa: BLE001
pot_up = False
self._json(
200,
{
"yt_dlp_version": ytdlp_version,
"pot_provider_up": pot_up,
"last_run_at": settings.raw("last_run_at") or None,
"queue_depth": videos.queue_depth(conn),
"channels": len(channels.all_channels(conn)),
},
)
# --------------------------------------------------------------- POST
def do_POST(self) -> None: # noqa: N802 - stdlib signature
path = urllib.parse.urlparse(self.path).path.rstrip("/") or "/"
conn, settings = self._open()
try:
form = self._form()
if path == "/login":
return self._login(settings, form)
token = self._authenticated(settings)
if not token:
return self._redirect("/login")
if not self._check_csrf(settings, token, form):
log.warning("CSRF check failed for %s from %s", path, self._client_key())
return self._send(
400,
templates.page(
"Bad request",
"<h1>Bad request</h1><p>Invalid form token. "
'<a href="/">Go back</a> and try again.</p>',
),
)
if path == "/logout":
return self._redirect("/login", {"Set-Cookie": auth.clear_cookie_header()})
if path == "/channels":
return self._add_channel(conn, settings, token, form)
if path == "/settings":
return self._save_settings(conn, settings, token, form)
parts = path.strip("/").split("/")
if len(parts) == 3 and parts[0] == "channels" and parts[1].isdigit():
pk = int(parts[1])
if parts[2] == "delete":
return self._delete_channel(conn, settings, pk)
if parts[2] == "retention":
return self._set_retention(conn, pk, form)
if parts[2] == "rescan":
return self._rescan(conn, settings, pk)
return self._send(404, templates.page("Not found", "<h1>Not found</h1>"))
finally:
conn.close()
def _login(self, settings: Settings, form: dict) -> None:
key = self._client_key()
if self.server.throttle.locked(key):
return self._send(
429, templates.login_page("Too many attempts. Wait a minute.")
)
stored = settings.raw("admin_password_hash")
if not stored:
log.warning("login attempted from %s but no password is set", key)
if stored and auth.verify_password(stored, form.get("password", "")):
self.server.throttle.record_success(key)
# Logged at INFO so "did my login work?" is answerable from the
# journal. A rejected login only ever re-renders the form, which
# looks identical to a session that failed to stick.
log.info("successful login from %s", key)
token = auth.issue_session(self._secret(settings))
return self._redirect(
"/",
{
"Set-Cookie": auth.cookie_header(
token, secure=self.server.secure_cookies
)
},
)
self.server.throttle.record_failure(key)
log.warning("failed login from %s", key)
return self._send(
401, templates.login_page(self._login_hint(settings) or "Wrong password.")
)
# ------------------------------------------------------------ actions
def _add_channel(self, conn, settings: Settings, token: str, form: dict) -> None:
url = (form.get("url") or "").strip()
try:
row = channels.subscribe(conn, settings, url)
except channels.ResolutionError as exc:
body = self._render_index(conn, settings, token, add_error=str(exc))
return self._send(400, body)
self._spawn_backfill(row["id"])
return self._redirect("/")
def _spawn_backfill(self, channel_pk: int) -> None:
"""Kick off discovery immediately rather than waiting for the hourly cron."""
try:
subprocess.Popen( # noqa: S603
[sys.executable, "-m", "ytstream", "run", "--channel",
str(channel_pk)],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
start_new_session=True,
)
except OSError as exc: # pragma: no cover
log.warning("could not spawn backfill for channel %d: %s", channel_pk, exc)
def _delete_channel(self, conn, settings: Settings, pk: int) -> None:
try:
channels.unsubscribe(conn, pk)
except LookupError:
return self._send(404, templates.page("Not found", "<h1>Not found</h1>"))
jellyfin.from_settings(settings).refresh()
return self._redirect("/")
def _set_retention(self, conn, pk: int, form: dict) -> None:
raw = (form.get("days") or "").strip()
value: int | None
if not raw:
value = None
else:
try:
value = max(1, int(raw))
except ValueError:
return self._redirect("/")
with conn:
conn.execute(
"UPDATE channel SET retention_days = ? WHERE id = ?", (value, pk)
)
return self._redirect("/")
def _rescan(self, conn, settings: Settings, pk: int) -> None:
channel = channels.get(conn, pk)
if channel is None:
return self._send(404, templates.page("Not found", "<h1>Not found</h1>"))
discovery.rescan_channel(conn, settings, channel)
self._spawn_backfill(pk)
return self._redirect("/")
def _save_settings(self, conn, settings: Settings, token: str, form: dict) -> None:
submitted = {key: form.get(key, "") for key in EDITABLE if key in form}
# A blank masked field means "keep what is stored", not "clear it".
for key in MASKED_KEYS:
if key in submitted and not submitted[key].strip():
submitted.pop(key)
errors = validate_all(submitted)
if errors:
body = self._render_index(
conn, settings, token, settings_errors=errors, submitted=submitted
)
return self._send(400, body)
for key, value in submitted.items():
settings.set(key, value.strip())
return self._redirect("/")
# ------------------------------------------------------------- render
def _render_index(
self,
conn,
settings: Settings,
token: str,
*,
add_error: str | None = None,
settings_errors: dict | None = None,
submitted: dict | None = None,
) -> bytes:
global_retention = settings.get_int("retention_days")
rows = []
for channel in channels.all_channels(conn):
stats = conn.execute(
"SELECT COUNT(*) AS n, MAX(upload_date) AS latest FROM video "
"WHERE channel_pk = ? AND state = ?",
(channel["id"], videos.MATERIALISED),
).fetchone()
latest_any = conn.execute(
"SELECT MAX(upload_date) AS latest FROM video WHERE channel_pk = ?",
(channel["id"],),
).fetchone()
rows.append(
{
"id": channel["id"],
"title": channel["title"],
"handle": channel["handle"],
"channel_id": channel["channel_id"],
"retention_days": channel["retention_days"],
"global_retention": global_retention,
"last_polled_at": channel["last_polled_at"],
"last_poll_ok": channel["last_poll_ok"],
"consecutive_poll_failures": channel["consecutive_poll_failures"],
"episodes": stats["n"],
"source": channel["source"],
"missing_syncs": channel["missing_syncs"],
"missing_threshold": settings.get_int(
"subsync_missing_threshold"),
"latest": latest_any["latest"],
}
)
values = settings.all_editable()
if submitted:
values.update(submitted)
return templates.index_page(
channels=rows,
settings_values=values,
settings_errors=settings_errors or {},
csrf=auth.csrf_token(self._secret(settings), token),
add_error=add_error,
queue_depth=videos.queue_depth(conn),
)
def serve(host: str = "127.0.0.1", port: int = 8085, *, secure_cookies: bool = True) -> None:
util.apply_umask()
server = AdminServer((host, port), Handler, secure_cookies=secure_cookies)
log.info("admin server listening on http://%s:%d", host, port)
try:
server.serve_forever()
except KeyboardInterrupt: # pragma: no cover
pass
finally:
server.server_close()
ytstream/web/templates.py
+253
View File
@@ -0,0 +1,253 @@
"""Server-rendered HTML. One embedded stylesheet, no JavaScript beyond a
confirm() on the destructive buttons."""
from __future__ import annotations
import html
from datetime import date
from .. import util
from ..settings import DEFAULTS, EDITABLE, MASKED_KEYS
STYLE = """
:root {
--bg: #14161a; --panel: #1c1f26; --line: #2c313b; --text: #e6e8ec;
--muted: #99a0ae; --accent: #6aa9ff; --warn: #ffb454; --bad: #ff6b6b;
--good: #6ade9b;
}
* { box-sizing: border-box; }
body { margin: 0; background: var(--bg); color: var(--text);
font: 15px/1.5 ui-sans-serif, system-ui, -apple-system, "Segoe UI", sans-serif; }
main { max-width: 62rem; margin: 0 auto; padding: 1.5rem 1rem 4rem; }
h1 { font-size: 1.4rem; margin: 0; }
h2 { font-size: 1.05rem; margin: 2rem 0 .75rem; color: var(--muted);
text-transform: uppercase; letter-spacing: .06em; }
header { display: flex; align-items: baseline; justify-content: space-between;
gap: 1rem; border-bottom: 1px solid var(--line); padding-bottom: .75rem; }
header .sub { color: var(--muted); font-size: .85rem; }
a { color: var(--accent); }
.panel { background: var(--panel); border: 1px solid var(--line);
border-radius: 10px; padding: 1rem; }
table { width: 100%; border-collapse: collapse; }
th, td { text-align: left; padding: .55rem .5rem; border-bottom: 1px solid var(--line);
vertical-align: middle; }
th { color: var(--muted); font-weight: 600; font-size: .78rem;
text-transform: uppercase; letter-spacing: .05em; }
td.num { text-align: right; font-variant-numeric: tabular-nums; }
.muted { color: var(--muted); }
.badge { display: inline-block; padding: .1rem .45rem; border-radius: 999px;
font-size: .75rem; border: 1px solid currentColor; }
.badge.warn { color: var(--warn); }
.badge.good { color: var(--good); }
.badge.bad { color: var(--bad); }
input[type=text], input[type=password], input[type=number], select {
background: #12141a; color: var(--text); border: 1px solid var(--line);
border-radius: 7px; padding: .45rem .55rem; font: inherit; width: 100%; }
button { background: var(--accent); color: #0b1017; border: 0; border-radius: 7px;
padding: .5rem .9rem; font: inherit; font-weight: 600; cursor: pointer; }
button.secondary { background: #2b3140; color: var(--text); }
button.danger { background: transparent; color: var(--bad);
border: 1px solid var(--bad); font-weight: 500; padding: .3rem .6rem; }
button.link { background: transparent; color: var(--accent); border: 0;
padding: .3rem .4rem; font-weight: 500; }
form.inline { display: inline; }
.row { display: flex; gap: .6rem; align-items: center; }
.grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(15rem, 1fr));
gap: .85rem; }
label { display: block; font-size: .82rem; color: var(--muted);
margin-bottom: .25rem; }
.field { margin-bottom: .3rem; }
.error { color: var(--bad); font-size: .8rem; margin-top: .2rem; }
.flash { border-radius: 8px; padding: .6rem .8rem; margin-bottom: 1rem;
border: 1px solid; }
.flash.ok { color: var(--good); border-color: var(--good); }
.flash.bad { color: var(--bad); border-color: var(--bad); }
.login { max-width: 21rem; margin: 6rem auto; }
footer { margin-top: 2.5rem; color: var(--muted); font-size: .8rem; }
@media (max-width: 40rem) {
th.hide, td.hide { display: none; }
}
"""
def _e(value) -> str:
return html.escape("" if value is None else str(value), quote=True)
def page(title: str, body: str) -> bytes:
return f"""<!doctype html>
<html lang="en"><head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>{_e(title)}</title>
<style>{STYLE}</style>
</head><body><main>{body}</main></body></html>""".encode("utf-8")
def login_page(error: str | None = None) -> bytes:
alert = f'<div class="flash bad">{_e(error)}</div>' if error else ""
body = f"""
<div class="login">
<h1>ytstream</h1>
<p class="muted">Sign in to manage subscriptions.</p>
{alert}
<form method="post" action="/login" class="panel">
<div class="field">
<label for="password">Password</label>
<input type="password" id="password" name="password" autofocus
autocomplete="current-password">
</div>
<div style="margin-top:.8rem"><button type="submit">Sign in</button></div>
</form>
<footer>You will stay signed in on this device for a year.</footer>
</div>"""
return page("Sign in — ytstream", body)
def _channel_row(channel: dict, csrf: str) -> str:
failures = channel["consecutive_poll_failures"]
if failures > 2:
badge = f'<span class="badge bad">{failures} failed polls</span>'
elif channel["last_poll_ok"] == 0:
badge = '<span class="badge warn">last poll failed</span>'
else:
badge = ""
# A channel counting up towards being unsubscribed should be visible before
# it disappears, not afterwards.
missing = ""
if channel.get("missing_syncs"):
missing = (f' <span class="badge warn">absent {channel["missing_syncs"]}'
f'/{channel.get("missing_threshold", 3)} syncs</span>')
retention = channel["retention_days"]
retention_value = "" if retention is None else str(retention)
placeholder = f"default ({channel['global_retention']})"
return f"""
<tr>
<td>
<strong>{_e(channel['title'])}</strong> {badge}<br>
<span class="muted">{_e(channel['handle'] or channel['channel_id'])}</span>
</td>
<td class="num">{channel['episodes']}</td>
<td class="hide muted">{_e(channel['source'])}{missing}</td>
<td class="hide muted">{_e(channel['latest'] or '—')}</td>
<td class="hide muted">{_e(channel['last_polled_at'] or 'never')}</td>
<td>
<form method="post" action="/channels/{channel['id']}/retention" class="row">
<input type="hidden" name="csrf" value="{_e(csrf)}">
<input type="number" name="days" min="1" style="width:6.5rem"
value="{_e(retention_value)}" placeholder="{_e(placeholder)}">
<button class="link" type="submit">save</button>
</form>
</td>
<td>
<form method="post" action="/channels/{channel['id']}/rescan" class="inline">
<input type="hidden" name="csrf" value="{_e(csrf)}">
<button class="link" type="submit"
title="Re-queue videos previously skipped as too old that the current
retention window now covers">rescan</button>
</form>
<form method="post" action="/channels/{channel['id']}/delete" class="inline"
onsubmit="return confirm('Permanently delete {_e(channel['title'])} and its whole tree? This cannot be undone.');">
<input type="hidden" name="csrf" value="{_e(csrf)}">
<button class="danger" type="submit">remove</button>
</form>
</td>
</tr>"""
def _settings_form(values: dict, errors: dict, csrf: str) -> str:
fields = []
for key in EDITABLE:
value = values.get(key, DEFAULTS[key])
error = errors.get(key)
if key in MASKED_KEYS and value:
shown, placeholder = "", "stored — leave blank to keep"
else:
shown, placeholder = value, ""
input_type = "password" if key in MASKED_KEYS else "text"
fields.append(
f"""<div class="field">
<label for="{_e(key)}">{_e(key.replace('_', ' '))}</label>
<input type="{input_type}" id="{_e(key)}" name="{_e(key)}"
value="{_e(shown)}" placeholder="{_e(placeholder)}" autocomplete="off">
{f'<div class="error">{_e(error)}</div>' if error else ''}
</div>"""
)
return f"""
<form method="post" action="/settings" class="panel">
<input type="hidden" name="csrf" value="{_e(csrf)}">
<div class="grid">{''.join(fields)}</div>
<div style="margin-top:1rem"><button type="submit">Save settings</button></div>
</form>"""
def index_page(
*,
channels: list[dict],
settings_values: dict,
settings_errors: dict,
csrf: str,
flash: tuple[str, str] | None = None,
add_error: str | None = None,
queue_depth: int = 0,
) -> bytes:
flash_html = ""
if flash:
kind, message = flash
flash_html = f'<div class="flash {kind}">{_e(message)}</div>'
if channels:
rows = "".join(_channel_row(channel, csrf) for channel in channels)
table = f"""
<div class="panel">
<table>
<thead><tr>
<th>Channel</th><th class="num">Episodes</th><th class="hide">Source</th>
<th class="hide">Latest upload</th><th class="hide">Last poll</th>
<th>Retention (days)</th><th></th>
</tr></thead>
<tbody>{rows}</tbody>
</table>
</div>"""
else:
table = '<div class="panel muted">No channels yet. Add one below.</div>'
add_error_html = f'<div class="error">{_e(add_error)}</div>' if add_error else ""
body = f"""
<header>
<h1>ytstream</h1>
<div class="sub">
{len(channels)} channel(s) · {queue_depth} queued
· <form method="post" action="/logout" class="inline">
<input type="hidden" name="csrf" value="{_e(csrf)}">
<button class="link" type="submit">sign out</button>
</form>
</div>
</header>
{flash_html}
<h2>Channels</h2>
{table}
<h2>Add a channel</h2>
<form method="post" action="/channels" class="panel">
<input type="hidden" name="csrf" value="{_e(csrf)}">
<div class="row">
<input type="text" name="url" placeholder="https://www.youtube.com/@handle, @handle, or UC..." autocomplete="off">
<button type="submit">Add</button>
</div>
{add_error_html}
</form>
<h2>Settings</h2>
{_settings_form(settings_values, settings_errors, csrf)}
<footer>Downloads run hourly. Videos are deleted once they pass the retention
window for their channel — this is a DVR, not an archive.</footer>"""
return page("ytstream", body)
ytstream/ytdlp.py
+109
View File
@@ -0,0 +1,109 @@
"""Thin wrapper around the venv's yt-dlp binary.
Everything that shells out to yt-dlp goes through here so the PATH handling (Deno
must be discoverable — see specs.md §3) and the shared extractor args live in one
place.
"""
from __future__ import annotations
import json
import logging
import os
import subprocess
import urllib.request
from pathlib import Path
from . import config
log = logging.getLogger(__name__)
class YtdlpError(RuntimeError):
pass
def binary() -> Path:
return config.VENV_BIN / "yt-dlp"
def environment() -> dict[str, str]:
"""Env for a yt-dlp subprocess.
yt-dlp locates the JS runtime by searching PATH, so the venv's bin directory
must come first — that is where Deno lives.
"""
env = dict(os.environ)
env["PATH"] = f"{config.VENV_BIN}:{env.get('PATH', '')}"
return env
def version() -> str:
result = subprocess.run(
[str(binary()), "--version"],
capture_output=True,
text=True,
env=environment(),
timeout=60,
)
if result.returncode != 0:
raise YtdlpError(result.stderr.strip() or "yt-dlp --version failed")
return result.stdout.strip()
def extractor_args(pot_provider_url: str) -> list[str]:
return [
"--extractor-args",
"youtube:player_client=default,mweb",
"--extractor-args",
f"youtubepot-bgutilhttp:base_url={pot_provider_url}",
]
def run_json(args: list[str], timeout: int = 300) -> dict:
"""Run yt-dlp with -J and parse the single JSON document it prints."""
cmd = [str(binary()), *args]
log.debug("yt-dlp %s", " ".join(args))
result = subprocess.run(
cmd, capture_output=True, text=True, env=environment(), timeout=timeout
)
if result.returncode != 0:
raise YtdlpError(first_error(result.stderr) or "yt-dlp failed")
try:
return json.loads(result.stdout)
except json.JSONDecodeError as exc:
raise YtdlpError(f"could not parse yt-dlp output: {exc}") from exc
def run(args: list[str], timeout: int = 7200) -> subprocess.CompletedProcess:
"""Run yt-dlp for its side effects, returning the completed process."""
cmd = [str(binary()), *args]
log.debug("yt-dlp %s", " ".join(args))
return subprocess.run(
cmd, capture_output=True, text=True, env=environment(), timeout=timeout
)
def first_error(stderr: str) -> str:
for line in (stderr or "").splitlines():
if line.startswith("ERROR:"):
return line[len("ERROR:") :].strip()
return (stderr or "").strip().splitlines()[-1] if stderr.strip() else ""
def pot_provider_ping(base_url: str, timeout: float = 5.0) -> dict:
"""GET /ping on the bgutil provider. Raises on any failure."""
url = base_url.rstrip("/") + "/ping"
request = urllib.request.Request(url, headers={"User-Agent": config.USER_AGENT})
with urllib.request.urlopen(request, timeout=timeout) as response:
return json.loads(response.read().decode("utf-8"))
def plugin_version() -> str | None:
"""Installed bgutil plugin version, for comparison against the server's."""
try:
from importlib.metadata import version as pkg_version
return pkg_version("bgutil-ytdlp-pot-provider")
except Exception: # pragma: no cover - only when the plugin is absent
return None