Build ytstream: catalogue, retention, subscription mirror, proxy
Phases 1-4 of plan.md §13. Forked from youtube-automate as planned rather than
written from scratch: naming, NFO, auth, the admin UI, settings and the DB layer
came across largely unchanged, download.py is gone, and the pieces that only make
sense for a streaming library are new.
api.py YouTube Data API v3 client. The whole metadata path.
strm.py Materialising: a .strm, an .nfo and a thumbnail. Replaces the
330-line download.py, because the job is writing a URL to a file.
subsync.py The subscription mirror, most of which is refusals.
reap.py Retention, rewritten around the 30-day window and min_keep_videos.
discovery.py RSS polling plus an API-backed, resumable, bounded backfill.
proxy/ The verified PoC, moved in with a systemd unit.
330 tests, all passing, no network and no yt-dlp in any of them. The suite leans
towards the failure paths, because that is where this design can actually hurt
someone: a 403 that looks like an unsubscribe, a video that ages out and comes
back, a title that never arrives. tests/test_proxy.py replaces the two standalone
scripts under proxy/ and now drives the real make_handler(mgr, ...) rather than the
PoC's make_handler(path, done), so routing and video-id validation are covered too.
Ran it end to end against the live API and it found three real bugs.
The first was mine and the tests caught it: strm.remove pruned empty directories
up to the media root, so a channel directory whose tvshow.nfo happened to be
missing would be deleted along with the season. It only looked safe because
tvshow.nfo normally stops the walk. The prune boundary is now the channel
directory explicitly.
The other two only showed up against real data, and they compounded. The backfill
inserted rows with no title and left the RSS poll to fill them in — but RSS returns
15 entries, which for Pitch Side spans 23 days against a 30-day window, so five of
twenty episodes were named after their video ids. Worse, strm.materialise wrote
that fallback back to the database as the title, which made the row look titled and
permanently disabled the repair path. Both fixed: playlistItems.list now requests
snippet as well as contentDetails, which costs the same single quota unit and
carries the title alongside the exact publish date, and the fallback is used for the
filename without being persisted. A title that does arrive late now also removes the
badly-named files and re-queues, so the episode is rewritten rather than keeping its
video-id name forever. Verified against the live API: all twenty Pitch Side episodes
now carry real titles.
Measured on the real account: 119 subscriptions queued for approval and none added
on the first sync, then a two-channel run backfilled and materialised 26 episodes in
under seven seconds.
Two deliberate departures from plan.md, both recorded there:
min_keep_videos defaults to 5 rather than being left as an open question. Without
it 52 of 117 measured channels are empty Jellyfin series that flicker in and out as
their single video crosses the retention line, and the plan already recommended it.
The Jellyfin refresh is a bare /Library/Refresh with a comment explaining why it
must stay that way. A normal scan makes zero media probes; FullRefresh does probe,
and at 400 episodes that is 400 cold starts.
Not yet done: no systemd units are installed (needs root — deploy/deploy.sh), the
admin UI has no routes for sources or the approval queue yet, and nothing has been
pointed at the real media root.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Tom Flux
co-authored by
Claude Opus 5
Claude Opus 5
parent
f640c064c6
commit
155f05773d
Executable
+46
@@ -0,0 +1,46 @@
|
||||
#!/bin/bash
|
||||
# Everything that does NOT need root. Run as susan, before deploy.sh.
|
||||
#
|
||||
# /opt/ytstream/deploy/bootstrap.sh
|
||||
set -euo pipefail
|
||||
|
||||
STATE=/var/lib/ytstream
|
||||
VENV=$STATE/venv
|
||||
REPO=/opt/ytstream
|
||||
|
||||
say() { printf '\n\033[1m==> %s\033[0m\n' "$1"; }
|
||||
|
||||
if [[ ! -d $STATE ]]; then
|
||||
echo "$STATE does not exist yet — run 'sudo $REPO/deploy/deploy.sh' first," >&2
|
||||
echo "or create it with: sudo install -d -o susan -g automation -m 0770 $STATE" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
say "Creating the virtualenv"
|
||||
python3 -m venv "$VENV"
|
||||
"$VENV/bin/pip" install --quiet --upgrade pip
|
||||
"$VENV/bin/pip" install --quiet -e "$REPO"
|
||||
|
||||
say "Verifying yt-dlp and the POT plugin"
|
||||
# The plugin is what makes YouTube reachable at all. A venv without it looks fine
|
||||
# until the first playback fails with a bare 403.
|
||||
"$VENV/bin/yt-dlp" --version
|
||||
"$VENV/bin/python3" - <<'PY'
|
||||
import importlib.util
|
||||
missing = [name for name in ("yt_dlp_plugins",) if importlib.util.find_spec(name) is None]
|
||||
print("POT plugin:", "MISSING" if missing else "present")
|
||||
raise SystemExit(1 if missing else 0)
|
||||
PY
|
||||
|
||||
say "Starting the POT provider container if it is not already up"
|
||||
if ! curl -sf --max-time 5 http://127.0.0.1:4416/ping >/dev/null 2>&1; then
|
||||
docker run -d --name bgutil-provider --restart unless-stopped \
|
||||
-p 4416:4416 brainicism/bgutil-ytdlp-pot-provider || true
|
||||
else
|
||||
echo " already responding on 127.0.0.1:4416"
|
||||
fi
|
||||
|
||||
say "Initialising the database"
|
||||
"$VENV/bin/ytstream" status || true
|
||||
|
||||
say "Done — now run 'sudo $REPO/deploy/deploy.sh'"
|
||||
@@ -0,0 +1,15 @@
|
||||
# ytstream — add these to susan's crontab (`crontab -e`).
|
||||
#
|
||||
# HC_API_URL is already set at the top of susan's crontab; these follow the
|
||||
# existing one-UUID-per-job convention. `sg mediaserver` guarantees new files are
|
||||
# group-owned by mediaserver even if the invoking shell's primary group differs.
|
||||
#
|
||||
# Get fresh UUIDs from hc.jihakuz.xyz before enabling these.
|
||||
|
||||
# Sync subscriptions, poll, materialise, reap. Hourly. Runs at :23 to stay clear
|
||||
# of youtube-automate's :17 entry during the overlap period.
|
||||
23 * * * * runitor -uuid REPLACE-WITH-UUID -- sg mediaserver "/usr/local/bin/ytstream run"
|
||||
|
||||
# Keep yt-dlp current — this is the thing that keeps playback working as YouTube
|
||||
# changes. Mondays 04:50, after youtube-automate's 04:40 slot.
|
||||
50 4 * * 1 runitor -uuid REPLACE-WITH-UUID -- /opt/ytstream/deploy/update-ytdlp.sh
|
||||
Executable
+89
@@ -0,0 +1,89 @@
|
||||
#!/bin/bash
|
||||
# Root-requiring installation steps for ytstream.
|
||||
#
|
||||
# susan has no passwordless sudo, so everything needing root is collected here for
|
||||
# the operator to run in one go:
|
||||
#
|
||||
# sudo /opt/ytstream/deploy/deploy.sh
|
||||
#
|
||||
# Everything that does NOT need root — the venv, the database, the POT provider
|
||||
# container, subscriptions, the API key — is handled by `bootstrap.sh` and the
|
||||
# application itself. Run bootstrap.sh (as susan) first.
|
||||
set -euo pipefail
|
||||
|
||||
REPO=/opt/ytstream
|
||||
STATE=/var/lib/ytstream
|
||||
VENV=$STATE/venv
|
||||
HOSTNAME_=tube.jihakuz.xyz
|
||||
|
||||
if [[ $EUID -ne 0 ]]; then
|
||||
echo "This script needs root. Run: sudo $0" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
say() { printf '\n\033[1m==> %s\033[0m\n' "$1"; }
|
||||
|
||||
say "Creating $STATE"
|
||||
# root-owned directory, group-writable by `automation` so susan's cron job and the
|
||||
# admin server can both write the database.
|
||||
install -d -o susan -g automation -m 0770 "$STATE"
|
||||
|
||||
say "Installing the /usr/local/bin shim"
|
||||
cat > /usr/local/bin/ytstream <<EOF
|
||||
#!/bin/sh
|
||||
# Thin shim onto the venv entry point.
|
||||
exec $VENV/bin/ytstream "\$@"
|
||||
EOF
|
||||
chmod 0755 /usr/local/bin/ytstream
|
||||
chown root:automation /usr/local/bin/ytstream
|
||||
echo " /usr/local/bin/ytstream"
|
||||
|
||||
say "Preparing the media root"
|
||||
# setgid so new directories inherit `mediaserver` — without it Jellyfin loses
|
||||
# access to anything created after the fact. See plan.md §9.
|
||||
install -d -o susan -g mediaserver -m 2770 /disks/Plex/_ytstream
|
||||
|
||||
say "Installing systemd units"
|
||||
for unit in ytstream-proxy ytstream-admin; do
|
||||
install -m 0644 "$REPO/deploy/$unit.service" "/etc/systemd/system/$unit.service"
|
||||
echo " $unit.service"
|
||||
done
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now ytstream-proxy.service ytstream-admin.service
|
||||
for unit in ytstream-proxy ytstream-admin; do
|
||||
systemctl --no-pager --lines=5 status "$unit.service" || true
|
||||
done
|
||||
|
||||
say "nginx vhost for $HOSTNAME_"
|
||||
# tube.jihakuz.xyz is served by a leftover TubeArchivist server block inside
|
||||
# sites-available/jihakuz.xyz, which owns the Let's Encrypt certificate and wins
|
||||
# because nginx uses the FIRST server block matching a name. Installing a second
|
||||
# vhost for the same name silently does nothing. youtube-automate repointed that
|
||||
# block at 8085; ytstream needs it on 8086.
|
||||
if grep -rql "server_name $HOSTNAME_" /etc/nginx/sites-enabled/ 2>/dev/null; then
|
||||
echo " $HOSTNAME_ is already served by an existing vhost."
|
||||
echo " Repoint its proxy_pass to http://127.0.0.1:8086 by hand, then:"
|
||||
echo " nginx -t && systemctl reload nginx"
|
||||
echo " (Deliberately not edited automatically — that block also serves"
|
||||
echo " other names and owns the TLS certificate.)"
|
||||
else
|
||||
echo " No existing vhost found. Install one proxying to 127.0.0.1:8086"
|
||||
echo " and run: certbot --nginx -d $HOSTNAME_"
|
||||
fi
|
||||
|
||||
say "Done"
|
||||
cat <<'EOF'
|
||||
Remaining steps, all as susan and none needing root:
|
||||
|
||||
ytstream set-password # admin UI login
|
||||
ytstream set-jellyfin-key # verified against the live server
|
||||
ytstream set-youtube-key # verified against the live API
|
||||
ytstream add-source @cflux1030 # the mirrored account
|
||||
ytstream sync # queues the subscriptions
|
||||
ytstream pending # review them
|
||||
ytstream approve --all # or approve a subset by id
|
||||
ytstream run # first real cycle
|
||||
ytstream doctor # confirm everything is wired up
|
||||
|
||||
Then add the cron entries from deploy/crontab.fragment to susan's crontab.
|
||||
EOF
|
||||
Executable
+23
@@ -0,0 +1,23 @@
|
||||
#!/bin/bash
|
||||
# Keep yt-dlp current inside ytstream's venv, then confirm the install still works.
|
||||
#
|
||||
# This is the single most important recurring job: YouTube changes its player and
|
||||
# streaming protocol frequently, and a stale yt-dlp means every playback fails.
|
||||
set -euo pipefail
|
||||
|
||||
VENV=/var/lib/ytstream/venv
|
||||
|
||||
before=$("$VENV/bin/yt-dlp" --version 2>/dev/null || echo "none")
|
||||
"$VENV/bin/pip" install --quiet --upgrade "yt-dlp[default]"
|
||||
after=$("$VENV/bin/yt-dlp" --version)
|
||||
|
||||
echo "yt-dlp: $before -> $after"
|
||||
|
||||
if [[ "$before" != "$after" ]]; then
|
||||
# A new yt-dlp can change format availability, so restart the proxy to drop
|
||||
# any cached extraction state and re-run the checks.
|
||||
systemctl restart ytstream-proxy.service 2>/dev/null \
|
||||
|| echo "could not restart the proxy (needs root); do it by hand"
|
||||
fi
|
||||
|
||||
exec "$VENV/bin/ytstream" doctor
|
||||
@@ -0,0 +1,34 @@
|
||||
[Unit]
|
||||
Description=ytstream admin server
|
||||
Documentation=file:///opt/ytstream/plan.md
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
# See plan.md §9 — the same ownership rules as the proxy, because subscribing a
|
||||
# channel through the UI writes into the media tree.
|
||||
User=susan
|
||||
Group=mediaserver
|
||||
UMask=0002
|
||||
|
||||
Environment=PATH=/var/lib/ytstream/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
|
||||
Environment=PYTHONUNBUFFERED=1
|
||||
|
||||
WorkingDirectory=/opt/ytstream
|
||||
# Port 8086: youtube-automate holds 8085 for as long as the two run side by side.
|
||||
ExecStart=/var/lib/ytstream/venv/bin/ytstream serve --host 127.0.0.1 --port 8086
|
||||
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectSystem=full
|
||||
ProtectControlGroups=true
|
||||
ProtectKernelTunables=true
|
||||
RestrictSUIDSGID=true
|
||||
ReadWritePaths=/var/lib/ytstream /disks/Plex/_ytstream
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,46 @@
|
||||
[Unit]
|
||||
Description=ytstream just-in-time YouTube streaming proxy
|
||||
Documentation=file:///opt/ytstream/plan.md
|
||||
After=network-online.target docker.service
|
||||
Wants=network-online.target
|
||||
# The bgutil POT provider runs in Docker on 127.0.0.1:4416 and yt-dlp cannot
|
||||
# fetch anything without it.
|
||||
Requires=docker.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
# Group=mediaserver and UMask=0002 are load-bearing: Jellyfin reaches the media
|
||||
# tree only through the mediaserver group. See plan.md §9.
|
||||
User=susan
|
||||
Group=mediaserver
|
||||
UMask=0002
|
||||
|
||||
# LOAD-BEARING. The only yt-dlp with the bgutil POT plugin is the one in this
|
||||
# venv. /usr/local/bin/yt-dlp is a 2023.11.16 binary that cannot talk to YouTube
|
||||
# at all, and if it wins the PATH race every playback fails with no clear reason.
|
||||
Environment=PATH=/var/lib/ytstream/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
|
||||
Environment=PYTHONUNBUFFERED=1
|
||||
|
||||
WorkingDirectory=/opt/ytstream
|
||||
ExecStart=/var/lib/ytstream/venv/bin/python3 /opt/ytstream/proxy/ytstream_proxy.py \
|
||||
--host 127.0.0.1 --port 8099 \
|
||||
--work /dev/shm/ytstream \
|
||||
--cache-gb 8 \
|
||||
--max-pipelines 2 \
|
||||
--max-retries 2 \
|
||||
--max-starts 20 --starts-window 3600
|
||||
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectSystem=full
|
||||
ProtectControlGroups=true
|
||||
ProtectKernelTunables=true
|
||||
RestrictSUIDSGID=true
|
||||
# /dev/shm is the cache; nothing else needs to be writable.
|
||||
ReadWritePaths=/dev/shm
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
Reference in New Issue
Block a user