Initial implementation of youtube-automate
A DVR for YouTube subscriptions, laid out so Jellyfin presents each channel as a show and each video as an episode. Cron-driven, idempotent, with a public admin UI for a non-operator. Verified end to end on susan against three real channels: PO tokens, h264 downloads, Jellyfin resolution from local NFOs with all providers disabled, retention and tombstones. Corrections to the original design handover (specs.md documents each with the evidence, and specs.handover-original.md preserves the original): - The format sort selected 360p. Ranking acodec above res makes `bv*` prefer the combined 360p stream, which carries AAC, over the 720p video-only stream whose acodec is none. vcodec now leads, so a video without h264 at 720p yields h264 lower down rather than VP9 this hardware cannot transcode. - yt-dlp now requires a JS runtime and the yt-dlp-ejs solver scripts, which only ship with the [default] extra. Without them the n challenge fails and the mweb formats disappear entirely. - --flat-playlist carries no upload dates, so the specced client-side date filter for backfill was impossible. Backfill is RSS-first. - skipped_old was terminal, so raising a channel's retention appeared to do nothing. Added an explicit rescan. - is_upcoming premieres now defer and retry instead of being skipped forever. - TubeArchivist is gone, so the media root and the tube.jihakuz.xyz vhost were both reclaimed; the latter still pointed at its dead port. 240 offline tests, no network and no real yt-dlp invocation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Tom Flux
@@ -0,0 +1,13 @@
|
||||
# youtube-automate — add these to susan's crontab (`crontab -e`).
|
||||
#
|
||||
# HC_API_URL is already set at the top of susan's crontab; these entries follow
|
||||
# the existing one-UUID-per-job convention.
|
||||
#
|
||||
# `sg mediaserver` matches how the radio jobs run and guarantees new files are
|
||||
# group-owned by mediaserver even if the invoking shell's primary group differs.
|
||||
|
||||
# Poll, download and reap. Hourly is ample for single-digit channels.
|
||||
17 * * * * runitor -uuid 41a4d61a-7743-43d9-9b5d-d37d536e4726 -- sg mediaserver "/usr/local/bin/youtube-automate run"
|
||||
|
||||
# Keep yt-dlp current, then re-run doctor. Mondays at 04:40.
|
||||
40 4 * * 1 runitor -uuid 721e4cf0-d796-48e7-a184-79d21e1ba373 -- /opt/youtube-automate/deploy/update-ytdlp.sh
|
||||
Executable
+88
@@ -0,0 +1,88 @@
|
||||
#!/bin/bash
|
||||
# Root-requiring installation steps for youtube-automate.
|
||||
#
|
||||
# susan has no passwordless sudo, so these are collected here for the operator
|
||||
# to run in one go:
|
||||
#
|
||||
# sudo /opt/youtube-automate/deploy/deploy.sh
|
||||
#
|
||||
# Everything that does NOT need root (the venv, the database, the POT provider
|
||||
# container, subscriptions) is already handled by the application itself.
|
||||
set -euo pipefail
|
||||
|
||||
REPO=/opt/youtube-automate
|
||||
VENV=/var/lib/youtube-automate/venv
|
||||
HOSTNAME_=tube.jihakuz.xyz
|
||||
|
||||
if [[ $EUID -ne 0 ]]; then
|
||||
echo "This script needs root. Run: sudo $0" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
say() { printf '\n\033[1m==> %s\033[0m\n' "$1"; }
|
||||
|
||||
say "Installing the /usr/local/bin shim"
|
||||
cat > /usr/local/bin/youtube-automate <<EOF
|
||||
#!/bin/sh
|
||||
# Thin shim onto the venv entry point.
|
||||
exec $VENV/bin/youtube-automate "\$@"
|
||||
EOF
|
||||
chmod 0755 /usr/local/bin/youtube-automate
|
||||
chown root:automation /usr/local/bin/youtube-automate
|
||||
echo " /usr/local/bin/youtube-automate"
|
||||
|
||||
say "Making the weekly updater executable"
|
||||
chmod 0755 "$REPO/deploy/update-ytdlp.sh"
|
||||
|
||||
say "Installing the systemd unit"
|
||||
install -m 0644 "$REPO/deploy/youtube-automate.service" \
|
||||
/etc/systemd/system/youtube-automate.service
|
||||
systemctl daemon-reload
|
||||
systemctl enable --now youtube-automate.service
|
||||
systemctl --no-pager --lines=5 status youtube-automate.service || true
|
||||
|
||||
say "Installing the nginx vhost"
|
||||
# tube.jihakuz.xyz may already be claimed by the leftover TubeArchivist server
|
||||
# block in sites-available/jihakuz.xyz. nginx uses the FIRST server block
|
||||
# matching a name, and that file loads first, so installing a second vhost for
|
||||
# the same name silently does nothing (and yields 502 from the dead upstream).
|
||||
if grep -rql --dereference-recursive "server_name $HOSTNAME_" /etc/nginx/sites-enabled/ \
|
||||
2>/dev/null | grep -qv "$HOSTNAME_\$"; then
|
||||
echo " $HOSTNAME_ is already served by an existing vhost."
|
||||
echo " Skipping the standalone file — run deploy/fix-nginx-tube.sh instead."
|
||||
else
|
||||
install -m 0644 "$REPO/deploy/nginx-tube.jihakuz.xyz.conf" \
|
||||
"/etc/nginx/sites-available/$HOSTNAME_"
|
||||
ln -sfn "/etc/nginx/sites-available/$HOSTNAME_" \
|
||||
"/etc/nginx/sites-enabled/$HOSTNAME_"
|
||||
nginx -t
|
||||
systemctl reload nginx
|
||||
fi
|
||||
|
||||
cat <<EOF
|
||||
|
||||
==> Done. Remaining manual steps, in order:
|
||||
|
||||
1. Issue the certificate (needs port 80 reachable from the internet):
|
||||
|
||||
sudo certbot --nginx -d $HOSTNAME_
|
||||
|
||||
2. Set the admin password (prompts; never pass it as an argument):
|
||||
|
||||
youtube-automate set-password
|
||||
|
||||
3. Add the two cron entries as susan (NOT root):
|
||||
|
||||
crontab -e
|
||||
# then paste $REPO/deploy/crontab.fragment
|
||||
|
||||
4. Confirm everything is healthy:
|
||||
|
||||
youtube-automate doctor
|
||||
|
||||
Note: the bgutil POT provider container starts itself on boot via
|
||||
--restart unless-stopped, so it needs nothing here. If it is ever missing:
|
||||
|
||||
docker run --name bgutil-provider -d --restart unless-stopped --init \\
|
||||
-p 127.0.0.1:4416:4416 brainicism/bgutil-ytdlp-pot-provider:1.3.1-deno
|
||||
EOF
|
||||
Executable
+66
@@ -0,0 +1,66 @@
|
||||
#!/bin/bash
|
||||
# Repoint the existing tube.jihakuz.xyz vhost at youtube-automate.
|
||||
#
|
||||
# Why this exists: tube.jihakuz.xyz was already served by a leftover
|
||||
# TubeArchivist server block inside /etc/nginx/sites-available/jihakuz.xyz,
|
||||
# proxying to 127.0.0.1:8003. That block loads before the standalone vhost
|
||||
# deploy.sh installs (nginx takes the first server block matching a name), so
|
||||
# every request went to the dead TubeArchivist port and returned 502.
|
||||
#
|
||||
# That old block already owns the Let's Encrypt certificate for the hostname,
|
||||
# so the right fix is to repoint it rather than duplicate it — no second
|
||||
# certbot run needed.
|
||||
#
|
||||
# sudo /opt/youtube-automate/deploy/fix-nginx-tube.sh
|
||||
set -euo pipefail
|
||||
|
||||
CONF=/etc/nginx/sites-available/jihakuz.xyz
|
||||
STANDALONE=/etc/nginx/sites-enabled/tube.jihakuz.xyz
|
||||
OLD_PORT=8003
|
||||
NEW_PORT=8085
|
||||
|
||||
if [[ $EUID -ne 0 ]]; then
|
||||
echo "This script needs root. Run: sudo $0" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if ! grep -q "127.0.0.1:${OLD_PORT}" "$CONF"; then
|
||||
if grep -q "127.0.0.1:${NEW_PORT}" "$CONF"; then
|
||||
echo "Already repointed at ${NEW_PORT}; nothing to do."
|
||||
exit 0
|
||||
fi
|
||||
echo "Did not find 127.0.0.1:${OLD_PORT} in $CONF — nothing to change." >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
BACKUP="${CONF}.bak-$(date +%Y%m%d%H%M%S)"
|
||||
cp -a "$CONF" "$BACKUP"
|
||||
echo "==> Backed up $CONF to $BACKUP"
|
||||
|
||||
# X-Forwarded-For is not cosmetic: the app throttles failed logins per client
|
||||
# address and reads it from that header. Without it every attempt looks like it
|
||||
# came from nginx itself, so one attacker would lock out everybody.
|
||||
sed -i \
|
||||
"s|proxy_pass http://127.0.0.1:${OLD_PORT};|proxy_pass http://127.0.0.1:${NEW_PORT};\n\t\tproxy_http_version 1.1;\n\t\tproxy_set_header Host \$host;\n\t\tproxy_set_header X-Real-IP \$remote_addr;\n\t\tproxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;\n\t\tproxy_set_header X-Forwarded-Proto \$scheme;|" \
|
||||
"$CONF"
|
||||
echo "==> Repointed tube.jihakuz.xyz at 127.0.0.1:${NEW_PORT}"
|
||||
|
||||
# The standalone vhost is now redundant and would only produce a
|
||||
# "conflicting server name" warning.
|
||||
if [[ -L "$STANDALONE" || -f "$STANDALONE" ]]; then
|
||||
rm -f "$STANDALONE"
|
||||
echo "==> Removed the redundant standalone vhost $STANDALONE"
|
||||
fi
|
||||
|
||||
if ! nginx -t; then
|
||||
echo "!! nginx config test failed — restoring the backup" >&2
|
||||
cp -a "$BACKUP" "$CONF"
|
||||
nginx -t
|
||||
exit 1
|
||||
fi
|
||||
|
||||
systemctl reload nginx
|
||||
echo "==> nginx reloaded"
|
||||
echo
|
||||
echo "Verify with:"
|
||||
echo " curl -sI https://tube.jihakuz.xyz/ | head -1 # expect 303 -> /login"
|
||||
@@ -0,0 +1,49 @@
|
||||
# youtube-automate admin UI — tube.jihakuz.xyz
|
||||
#
|
||||
# NOTE: on susan this file is NOT the vhost in use. tube.jihakuz.xyz was already
|
||||
# served by a leftover TubeArchivist server block inside
|
||||
# sites-available/jihakuz.xyz (proxying to the now-dead 127.0.0.1:8003), and
|
||||
# nginx uses the first server block matching a name. That block also already
|
||||
# owns the Let's Encrypt certificate, so the fix was to repoint it — see
|
||||
# deploy/fix-nginx-tube.sh. This file is kept as the reference config for a
|
||||
# clean install on a host without that history.
|
||||
#
|
||||
# Install this as /etc/nginx/sites-available/tube.jihakuz.xyz and symlink it into
|
||||
# sites-enabled, then run:
|
||||
#
|
||||
# sudo certbot --nginx -d tube.jihakuz.xyz
|
||||
#
|
||||
# certbot rewrites this file to add the TLS server block and the 80->443
|
||||
# redirect, matching how the other vhosts on susan are set up.
|
||||
#
|
||||
# The DNS record and njal.la update key for tube.jihakuz.xyz already exist in
|
||||
# ~/.local/bin/update-dns.sh, so no DNS work is needed.
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
listen [::]:80;
|
||||
server_name tube.jihakuz.xyz;
|
||||
|
||||
# Small admin forms only; nothing here accepts uploads.
|
||||
client_max_body_size 256k;
|
||||
|
||||
# Belt and braces — the app sets these too, but a misconfigured upstream
|
||||
# should not be able to drop them.
|
||||
add_header X-Content-Type-Options nosniff always;
|
||||
add_header X-Frame-Options DENY always;
|
||||
add_header Referrer-Policy same-origin always;
|
||||
|
||||
location / {
|
||||
proxy_pass http://127.0.0.1:8085;
|
||||
proxy_http_version 1.1;
|
||||
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
# The app throttles failed logins per client address and reads it from
|
||||
# this header, so it must be set correctly.
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
proxy_read_timeout 120s;
|
||||
}
|
||||
}
|
||||
Executable
+20
@@ -0,0 +1,20 @@
|
||||
#!/bin/bash
|
||||
# Weekly yt-dlp update, run under runitor from susan's crontab.
|
||||
#
|
||||
# specs.md §12 calls a stale yt-dlp the single most likely cause of "everything
|
||||
# broke", so this is monitored with its own Healthchecks UUID. It deliberately
|
||||
# does NOT touch the bgutil plugin: that is pinned to match the container tag,
|
||||
# and upgrading one half alone is how you get silent version skew.
|
||||
#
|
||||
# doctor runs afterwards so a successful install that nonetheless leaves the
|
||||
# stack broken still fails the check.
|
||||
set -euo pipefail
|
||||
|
||||
VENV=/var/lib/youtube-automate/venv
|
||||
UV=/home/susan/.local/bin/uv
|
||||
|
||||
"$UV" pip install --python "$VENV" --quiet --upgrade 'yt-dlp[default]'
|
||||
|
||||
echo "yt-dlp now: $("$VENV/bin/yt-dlp" --version)"
|
||||
|
||||
exec "$VENV/bin/youtube-automate" doctor
|
||||
@@ -0,0 +1,31 @@
|
||||
[Unit]
|
||||
Description=youtube-automate admin server
|
||||
Documentation=file:///opt/youtube-automate/specs.md
|
||||
After=network-online.target
|
||||
Wants=network-online.target
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
# Group=mediaserver and UMask=0002 are load-bearing: Jellyfin reaches the media
|
||||
# tree only through the mediaserver group, and anything this process writes must
|
||||
# stay group-readable. See specs.md §2.
|
||||
User=susan
|
||||
Group=mediaserver
|
||||
UMask=0002
|
||||
|
||||
WorkingDirectory=/opt/youtube-automate
|
||||
ExecStart=/var/lib/youtube-automate/venv/bin/youtube-automate serve --host 127.0.0.1 --port 8085
|
||||
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectSystem=full
|
||||
ProtectControlGroups=true
|
||||
ProtectKernelTunables=true
|
||||
RestrictSUIDSGID=true
|
||||
ReadWritePaths=/var/lib/youtube-automate /disks/Plex/YouTube
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
Reference in New Issue
Block a user