tom/youtube-automate
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial implementation of youtube-automate

Browse Source 
A DVR for YouTube subscriptions, laid out so Jellyfin presents each channel
as a show and each video as an episode. Cron-driven, idempotent, with a
public admin UI for a non-operator.

Verified end to end on susan against three real channels: PO tokens, h264
downloads, Jellyfin resolution from local NFOs with all providers disabled,
retention and tombstones.

Corrections to the original design handover (specs.md documents each with
the evidence, and specs.handover-original.md preserves the original):

- The format sort selected 360p. Ranking acodec above res makes `bv*` prefer
  the combined 360p stream, which carries AAC, over the 720p video-only
  stream whose acodec is none. vcodec now leads, so a video without h264 at
  720p yields h264 lower down rather than VP9 this hardware cannot transcode.
- yt-dlp now requires a JS runtime and the yt-dlp-ejs solver scripts, which
  only ship with the [default] extra. Without them the n challenge fails and
  the mweb formats disappear entirely.
- --flat-playlist carries no upload dates, so the specced client-side date
  filter for backfill was impossible. Backfill is RSS-first.
- skipped_old was terminal, so raising a channel's retention appeared to do
  nothing. Added an explicit rescan.
- is_upcoming premieres now defer and retry instead of being skipped forever.
- TubeArchivist is gone, so the media root and the tube.jihakuz.xyz vhost
  were both reclaimed; the latter still pointed at its dead port.

240 offline tests, no network and no real yt-dlp invocation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Tom Flux
2026-08-11 21:42:48 +01:00
co-authored by Claude Opus 5
commit 18bb2e420b
44 changed files with 7188 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+5
View File
@@ -0,0 +1,5 @@
__pycache__/
*.py[cod]
*.egg-info/
.pytest_cache/
.venv/
deploy/crontab.fragment
+13
View File
@@ -0,0 +1,13 @@
# youtube-automate — add these to susan's crontab (`crontab -e`).
#
# HC_API_URL is already set at the top of susan's crontab; these entries follow
# the existing one-UUID-per-job convention.
#
# `sg mediaserver` matches how the radio jobs run and guarantees new files are
# group-owned by mediaserver even if the invoking shell's primary group differs.
# Poll, download and reap. Hourly is ample for single-digit channels.
17 * * * * runitor -uuid 41a4d61a-7743-43d9-9b5d-d37d536e4726 -- sg mediaserver "/usr/local/bin/youtube-automate run"
# Keep yt-dlp current, then re-run doctor. Mondays at 04:40.
40 4 * * 1 runitor -uuid 721e4cf0-d796-48e7-a184-79d21e1ba373 -- /opt/youtube-automate/deploy/update-ytdlp.sh
deploy/deploy.sh
Executable
+88
View File
@@ -0,0 +1,88 @@
#!/bin/bash
# Root-requiring installation steps for youtube-automate.
#
# susan has no passwordless sudo, so these are collected here for the operator
# to run in one go:
#
# sudo /opt/youtube-automate/deploy/deploy.sh
#
# Everything that does NOT need root (the venv, the database, the POT provider
# container, subscriptions) is already handled by the application itself.
set -euo pipefail
REPO=/opt/youtube-automate
VENV=/var/lib/youtube-automate/venv
HOSTNAME_=tube.jihakuz.xyz
if [[ $EUID -ne 0 ]]; then
echo "This script needs root. Run: sudo $0" >&2
exit 1
fi
say() { printf '\n\033[1m==> %s\033[0m\n' "$1"; }
say "Installing the /usr/local/bin shim"
cat > /usr/local/bin/youtube-automate <<EOF
#!/bin/sh
# Thin shim onto the venv entry point.
exec $VENV/bin/youtube-automate "\$@"
EOF
chmod 0755 /usr/local/bin/youtube-automate
chown root:automation /usr/local/bin/youtube-automate
echo " /usr/local/bin/youtube-automate"
say "Making the weekly updater executable"
chmod 0755 "$REPO/deploy/update-ytdlp.sh"
say "Installing the systemd unit"
install -m 0644 "$REPO/deploy/youtube-automate.service" \
/etc/systemd/system/youtube-automate.service
systemctl daemon-reload
systemctl enable --now youtube-automate.service
systemctl --no-pager --lines=5 status youtube-automate.service || true
say "Installing the nginx vhost"
# tube.jihakuz.xyz may already be claimed by the leftover TubeArchivist server
# block in sites-available/jihakuz.xyz. nginx uses the FIRST server block
# matching a name, and that file loads first, so installing a second vhost for
# the same name silently does nothing (and yields 502 from the dead upstream).
if grep -rql --dereference-recursive "server_name $HOSTNAME_" /etc/nginx/sites-enabled/ \
2>/dev/null | grep -qv "$HOSTNAME_\$"; then
echo " $HOSTNAME_ is already served by an existing vhost."
echo " Skipping the standalone file — run deploy/fix-nginx-tube.sh instead."
else
install -m 0644 "$REPO/deploy/nginx-tube.jihakuz.xyz.conf" \
"/etc/nginx/sites-available/$HOSTNAME_"
ln -sfn "/etc/nginx/sites-available/$HOSTNAME_" \
"/etc/nginx/sites-enabled/$HOSTNAME_"
nginx -t
systemctl reload nginx
fi
cat <<EOF
==> Done. Remaining manual steps, in order:
1. Issue the certificate (needs port 80 reachable from the internet):
sudo certbot --nginx -d $HOSTNAME_
2. Set the admin password (prompts; never pass it as an argument):
youtube-automate set-password
3. Add the two cron entries as susan (NOT root):
crontab -e
# then paste $REPO/deploy/crontab.fragment
4. Confirm everything is healthy:
youtube-automate doctor
Note: the bgutil POT provider container starts itself on boot via
--restart unless-stopped, so it needs nothing here. If it is ever missing:
docker run --name bgutil-provider -d --restart unless-stopped --init \\
-p 127.0.0.1:4416:4416 brainicism/bgutil-ytdlp-pot-provider:1.3.1-deno
EOF
deploy/fix-nginx-tube.sh
Executable
+66
View File
@@ -0,0 +1,66 @@
#!/bin/bash
# Repoint the existing tube.jihakuz.xyz vhost at youtube-automate.
#
# Why this exists: tube.jihakuz.xyz was already served by a leftover
# TubeArchivist server block inside /etc/nginx/sites-available/jihakuz.xyz,
# proxying to 127.0.0.1:8003. That block loads before the standalone vhost
# deploy.sh installs (nginx takes the first server block matching a name), so
# every request went to the dead TubeArchivist port and returned 502.
#
# That old block already owns the Let's Encrypt certificate for the hostname,
# so the right fix is to repoint it rather than duplicate it — no second
# certbot run needed.
#
# sudo /opt/youtube-automate/deploy/fix-nginx-tube.sh
set -euo pipefail
CONF=/etc/nginx/sites-available/jihakuz.xyz
STANDALONE=/etc/nginx/sites-enabled/tube.jihakuz.xyz
OLD_PORT=8003
NEW_PORT=8085
if [[ $EUID -ne 0 ]]; then
echo "This script needs root. Run: sudo $0" >&2
exit 1
fi
if ! grep -q "127.0.0.1:${OLD_PORT}" "$CONF"; then
if grep -q "127.0.0.1:${NEW_PORT}" "$CONF"; then
echo "Already repointed at ${NEW_PORT}; nothing to do."
exit 0
fi
echo "Did not find 127.0.0.1:${OLD_PORT} in $CONF — nothing to change." >&2
exit 1
fi
BACKUP="${CONF}.bak-$(date +%Y%m%d%H%M%S)"
cp -a "$CONF" "$BACKUP"
echo "==> Backed up $CONF to $BACKUP"
# X-Forwarded-For is not cosmetic: the app throttles failed logins per client
# address and reads it from that header. Without it every attempt looks like it
# came from nginx itself, so one attacker would lock out everybody.
sed -i \
"s|proxy_pass http://127.0.0.1:${OLD_PORT};|proxy_pass http://127.0.0.1:${NEW_PORT};\n\t\tproxy_http_version 1.1;\n\t\tproxy_set_header Host \$host;\n\t\tproxy_set_header X-Real-IP \$remote_addr;\n\t\tproxy_set_header X-Forwarded-For \$proxy_add_x_forwarded_for;\n\t\tproxy_set_header X-Forwarded-Proto \$scheme;|" \
"$CONF"
echo "==> Repointed tube.jihakuz.xyz at 127.0.0.1:${NEW_PORT}"
# The standalone vhost is now redundant and would only produce a
# "conflicting server name" warning.
if [[ -L "$STANDALONE" || -f "$STANDALONE" ]]; then
rm -f "$STANDALONE"
echo "==> Removed the redundant standalone vhost $STANDALONE"
fi
if ! nginx -t; then
echo "!! nginx config test failed — restoring the backup" >&2
cp -a "$BACKUP" "$CONF"
nginx -t
exit 1
fi
systemctl reload nginx
echo "==> nginx reloaded"
echo
echo "Verify with:"
echo " curl -sI https://tube.jihakuz.xyz/ | head -1 # expect 303 -> /login"
deploy/nginx-tube.jihakuz.xyz.conf
+49
View File
@@ -0,0 +1,49 @@
# youtube-automate admin UI — tube.jihakuz.xyz
#
# NOTE: on susan this file is NOT the vhost in use. tube.jihakuz.xyz was already
# served by a leftover TubeArchivist server block inside
# sites-available/jihakuz.xyz (proxying to the now-dead 127.0.0.1:8003), and
# nginx uses the first server block matching a name. That block also already
# owns the Let's Encrypt certificate, so the fix was to repoint it — see
# deploy/fix-nginx-tube.sh. This file is kept as the reference config for a
# clean install on a host without that history.
#
# Install this as /etc/nginx/sites-available/tube.jihakuz.xyz and symlink it into
# sites-enabled, then run:
#
# sudo certbot --nginx -d tube.jihakuz.xyz
#
# certbot rewrites this file to add the TLS server block and the 80->443
# redirect, matching how the other vhosts on susan are set up.
#
# The DNS record and njal.la update key for tube.jihakuz.xyz already exist in
# ~/.local/bin/update-dns.sh, so no DNS work is needed.
server {
listen 80;
listen [::]:80;
server_name tube.jihakuz.xyz;
# Small admin forms only; nothing here accepts uploads.
client_max_body_size 256k;
# Belt and braces — the app sets these too, but a misconfigured upstream
# should not be able to drop them.
add_header X-Content-Type-Options nosniff always;
add_header X-Frame-Options DENY always;
add_header Referrer-Policy same-origin always;
location / {
proxy_pass http://127.0.0.1:8085;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
# The app throttles failed logins per client address and reads it from
# this header, so it must be set correctly.
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_read_timeout 120s;
}
}
deploy/update-ytdlp.sh
Executable
+20
View File
@@ -0,0 +1,20 @@
#!/bin/bash
# Weekly yt-dlp update, run under runitor from susan's crontab.
#
# specs.md §12 calls a stale yt-dlp the single most likely cause of "everything
# broke", so this is monitored with its own Healthchecks UUID. It deliberately
# does NOT touch the bgutil plugin: that is pinned to match the container tag,
# and upgrading one half alone is how you get silent version skew.
#
# doctor runs afterwards so a successful install that nonetheless leaves the
# stack broken still fails the check.
set -euo pipefail
VENV=/var/lib/youtube-automate/venv
UV=/home/susan/.local/bin/uv
"$UV" pip install --python "$VENV" --quiet --upgrade 'yt-dlp[default]'
echo "yt-dlp now: $("$VENV/bin/yt-dlp" --version)"
exec "$VENV/bin/youtube-automate" doctor
deploy/youtube-automate.service
+31
View File
@@ -0,0 +1,31 @@
[Unit]
Description=youtube-automate admin server
Documentation=file:///opt/youtube-automate/specs.md
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
# Group=mediaserver and UMask=0002 are load-bearing: Jellyfin reaches the media
# tree only through the mediaserver group, and anything this process writes must
# stay group-readable. See specs.md §2.
User=susan
Group=mediaserver
UMask=0002
WorkingDirectory=/opt/youtube-automate
ExecStart=/var/lib/youtube-automate/venv/bin/youtube-automate serve --host 127.0.0.1 --port 8085
Restart=always
RestartSec=5
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=full
ProtectControlGroups=true
ProtectKernelTunables=true
RestrictSUIDSGID=true
ReadWritePaths=/var/lib/youtube-automate /disks/Plex/YouTube
[Install]
WantedBy=multi-user.target
pyproject.toml
+32
View File
@@ -0,0 +1,32 @@
[project]
name = "youtube-automate"
version = "1.0.0"
description = "A DVR for YouTube subscriptions, laid out for Jellyfin"
requires-python = ">=3.11"
# Pins that matter are documented in specs.md §3:
# - the [default] extra is what ships yt-dlp-ejs (the JS challenge solver)
# - curl-cffi must stay below 0.16 or yt-dlp rejects it as unsupported
# - the bgutil plugin must match the container tag
dependencies = [
"yt-dlp[default]",
"bgutil-ytdlp-pot-provider==1.3.1",
"curl-cffi<0.16",
]
[project.optional-dependencies]
dev = ["pytest>=8"]
[project.scripts]
youtube-automate = "youtube_automate.cli:main"
[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.build_meta"
[tool.setuptools]
packages = ["youtube_automate", "youtube_automate.web"]
[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-q"
specs.handover-original.md
+656
View File
@@ -0,0 +1,656 @@
# `ytsubs` — implementation handover
**Target machine:** `susan`
**Audience:** a Claude Code agent implementing this from scratch
**Status:** design agreed, not started
---
## 1. What we're building
A self-hosted pipeline that keeps a rolling 14-day window of recent YouTube uploads from a
handful of subscribed channels, laid out on disk so Jellyfin presents each channel as a TV
show and each video as an episode. A small web UI lets a non-operator (the requester's
brother) subscribe, unsubscribe, and tweak settings.
The mental model is **a DVR for YouTube subscriptions**, not an archiver. Videos are
disposable. If a video ages out or a channel is unsubscribed, it goes away permanently.
### Explicitly in scope
- Poll subscribed channels for new uploads
- Download at ≤720p, h264/AAC, into a Jellyfin-friendly layout
- Generate Kodi-style `.nfo` metadata + artwork (no Jellyfin plugin)
- Delete videos older than the retention window
- Web admin: add/remove channels, edit settings
- Cron-driven, idempotent, monitored via Healthchecks
### Explicitly out of scope — do not build these
- **Cookie handling of any kind.** No browser extension, no cookie-receiving API, no
`cookies.txt`. See §3.
- Members-only, age-restricted, or private content (follows from the above)
- Migration from the existing TubeArchivist install
- Multi-user support, authentication beyond a single shared credential
- Transcoding of any kind (see §6 — this matters on this hardware)
---
## 2. Environment facts you need
- **susan**: Linux rack server, dual Xeon X58, 12 threads, **no AVX**. Old and slow.
Software video transcoding is effectively off the table.
- Already running on susan: Jellyfin, Docker, nginx (Let's Encrypt), Netdata,
Healthchecks (`hc.jihakuz.xyz`), `runitor` for cron wrapping.
- susan is on a **residential IP**. This matters — YouTube flags datacenter IPs far more
aggressively. Do not move any part of this to `victoria` (the Linode VPS).
- Tailnet: Headscale at `headscale.jihakuz.xyz`. The admin UI should bind to the tailnet
interface rather than being published publicly (see §9).
- `/disks/Plex/YouTube` is **already in use by TubeArchivist**. Do not touch it. Use a
fresh root (§5).
- Existing conventions to match: state under `/var/lib/<service>/`, deployed executables
under `/usr/local/bin/`, bare git repos under `/disks/git-repos/`.
### Proposed paths
| Purpose | Path |
|---|---|
| Source (bare repo) | `/disks/git-repos/youtube-automate.git` |
| Checkout | `/opt/youtube-automate` |
| Entry point | `/usr/local/bin/youtube-automate` |
| State DB | `/var/lib/youtube-automate/subs.db` |
| Lock file | `/var/lib/youtube-automate/run.lock` |
| Media root | `/disks/Plex/YouTube/` |
| Scratch/work dir | `/disks/Plex/YouTube/.ingest/` |
The work dir **must** be on the same filesystem as the media root so finished downloads
move into place with an atomic `rename()` rather than a copy. Jellyfin should be
configured to ignore `.ingest` (it's dot-prefixed, which Jellyfin skips by default —
verify).
---
## 3. YouTube access: PO tokens, not cookies
This was the main open question in design and the answer is that it collapses.
yt-dlp's current recommended setup for reliable downloads is the **mweb client plus a PO
Token Provider plugin** supplying tokens for GVS (Google Video Server) requests. No
account, no cookies, no browser. Cookies only unlock age-restricted / members-only /
private content, which is out of scope, and carry a real risk of the Google account being
banned.
### Setup
1. Run the provider as a Docker sidecar:
!! tom: needs a way of starting up when system reboots
```
docker run --name bgutil-provider -d --restart unless-stopped --init \
-p 127.0.0.1:4416:4416 brainicism/bgutil-ytdlp-pot-provider
```
2. Install the matching plugin into the same Python environment as yt-dlp:
!! tom: do this in uv venv
```
pip install bgutil-ytdlp-pot-provider
```
The plugin and server versions should match.
3. Verify before writing any other code:
```
yt-dlp -v 'https://www.youtube.com/watch?v=<some_id>' -F
```
You must see a line like
`[debug] [youtube] [pot] PO Token Providers: bgutil:http-1.x.x (external), ...`
If you don't, stop and fix this first — everything downstream depends on it.
4. `ytsubs` should health-check the provider before each download batch (a plain HTTP
GET against `http://127.0.0.1:4416/ping`) and fail loudly if it's down, rather than
silently accumulating 403s.
### Caveats to be aware of
- The bgutil README carries a warning that PO tokens no longer bypass the "Sign in to
confirm you're not a bot" interstitial in most cases. That is a **different failure
mode** (IP reputation) from the 403s on format URLs that this solves. Residential IP +
low request volume + sleep intervals should keep us clear of it. If it does start
firing, that's the point at which cookies get reconsidered — not before.
- **yt-dlp's YouTube handling churns every few weeks.** Client recommendations,
extractor-arg names, and which clients need tokens all move. Before implementing, read
the current `PO Token Guide` and `Extractors` pages on the yt-dlp wiki and adjust the
flags in §6 accordingly. Do not treat the flags in this document as authoritative —
treat them as the shape of the answer.
- Keep yt-dlp itself updated (weekly `pip install -U yt-dlp` under the same runitor
wrapper). A stale yt-dlp is the single most likely cause of "everything broke".
---
## 4. Discovery: RSS feeds
YouTube publishes an unauthenticated Atom feed per channel. Use the **undocumented
`UULF` playlist variant**, which returns long-form videos only — no Shorts, no
livestreams:
```
https://www.youtube.com/feeds/videos.xml?playlist_id=UULF<channel_id without the UC prefix>
```
So `UCabc123...` → `https://www.youtube.com/feeds/videos.xml?playlist_id=UULFabc123...`
This does most of the Shorts/livestream filtering for free, at the cheapest possible
point in the pipeline.
Related prefixes, for reference: `UU` all uploads, `UUSH` shorts, `UULV` livestreams,
`UUMF`/`UUMO` members-only.
### Robustness
These prefixes are undocumented and there have been reports through 2026 of intermittent
failures and missing entries in YouTube's native feeds. So:
- If the `UULF` feed 404s or returns zero entries, **fall back** to
`?channel_id=UC...` and record `discovery_source = 'uc_feed'` on the resulting rows.
- Rows discovered via the fallback path get the duration/live filter applied at download
time (§6). Rows from `UULF` don't need it.
- Track consecutive poll failures per channel and surface the count in the admin UI. Two
channels quietly failing for a month is the realistic failure mode here.
### Parsing
`xml.etree.ElementTree`, no dependencies. Namespaces:
```python
NS = {
"atom": "http://www.w3.org/2005/Atom",
"yt": "http://www.youtube.com/xml/schemas/2015",
"media": "http://search.yahoo.com/mrss/",
}
```
Per `atom:entry`: `yt:videoId`, `atom:title`, `atom:published`,
`media:group/media:description`. The feed does **not** carry duration.
### Backfill on subscribe
New subscriptions pull the last **7 days**. The RSS feed only returns ~15 items, which
can be under 7 days for a prolific channel. For the initial backfill only, use:
```
yt-dlp --flat-playlist --playlist-end 50 -J \
'https://www.youtube.com/playlist?list=UULF<...>'
```
and date-filter the entries client-side. Steady-state polling uses the RSS feed.
### Getting the subscription list in
There is no need for a Google account or a subscriptions RSS feed. Either the brother
pastes channel URLs into the admin page one at a time, or he does a one-off Google
Takeout export (`subscriptions.csv`) and we bulk-insert. With single-digit channels,
manual entry is fine — do not build a CSV importer unless asked.
---
## 5. On-disk layout
Media root: `/disks/Plex/YouTubeSubs/`
```
/disks/Plex/YouTubeSubs/
├── .work/ # scratch, ignored by Jellyfin
└── Some Channel/
├── tvshow.nfo
├── poster.jpg
├── fanart.jpg
└── Season 2026/
├── Some Channel - S2026E08110 - Video Title [dQw4w9WgXcQ].mp4
├── Some Channel - S2026E08110 - Video Title [dQw4w9WgXcQ].nfo
├── Some Channel - S2026E08110 - Video Title [dQw4w9WgXcQ]-thumb.jpg
├── Some Channel - S2026E08110 - Video Title [dQw4w9WgXcQ].en.srt
└── Some Channel - S2026E08110 - Video Title [dQw4w9WgXcQ].info.json
```
### Season / episode numbering
- **Season** = upload year (`2026`)
- **Episode** = `MMDD * 10 + ordinal_within_day`
So the first video uploaded on 11 August is `8110`, the second that day is `8111`, and
the first on 12 August is `8120`. This scheme sorts correctly across the whole year
(unlike appending a suffix to a bare `MMDD`) and tolerates up to 10 uploads per channel
per day. Clamp the ordinal at 9 and log a warning if exceeded.
The ordinal must be computed against **what's already in the DB for that channel+date**,
not against the current batch, so it's stable across runs.
### Filename sanitisation
Strip `/ \ : * ? " < > |`, collapse runs of whitespace, strip leading/trailing dots and
spaces, truncate the title component to 120 chars on a word boundary. The `[videoid]`
suffix guarantees uniqueness regardless.
Store the channel directory name in the DB (`channel.dir_name`) at subscribe time and
never recompute it — channels rename themselves and we don't want orphaned directories.
---
## 6. Download
### Format selection — read this bit carefully
susan has no AVX and cannot realistically transcode. YouTube serves 720p as VP9 or AV1 by
default. If a client can't direct-play those, Jellyfin will try to transcode and it will
be miserable. So we force h264 + AAC in mp4 at download time:
```
-f "bv*[height<=720]+ba/b[height<=720]"
-S "vcodec:h264,acodec:aac,res:720"
--merge-output-format mp4
```
If a video genuinely has no h264 rendition at ≤720p, `-S` will fall back to VP9 rather
than fail. Log those cases; they should be rare.
Do **not** use `--sponsorblock-remove` — it requires cutting and re-encoding. Use
`--sponsorblock-mark all` with `--embed-chapters`, which writes chapter markers only and
is free.
### Full invocation
```
yt-dlp \
--extractor-args "youtube:player_client=default,mweb" \
--extractor-args "youtubepot-bgutilhttp:base_url=http://127.0.0.1:4416" \
-f "bv*[height<=720]+ba/b[height<=720]" \
-S "vcodec:h264,acodec:aac,res:720" \
--merge-output-format mp4 \
--no-playlist \
--write-info-json \
--write-thumbnail --convert-thumbnails jpg \
--write-auto-subs --sub-langs "en.*" --convert-subs srt \
--sponsorblock-mark all --embed-chapters \
--retries 3 --fragment-retries 10 \
--sleep-requests 2 --sleep-interval 5 --max-sleep-interval 15 \
-P "/disks/Plex/YouTubeSubs/.work" \
-o "%(id)s.%(ext)s" \
"https://www.youtube.com/watch?v=<VIDEO_ID>"
```
Verify the `youtubepot-bgutilhttp:base_url` arg name against the current plugin README —
this is the kind of thing that gets renamed.
### Match filter (fallback-discovered videos only)
For rows with `discovery_source = 'uc_feed'`, append:
```
--match-filter "duration>?{min_duration_seconds} & live_status!=is_live & live_status!=is_upcoming & !was_live"
```
The `>?` form allows videos with unknown duration through rather than rejecting them.
`min_duration_seconds` defaults to 120 (Shorts can now run to 3 minutes, but so can
legitimate short videos — this is the tradeoff, and it's why `UULF` is the primary path).
A video rejected by the match filter should be recorded as `skipped_short` /
`skipped_live` and **never retried**.
### Concurrency and politeness
One download at a time. Single-digit channels over a 14-day window is a small workload;
there is no reason to be aggressive and every reason not to be.
### After a successful download
1. Read the `.info.json` from `.work/`
2. Compute season/episode/filename
3. Generate the episode `.nfo` (§7)
4. `os.rename()` all artefacts into the season directory
5. Update the DB row: `state='downloaded'`, `rel_path`, `downloaded_at`
6. After the whole batch, trigger a Jellyfin library refresh:
`POST {jellyfin_url}/Library/Refresh` with header `X-Emby-Token: {api_key}`
If any step 1–4 fails, clean up `.work/` for that video ID and mark `failed` with
`attempts += 1`. Give up after 5 attempts and surface it in the UI.
---
## 7. Metadata — NFO files, no Jellyfin plugin
Jellyfin reads Kodi-style NFO sidecars natively. Configure the library as **Shows**,
disable all internet metadata providers, enable *Prefer local metadata* and *Save
artwork/metadata into media folders*. Do not write or install a metadata provider plugin.
### `tvshow.nfo` (per channel, written at subscribe time, refreshed on title change)
```xml
<?xml version="1.0" encoding="utf-8"?>
<tvshow>
<title>Some Channel</title>
<plot>Channel description from yt-dlp.</plot>
<studio>YouTube</studio>
<uniqueid type="youtube" default="true">UCabc123...</uniqueid>
</tvshow>
```
### Episode `.nfo` (one per video, filename matches the media file)
```xml
<?xml version="1.0" encoding="utf-8"?>
<episodedetails>
<title>Video Title</title>
<showtitle>Some Channel</showtitle>
<season>2026</season>
<episode>8110</episode>
<plot>Video description.</plot>
<aired>2026-08-11</aired>
<runtime>12</runtime>
<studio>YouTube</studio>
<uniqueid type="youtube" default="true">dQw4w9WgXcQ</uniqueid>
</episodedetails>
```
`runtime` is in **minutes**. Escape all text content properly — video descriptions
contain everything.
### Artwork
At subscribe time, `yt-dlp --flat-playlist --playlist-items 0 -J <channel_url>` returns a
`thumbnails` array containing entries with `id` values like `avatar_uncropped` and
`banner_uncropped`. Download the avatar to `poster.jpg` and the banner to `fanart.jpg`.
Treat both as best-effort — if they're missing, carry on without them.
Per-episode thumbnails come from `--write-thumbnail`; rename to `<basename>-thumb.jpg`.
---
## 8. Data model
SQLite at `/var/lib/ytsubs/subs.db`. **Enable WAL mode** — the cron job and the web
server both write.
```sql
PRAGMA journal_mode = WAL;
PRAGMA foreign_keys = ON;
CREATE TABLE channel (
id INTEGER PRIMARY KEY,
channel_id TEXT NOT NULL UNIQUE, -- UC...
handle TEXT, -- @handle, informational
title TEXT NOT NULL,
description TEXT,
dir_name TEXT NOT NULL UNIQUE, -- sanitised, immutable after creation
added_at TEXT NOT NULL,
backfilled INTEGER NOT NULL DEFAULT 0,
last_polled_at TEXT,
last_poll_ok INTEGER,
consecutive_poll_failures INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE video (
id INTEGER PRIMARY KEY,
video_id TEXT NOT NULL UNIQUE,
channel_pk INTEGER NOT NULL REFERENCES channel(id) ON DELETE CASCADE,
title TEXT,
upload_date TEXT, -- YYYY-MM-DD
duration INTEGER,
season INTEGER,
episode INTEGER,
state TEXT NOT NULL,
discovery_source TEXT NOT NULL, -- 'uulf_feed' | 'uc_feed' | 'backfill'
rel_path TEXT, -- relative to media root, NULL unless downloaded
size_bytes INTEGER,
attempts INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
discovered_at TEXT NOT NULL,
downloaded_at TEXT,
deleted_at TEXT
);
CREATE INDEX idx_video_state ON video(state);
CREATE INDEX idx_video_upload_date ON video(upload_date);
CREATE TABLE setting (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
```
### `video.state` values
| State | Meaning | Retried? |
|---|---|---|
| `pending` | discovered, queued | — |
| `downloading` | claimed by a worker | recovered on startup → `pending` |
| `downloaded` | on disk, `rel_path` set | — |
| `deleted` | aged out; tombstone prevents re-download | never |
| `skipped_short` | rejected by match filter | never |
| `skipped_live` | rejected by match filter | never |
| `skipped_old` | discovered but already outside the window | never |
| `failed` | download error | yes, up to 5 attempts |
The tombstone behaviour is the important part: a `deleted` row must never be
re-discovered and re-downloaded. This is why we keep our own state table rather than
using yt-dlp's `--download-archive`.
### Settings (all editable from the admin page)
Store as strings; provide typed accessors with defaults so a missing key never crashes.
| Key | Default | Notes |
|---|---|---|
| `retention_days` | `14` | delete videos with `upload_date` older than this |
| `backfill_days` | `7` | window pulled when a channel is added |
| `max_height` | `720` | |
| `min_duration_seconds` | `120` | fallback-path Shorts filter |
| `sponsorblock_mark` | `true` | |
| `write_subs` | `true` | |
| `sub_langs` | `en.*` | |
| `respect_jellyfin_watch_state` | `true` | §10 |
| `jellyfin_url` | `http://127.0.0.1:8096` | |
| `jellyfin_api_key` | *(empty)* | masked in the UI |
| `jellyfin_user_id` | *(empty)* | |
| `pot_provider_url` | `http://127.0.0.1:4416` | |
| `max_attempts` | `5` | |
| `disk_cap_gb` | `0` | `0` = disabled; if set, evict oldest downloaded videos |
Validate on save (integers parse, URLs well-formed) and re-render the form with an inline
error rather than 500-ing.
---
## 9. Entry point and scheduling
Single executable `ytsubs` with subcommands:
| Command | Purpose |
|---|---|
| `ytsubs run` | poll → download → reap. This is what cron calls. |
| `ytsubs poll [--channel ID]` | discovery only |
| `ytsubs download` | drain the pending queue |
| `ytsubs reap` | retention pass |
| `ytsubs serve` | admin HTTP server (systemd unit) |
| `ytsubs subscribe <url>` / `unsubscribe <id>` | CLI equivalents, useful for debugging |
| `ytsubs doctor` | check yt-dlp version, POT provider reachable, DB writable, media root writable, Jellyfin reachable |
`run` takes a **non-blocking `flock`** on `/var/lib/ytsubs/run.lock` and exits 0 silently
if already held. On startup, reset any `downloading` rows to `pending` (crash recovery).
### Cron
```
17 * * * * runitor -uuid <hc-uuid> -- /usr/local/bin/ytsubs run
```
Hourly is ample for single-digit channels. Register the check in Healthchecks with a
generous grace period.
### systemd unit for the admin server
Standard `simple` unit, `Restart=always`, running as the same unprivileged user that owns
the DB and media root. When the user subscribes to a channel via the web UI, the handler
should spawn `ytsubs run --channel <id>` detached so the backfill starts immediately
rather than waiting up to an hour.
### nginx
Bind the admin server to `127.0.0.1:8085`. Front it with nginx on the **tailnet
interface only** — `yt.jihakuz.xyz` resolving inside Headscale, HTTP basic auth on top.
Given the brother will be on the tailnet anyway, there's no reason to expose this
publicly, and it removes the need for a Let's Encrypt cert and any real CSRF story.
Confirm this before implementing; if it does need to be public, add proper CSRF tokens.
---
## 10. Retention and deletion
### Aging out (`ytsubs reap`)
Candidates: `state = 'downloaded'` and `upload_date < today - retention_days`.
Before deleting, if `respect_jellyfin_watch_state` is on:
1. `GET {jellyfin_url}/Items?userId={user_id}&recursive=true&includeItemTypes=Episode&fields=Path,UserData`
with header `X-Emby-Token: {api_key}` — one call per reap, build a `path → UserData` map.
2. **Skip** any candidate where `UserData.PlaybackPositionTicks > 0 and not UserData.Played`
(part-watched) or `UserData.IsFavorite` is true.
3. Apply a hard backstop: delete anyway once `upload_date` is older than
`retention_days * 2`, so a half-watched video doesn't live forever.
Nothing wrecks "set and forget" like a video disappearing at the 20-minute mark. This
check is worth the complexity.
Deletion removes the media file plus its `.nfo`, `-thumb.jpg`, `.srt`, and `.info.json`
siblings, then prunes the season directory if empty. Set `state='deleted'`,
`deleted_at`, and `rel_path=NULL`. **Keep the row** — it's the tombstone.
If the Jellyfin API is unreachable, skip the whole reap for that run and log it. Do not
delete blindly.
### Unsubscribe
Hard delete, as agreed: `shutil.rmtree()` the channel directory, then `DELETE FROM
channel` (cascades to `video`). Re-subscribing later starts from scratch and re-downloads
the last 7 days. Put a confirmation step in the UI — this is destructive and irreversible.
### Disk cap (optional, `disk_cap_gb`)
If enabled, after each reap, sum `size_bytes` over `downloaded` rows and evict oldest-by-
`upload_date` until under the cap, honouring the same watch-state protection.
---
## 11. Admin UI
Stdlib only — `http.server.ThreadingHTTPServer` + `BaseHTTPRequestHandler`. No Flask, no
FastAPI, no npm. Server-rendered HTML with a single embedded `<style>` block. The only
JavaScript should be a `confirm()` on the unsubscribe button.
### Routes
| Method | Path | Behaviour |
|---|---|---|
| `GET` | `/` | Channel list + add form + settings form |
| `POST` | `/channels` | Resolve URL → insert → spawn backfill → redirect to `/` |
| `POST` | `/channels/<id>/delete` | Confirm-guarded hard delete → redirect |
| `POST` | `/settings` | Validate + persist → redirect |
| `GET` | `/health` | JSON: yt-dlp version, POT provider up, last run time, queue depth |
All POSTs redirect (303) so refresh doesn't resubmit.
### Channel list should show, per channel
Title, `@handle`, video count on disk, disk usage, most recent upload date, last poll
time, and a warning badge if `consecutive_poll_failures > 2`.
### Channel URL resolution
Accept: `https://www.youtube.com/@handle`, `/channel/UC...`, `/c/name`, `/user/name`,
a bare `@handle`, or a bare `UC...` ID.
```
yt-dlp --flat-playlist --playlist-items 0 -J <url>
```
`--playlist-items 0` fetches channel metadata **without enumerating the uploads**, which
is the cheap way to do this. Pull `channel_id`, `channel`, `description`, `thumbnails`
from the result. If resolution fails or returns no `channel_id`, re-render the form with
an error — don't insert a half-formed row.
Reject duplicates on `channel_id` (not on the submitted URL — the same channel has many
URL forms).
---
## 12. Known gotchas
- **yt-dlp churn is the top operational risk.** Extractor args, client names, and PO
token requirements change frequently. Weekly `pip install -U yt-dlp` under runitor.
When something breaks, check the yt-dlp issue tracker before debugging our code.
- **The `UULF` prefix is undocumented** and could vanish. The `channel_id` fallback path
is not optional.
- **RSS feeds cap at ~15 entries.** Fine for hourly polling, insufficient for backfill —
hence the separate `--flat-playlist` path in §4.
- **`.work` must share a filesystem with the media root**, or every completed download
becomes a full copy.
- **Don't transcode.** If Jellyfin is transcoding these files, the format selection in §6
is wrong — fix it there, not by throwing hardware at it.
- **Descriptions are hostile input** for XML generation. Use `xml.etree`'s serialiser
rather than string-formatting the NFO by hand.
- **WAL mode is required** — two writers.
- Don't reuse `/disks/Plex/YouTube` (TubeArchivist's tree).
---
## 13. Acceptance criteria
Work through these in order; each is a real check, not a code-reading exercise.
1. `ytsubs doctor` passes on a clean install.
2. Subscribing to a real, active channel creates the directory, `tvshow.nfo`,
`poster.jpg`, and queues the last 7 days of videos.
3. At least one real video downloads end to end and lands in the right season directory
with a correct filename, NFO, thumbnail, and subtitle sidecar.
4. `ffprobe` on the downloaded file shows **h264 video and AAC audio**.
5. Jellyfin, after a library scan, shows the channel as a show, the year as a season, and
the video as an episode with the correct title, description, and air date — with all
internet metadata providers disabled.
6. The video **direct-plays** on a client with no transcoding (check the Jellyfin
dashboard's active-streams panel).
7. Re-running `ytsubs run` immediately downloads nothing and errors on nothing
(idempotency).
8. Manually backdating a video's `upload_date` past the retention window causes `reap` to
delete it and its sidecars, prune the empty directory, and leave a `deleted` tombstone.
9. Re-running `poll` after that does **not** re-download the deleted video.
10. Marking a video part-watched in Jellyfin protects it from reap (with
`respect_jellyfin_watch_state` on).
11. Unsubscribing removes the directory and all rows.
12. Killing `ytsubs run` mid-download leaves no orphan in `.work`, and the next run
recovers the `downloading` row to `pending`.
13. The admin page renders, adds, removes, and persists settings; a bad settings value
produces an inline error rather than a traceback.
14. Editing `retention_days` in the UI visibly changes reap behaviour on the next run.
---
## 14. Suggested build order
!! tom: we also wants tests through this
1. Skeleton: DB schema, migrations, settings accessors, `doctor`
2. POT provider + a hardcoded single-video download proving §3 and §6 work — **do this
before writing anything else of substance**
3. Channel resolution + `subscribe`/`unsubscribe` on the CLI
4. Discovery (`poll`), both feed paths
5. Download worker + NFO/artwork generation + move-into-place
6. Jellyfin library verification (acceptance criteria 5 and 6) — a checkpoint, not a step
7. `reap`, including the watch-state check
8. `run` orchestration, flock, crash recovery, cron + runitor + Healthchecks
9. Admin server
10. systemd unit, nginx on the tailnet, deploy scripts
Steps 2 and 6 are the two places where this design could turn out to be wrong. Hit them
early and report back rather than building on top of an unverified assumption.
specs.md
+995
View File
@@ -0,0 +1,995 @@
# `youtube-automate` — implementation spec
**Target machine:** `susan`
**Status:** design agreed and verified against the live machine on 2026-08-11. Ready to build.
**Supersedes:** `specs.handover-original.md` (the original design handover). Section 15 lists
every change and why.
---
## 1. What we're building
A self-hosted pipeline that keeps a rolling retention window of recent YouTube uploads from a
handful of subscribed channels, laid out on disk so Jellyfin presents each channel as a TV show
and each video as an episode. A small web UI lets a non-operator (the requester's brother)
subscribe, unsubscribe, and tweak settings.
The mental model is **a DVR for YouTube subscriptions**, not an archiver. Videos are disposable.
If a video ages out or a channel is unsubscribed, it goes away permanently.
### Explicitly in scope
- Poll subscribed channels for new uploads
- Download at ≤720p, h264/AAC, into a Jellyfin-friendly layout
- Generate Kodi-style `.nfo` metadata + artwork (no Jellyfin plugin)
- Delete videos older than the retention window
- Web admin over public HTTPS: add/remove channels, edit settings
- Cron-driven, idempotent, monitored via Healthchecks
- Offline unit tests
### Explicitly out of scope — do not build these
- **Cookie handling of any kind.** No browser extension, no cookie-receiving API, no
`cookies.txt`. See §3.
- Members-only, age-restricted, or private content (follows from the above)
- Transcoding of any kind (see §6 — this matters on this hardware)
- Multi-user support. One shared credential, one admin UI.
- Jellyfin watch-state protection (declined; see §10 and §15)
- CSV / Takeout importers
---
## 2. Environment facts
All of the following was verified on the live machine on 2026-08-11.
- **susan**: Debian 12, kernel 6.1.0-41. Dual **Xeon X5675** (Westmere), 2×6 cores = 12 threads,
47 GB RAM. **No AVX** — confirmed absent from `/proc/cpuinfo`. Software video transcoding is
effectively off the table.
- **Jellyfin 10.11.4**, running as a native systemd service (`User=jellyfin`, `Group=jellyfin`,
`UMask=0022`) on `0.0.0.0:8096`, enabled at boot. Not containerised.
- **Docker** running and `enabled` at boot, so `--restart unless-stopped` containers do come back
after a reboot. No extra unit needed for the POT provider.
- **Healthchecks** container on `127.0.0.1:8001`, published at `hc.jihakuz.xyz`.
- **nginx** active, Let's Encrypt via certbot, DNS at njal.la driven by
`~/.local/bin/update-dns.sh` (dynamic-DNS update keys, run every 15 min from cron).
- **runitor** v1.4.1 at `/usr/local/bin/runitor`. Cron convention: entries live in **susan's**
crontab with `HC_API_URL=https://hc.jihakuz.xyz/ping` set at the top and one UUID per job.
- **Tailscale** 1.98.2, susan is `100.64.0.2`. Headscale runs on the Linode, not here.
- susan is on a **residential IP**. This matters — YouTube flags datacenter IPs far more
aggressively. Do not move any part of this to `victoria` (the Linode VPS).
- **Toolchain**: uv 0.11.23, Python 3.11.2, ffmpeg/ffprobe 5.1.7, sqlite 3.40.1, flock 2.38.1.
- **`/usr/local/bin/yt-dlp` is version 2023.11.16** — a standalone binary from Dec 2023. It is
far too old to work against YouTube today. Nothing on the machine references it (checked
`/usr/local/bin`, `/var/lib/radio`, `~/.local/bin`, and all crontabs), so it is left alone and
we use our own venv copy instead.
- **susan has no passwordless sudo.** Anything touching `/usr/local/bin`,
`/etc/systemd/system` or `/etc/nginx` must be run by the operator via `deploy/deploy.sh`.
### TubeArchivist is gone
The original handover said `/disks/Plex/YouTube` was in use by TubeArchivist and must not be
touched. **This is no longer true.** Verified: no TubeArchivist container exists (running or
stopped), the directory is empty, and no Jellyfin library references it — the libraries are
Audiobooks, Anime, Recordings, TV Shows, Music and Films. `tube.jihakuz.xyz` still has a DNS
record in `update-dns.sh` but nothing serves it, so we reuse that hostname for the admin UI.
### Paths
| Purpose | Path |
|---|---|
| Source (bare repo) | `/disks/git-repos/youtube-automate.git` |
| Checkout | `/opt/youtube-automate` |
| Entry point | `/usr/local/bin/youtube-automate` |
| Virtualenv | `/var/lib/youtube-automate/venv` |
| Deno (JS runtime, §3) | `/var/lib/youtube-automate/venv/bin/deno` |
| State DB | `/var/lib/youtube-automate/subs.db` |
| Lock file | `/var/lib/youtube-automate/run.lock` |
| Media root | `/disks/Plex/YouTube/` |
| Scratch/work dir | `/disks/Plex/YouTube/.work/` |
`/disks` is a single 5.5 TB ext4 volume (`/dev/sdb2`) with 1.3 TB free, so the work dir and the
media root share a filesystem and finished downloads move into place with an atomic `rename()`.
Estimated steady-state usage for the three test channels is 5–10 GB. Disk is a non-issue.
### Ownership and permissions — do not skip this
The media tree convention on susan is `susan:mediaserver`, directories `0770`, files `0664`.
`/disks/Plex` is `0770` with **no setgid bit**. Jellyfin reaches the tree only via its
`mediaserver` supplementary group.
If the service runs as susan with the default `umask 0022`, new directories land `susan:susan`
and only work by the `o+rx` bit — which breaks the moment the unit is hardened. Therefore:
- systemd unit runs `User=susan`, `Group=mediaserver`, `UMask=0002`
- the media root gets the **setgid bit** so new directories inherit `mediaserver`
- cron entries that call the CLI directly go through `sg mediaserver "..."`, matching the
existing `radio` entries in susan's crontab
---
## 3. YouTube access: PO tokens, not cookies
Verified against the yt-dlp wiki on 2026-08-11: the current recommendation is still the **`mweb`
client plus a PO Token Provider plugin** supplying tokens for GVS (Google Video Server)
requests. No account, no cookies, no browser. Cookies only unlock age-restricted / members-only
/ private content, which is out of scope, and carry a real risk of the Google account being
banned.
### Setup
1. Run the provider as a Docker sidecar. Docker is enabled at boot and `unless-stopped` survives
a reboot, so no systemd unit is required:
```
docker run --name bgutil-provider -d --restart unless-stopped --init \
-p 127.0.0.1:4416:4416 brainicism/bgutil-ytdlp-pot-provider:1.3.1-deno
```
**Pin the tag.** Floating `latest` on both halves is how you get a silent version skew.
2. Build the venv. **All four of these are required** — see "JS challenge solving" below for
why the last two are not optional:
```
uv venv --python 3.11 /var/lib/youtube-automate/venv
VIRTUAL_ENV=/var/lib/youtube-automate/venv uv pip install \
'yt-dlp[default]' \
'bgutil-ytdlp-pot-provider==1.3.1' \
'curl-cffi<0.16'
```
- `yt-dlp[default]` — the `[default]` extra is what pulls in **`yt-dlp-ejs`**, the challenge
solver scripts. Plain `pip install yt-dlp` does **not** include it.
- `bgutil-ytdlp-pot-provider==1.3.1` — must match the container tag.
- `curl-cffi<0.16` — enables request impersonation. yt-dlp 2026.7.4 accepts `0.5.10` or
`0.10.x`–`0.15.x` only; the current release (0.16.0) is rejected as "unsupported", and an
unpinned install silently gets you that. Verify with `yt-dlp --list-impersonate-targets`,
which must list Chrome/Safari targets rather than only `(unavailable)` lines.
3. Install a JavaScript runtime. Deno is the recommended runtime and there is none on susan.
Install it **inside the venv's `bin/`** so it needs no root and no PATH changes beyond the
venv directory the CLI already uses:
```
curl -fsSL -o deno.zip \
https://github.com/denoland/deno/releases/latest/download/deno-x86_64-unknown-linux-gnu.zip
unzip -q deno.zip && install -m 0755 deno /var/lib/youtube-automate/venv/bin/deno
```
Deno 2.9.5 (V8 15.0) runs fine on Westmere despite the absent AVX — verified. Minimum
supported version is 2.3.0. yt-dlp locates it on `PATH`, so every yt-dlp invocation must run
with `/var/lib/youtube-automate/venv/bin` prepended to `PATH`.
### JS challenge solving is mandatory — this is new since the original handover
Without a JS runtime and `yt-dlp-ejs`, yt-dlp emits:
```
WARNING: n challenge solving failed: Some formats may be missing. Ensure you have a
supported JavaScript runtime and challenge solver script distribution installed.
```
That's the `nsig` throttling signature. Unsolved, formats go missing and downloads can be
throttled. Measured on susan: without the runtime, 22 formats and **no `mweb` formats at all**
(everything came from the android client); with Deno + `yt-dlp-ejs`, **29 formats including the
12 `mweb` ones**, and a 44 MB download completed at 7.7 MiB/s with no throttling.
Confirm the good state in `-v` output — all three lines must be present:
```
[debug] JS runtimes: deno-2.9.5
[debug] [youtube] [jsc] JS Challenge Providers: ... deno ...
[youtube] [jsc:deno] Solving JS challenges using deno
```
A benign line you can ignore: `[pot:bgutil:script-deno] Script path doesn't exist:
/home/susan/bgutil-ytdlp-pot-provider/server/...`. That is the *script* variant of the bgutil
plugin probing for a local checkout we deliberately don't have; the *http* variant is the one in
use and it reports as available.
### Verification
Before writing any other code:
```
PATH=/var/lib/youtube-automate/venv/bin:$PATH yt-dlp -v '<any watch URL>' -F
```
You must see:
```
[debug] [youtube] [pot] PO Token Providers: bgutil:http-1.3.1 (external), ...
[youtube] [pot:bgutil:http] Generating a gvs PO Token for mweb client via bgutil HTTP server
[debug] [youtube] <id>: Retrieved a gvs PO Token for mweb client
```
If you don't, stop and fix this first — everything downstream depends on it.
`doctor` and every `run` health-check the provider with a plain `GET` against
`http://127.0.0.1:4416/ping` before a download batch, and fail loudly if it's down rather than
silently accumulating 403s. The endpoint is confirmed to exist and returns
`{"server_uptime": <n>, "version": "<s>"}` — `doctor` should also compare that `version` against
the installed plugin version and warn on a mismatch.
The exact extractor-arg key `youtubepot-bgutilhttp:base_url` was confirmed against the current
plugin README.
**Status: proven on susan on 2026-08-11.** Video `cBwBMyJ_bxU` downloaded end to end and
`ffprobe` reports `h264 (Main) 1280x720 @60fps` + `aac (LC) stereo` in an mp4 container, with 3
SponsorBlock chapter marks embedded and no re-encoding. Files landed `susan:mediaserver` `0664`
via the setgid bit and `umask 0002`.
### Caveats
- The bgutil README warns that PO tokens no longer bypass the "Sign in to confirm you're not a
bot" interstitial in most cases. That is a **different failure mode** (IP reputation) from the
403s on format URLs that this solves. Residential IP + low request volume + sleep intervals
should keep us clear of it. If it starts firing, that's the point at which cookies get
reconsidered — not before.
- **yt-dlp's YouTube handling churns every few weeks.** Client recommendations, extractor-arg
names, and which clients need tokens all move. Treat the flags in §6 as the shape of the
answer, not as gospel, and re-read the wiki when something breaks.
- Keep yt-dlp updated weekly under runitor (§9). A stale yt-dlp is the single most likely cause
of "everything broke" — the 2023 binary already on this box is the cautionary example.
---
## 4. Discovery: RSS feeds
YouTube publishes an unauthenticated Atom feed per channel. Use the **undocumented `UULF`
playlist variant**, which returns long-form videos only — no Shorts, no livestreams:
```
https://www.youtube.com/feeds/videos.xml?playlist_id=UULF<channel_id without the UC prefix>
```
So `UCabc123...` → `https://www.youtube.com/feeds/videos.xml?playlist_id=UULFabc123...`
Related prefixes: `UU` all uploads, `UUSH` shorts, `UULV` livestreams, `UUMF`/`UUMO`
members-only.
### The filtering is real — verified
Tested on 2026-08-11 against the three subscription targets:
| Channel | `UULF` | `UUSH` | `UULV` | Shorts leaking into `UULF` |
|---|---|---|---|---|
| clabretro | 200, 15 entries | 200, 2 entries | 200, 1 entry | **0** |
| EthosLab | 200, 15 entries | 404 | 404 | n/a |
| DolanDarkest | 200, 15 entries | 404 | 404 | n/a |
clabretro is the useful case: it genuinely has 2 Shorts and 1 livestream, and **none of them
appear in the `UULF` feed**. The primary discovery path does what it claims.
Note also that **`UUSH`/`UULV` return HTTP 404 when a channel has none** — that means "empty",
not "broken". Only a `UULF` 404 (or an empty `UULF` result) triggers the fallback.
### Robustness
These prefixes are undocumented and there have been reports of intermittent failures and missing
entries in YouTube's native feeds. So:
- If the `UULF` feed 404s or returns zero entries, **fall back** to `?channel_id=UC...` and
record `discovery_source = 'uc_feed'` on the resulting rows.
- Rows discovered via the fallback path get the duration/live filter applied at download time
(§6). Rows from `UULF` don't need it.
- **A `skipped_short` row from the fallback path is re-queued if a later `UULF` poll lists that
video.** `UULF` is authoritative about what is and isn't a Short, so this repairs the case
where a transient feed outage would otherwise permanently drop a legitimate short-but-not-Short
video. See §6 for why this matters concretely.
- Track consecutive poll failures per channel and surface the count in the admin UI. Two channels
quietly failing for a month is the realistic failure mode here.
### Parsing
`xml.etree.ElementTree`, no dependencies. Namespaces:
```python
NS = {
"atom": "http://www.w3.org/2005/Atom",
"yt": "http://www.youtube.com/xml/schemas/2015",
"media": "http://search.yahoo.com/mrss/",
}
```
Per `atom:entry`: `yt:videoId`, `atom:title`, `atom:published`,
`media:group/media:description`. The feed does **not** carry duration.
### Backfill on subscribe
New subscriptions pull the last `backfill_days` (default 7).
**`--flat-playlist` carries no upload dates.** The original handover said to pull 50 entries with
`--flat-playlist -J` and "date-filter the entries client-side". Verified against clabretro: every
entry comes back with `timestamp: None`. There is no date to filter on. Entries *do* carry `id`,
`title`, `duration` and `live_status`.
So the backfill is RSS-first:
1. Read the `UULF` feed (falling back to `channel_id`), which is the only source that carries
dates. It returns ~15 entries — enough to cover a 7-day window for any channel uploading less
than twice a day, which is all three subscription targets.
2. Only if the feed's *oldest* entry is still inside the backfill window might there be more, in
which case extend via `--flat-playlist --playlist-end 50` and resolve those extra dates one
video at a time with `--print "%(upload_date)s"`. The playlist is reverse-chronological, so
stop at the first video outside the window.
In practice step 2 never runs for the current subscriptions. It exists so that raising
`backfill_days` doesn't silently truncate at 15 videos.
### `skipped_old` and rescan
A video discovered outside the window is `skipped_old`, which is terminal. That creates a trap:
raising a channel's `retention_days` afterwards appears to do nothing, because every one of that
channel's videos is already marked old. Observed with EthosLab — his most recent upload was 22
days before subscribe, so a 7-day backfill queued **nothing at all** for him.
`rescan` re-queues `skipped_old` rows whose `upload_date` now falls inside the channel's current
effective retention window. It is an explicit action (`poll --rescan`, or a button in the admin
UI), deliberately *not* something poll does silently — doing it on every poll would make
`backfill_days` meaningless, since it would immediately re-queue everything the initial backfill
had just decided to leave behind.
Tombstones are never touched: only `skipped_old` is eligible, never `deleted`.
Verified: setting EthosLab to 60 days and rescanning queued exactly the 3 videos inside that
window and left the other 12 alone.
### Getting the subscription list in
Manual entry via the admin page, one channel URL at a time. With single-digit channels that is
enough. No CSV importer.
---
## 5. On-disk layout
Media root: `/disks/Plex/YouTube/`
```
/disks/Plex/YouTube/
├── .work/ # scratch, ignored by Jellyfin
│ └── .ignore # belt-and-braces, see below
└── Some Channel/
├── tvshow.nfo
├── poster.jpg
├── fanart.jpg
└── Season 2026/
├── Some Channel - S2026E8110 - Video Title [dQw4w9WgXcQ].mp4
├── Some Channel - S2026E8110 - Video Title [dQw4w9WgXcQ].nfo
├── Some Channel - S2026E8110 - Video Title [dQw4w9WgXcQ]-thumb.jpg
├── Some Channel - S2026E8110 - Video Title [dQw4w9WgXcQ].en.srt
└── Some Channel - S2026E8110 - Video Title [dQw4w9WgXcQ].info.json
```
The work dir is dot-prefixed (which Jellyfin skips) **and** contains an `.ignore` file, which
Jellyfin also honours. Two independent mechanisms, because relying on undocumented
hidden-folder behaviour for the directory holding partial downloads is not worth the risk.
### Season / episode numbering
- **Season** = upload year (`2026`)
- **Episode** = `MMDD * 10 + ordinal_within_day`
So the first video uploaded on 11 August is `8110`, the second that day is `8111`, and the first
on 12 August is `8120`. This sorts correctly across the whole year — 1 Jan is `1010` and 31 Dec
is `12310` — and tolerates up to 10 uploads per channel per day. Clamp the ordinal at 9 and log
a warning if exceeded.
The ordinal must be computed against **what's already in the DB for that channel+date**, not
against the current batch, so it's stable across runs.
Write the episode number **unpadded** (`E8110`), matching the `<episode>` value in the NFO
exactly.
### Filename sanitisation
Strip `/ \ : * ? " < > |`, collapse runs of whitespace, strip leading/trailing dots and spaces,
truncate the title component to 120 chars on a word boundary. The `[videoid]` suffix guarantees
uniqueness regardless.
Store the channel directory name in the DB (`channel.dir_name`) at subscribe time and never
recompute it — channels rename themselves and we don't want orphaned directories.
---
## 6. Download
### Format selection — read this bit carefully
susan has no AVX and cannot realistically transcode. YouTube serves 720p as VP9 or AV1 by
default. If a client can't direct-play those, Jellyfin will try to transcode and it will be
miserable. So we force h264 + AAC in mp4 at download time:
```
-f "bv*[height<=720]+ba/b[height<=720]"
-S "vcodec:h264,res:720,acodec:aac"
--merge-output-format mp4
```
**The sort field order matters and the original handover had it wrong.** With
`-S "vcodec:h264,acodec:aac,res:720"` — `acodec` ranked above `res` — this selects the legacy
combined **360p** format 18 rather than 720p. `bv*` means "best video, possibly with audio", so
ranking `acodec:aac` before `res` makes a combined 360p stream (which has AAC) outrank a 720p
video-only stream (whose `acodec` is `none`). Verified empirically: the original ordering yields
`18 | 640x360`, the corrected ordering yields `298+140 | 1280x720 | avc1 | mp4a`.
Keep `vcodec` **first**, ahead of `res`. If a video has no h264 at 720p but does at 480p, this
picks h264 at 480p — which direct-plays — rather than 720p VP9, which would force a transcode.
On hardware that cannot transcode, codec fidelity beats resolution. Log any case that still
falls back to VP9/AV1; they should be rare.
Do **not** use `--sponsorblock-remove` — it requires cutting and re-encoding. Use
`--sponsorblock-mark all` with `--embed-chapters`, which writes chapter markers only and is free.
### Full invocation
```
yt-dlp \
--extractor-args "youtube:player_client=default,mweb" \
--extractor-args "youtubepot-bgutilhttp:base_url=http://127.0.0.1:4416" \
-f "bv*[height<=720]+ba/b[height<=720]" \
-S "vcodec:h264,res:720,acodec:aac" \
--merge-output-format mp4 \
--no-playlist \
--write-info-json \
--write-thumbnail --convert-thumbnails jpg \
--write-subs --write-auto-subs --sub-langs "en.*" --convert-subs srt \
--sponsorblock-mark all --embed-chapters \
--retries 3 --fragment-retries 10 \
--sleep-requests 2 --sleep-interval 5 --max-sleep-interval 15 \
-P "/disks/Plex/YouTube/.work" \
-o "%(id)s.%(ext)s" \
"https://www.youtube.com/watch?v=<VIDEO_ID>"
```
`--write-subs` is included alongside `--write-auto-subs` so channels with real uploaded
subtitles aren't silently skipped in favour of nothing.
**Subtitle de-duplication.** `--sub-langs "en.*"` matches both `en` and `en-orig`, so a typical
video yields *two* sidecars (`<id>.en.srt` and `<id>.en-orig.srt`) which Jellyfin would present
as two identical "English" tracks. Observed on the test download. In the move-into-place step,
keep `.en.srt` if present and discard `.en-orig.srt`; if only `en-orig` exists, rename it to
`.en.srt`.
### Match filter (fallback-discovered videos only)
For rows with `discovery_source = 'uc_feed'`, append:
```
--match-filter "duration>?{min_duration_seconds} & live_status!=?is_live & live_status!=?is_upcoming & !was_live"
```
The `>?` and `!=?` forms allow videos with unknown values through rather than rejecting them —
an absent `live_status` should not disqualify a video any more than an absent duration does.
`min_duration_seconds` defaults to **120**.
### Why 120 is a knowingly tight default
DolanDarkest — one of the three subscription targets — uploads daily videos of **122–167
seconds**. One of them clears the filter by two seconds. Those videos arrive via `UULF`, which
is exempt from the match filter, so in normal operation this is fine.
The risk is a transient `UULF` outage: rows then get `discovery_source='uc_feed'`, the duration
filter applies, and anything under two minutes would be dropped. This is why §4 re-queues a
`skipped_short` row when a later `UULF` poll lists the video. With that repair in place the only
residual loss case is a sub-120s video whose `UULF` feed never recovers inside the retention
window, which would also be visible in the poll-failure counter.
A per-channel `min_duration_seconds` override was considered and declined; revisit if
DolanDarkest starts posting sub-two-minute videos regularly.
### Skip and defer semantics
| Outcome | State | Retried? |
|---|---|---|
| Below `min_duration_seconds` | `skipped_short` | Only if a later `UULF` poll lists it (§4) |
| `is_live` or `was_live` | `skipped_live` | Never — livestreams are not wanted |
| `is_upcoming` | `deferred` | **Yes** — re-queued by later polls |
`is_upcoming` must not be a permanent skip. A scheduled premiere becomes an ordinary
downloadable video once it airs, so permanently rejecting it would silently lose content from
any channel that uses premieres. Give up on a `deferred` row once it falls outside the retention
window.
### Concurrency and politeness
One download at a time. Single-digit channels over a 14-day window is a small workload; there is
no reason to be aggressive and every reason not to be.
### After a successful download
1. Read the `.info.json` from `.work/`
2. Compute season/episode/filename
3. Generate the episode `.nfo` (§7)
4. `os.rename()` all artefacts into the season directory
5. Update the DB row: `state='downloaded'`, `rel_path`, `downloaded_at`, **`size_bytes`**
6. After the whole batch, trigger a Jellyfin library refresh (§7)
`size_bytes` is required — `disk_cap_gb` cannot work without it.
If any step 1–4 fails, clean up `.work/` for that video ID and mark `failed` with
`attempts += 1`. Give up after `max_attempts` (5) and surface it in the UI.
---
## 7. Metadata — NFO files, no Jellyfin plugin
Jellyfin reads Kodi-style NFO sidecars natively. The library is created as **Shows** with all
internet metadata providers disabled, *Prefer local metadata* on, and *Save artwork/metadata
into media folders* on. Do not write or install a metadata provider plugin.
The library is created programmatically via `POST /Library/VirtualFolders` using the stored admin
API key, so the setup is reproducible rather than a hand-clicked state.
### `tvshow.nfo` (per channel, written at subscribe time, refreshed on title change)
```xml
<?xml version="1.0" encoding="utf-8"?>
<tvshow>
<title>Some Channel</title>
<plot>Channel description from yt-dlp.</plot>
<studio>YouTube</studio>
<uniqueid type="youtube" default="true">UCabc123...</uniqueid>
</tvshow>
```
### Episode `.nfo` (one per video, filename matches the media file)
```xml
<?xml version="1.0" encoding="utf-8"?>
<episodedetails>
<title>Video Title</title>
<showtitle>Some Channel</showtitle>
<season>2026</season>
<episode>8110</episode>
<plot>Video description.</plot>
<aired>2026-08-11</aired>
<runtime>12</runtime>
<studio>YouTube</studio>
<uniqueid type="youtube" default="true">dQw4w9WgXcQ</uniqueid>
</episodedetails>
```
`runtime` is in **minutes**. Build these with `xml.etree`'s serialiser, never string formatting
— video descriptions are hostile input and contain everything.
### Artwork
At subscribe time, `yt-dlp --flat-playlist --playlist-items 0 -J <channel_url>` returns a
`thumbnails` array containing entries with `id` values like `avatar_uncropped` and
`banner_uncropped`. Download the avatar to `poster.jpg` and the banner to `fanart.jpg`. Treat
both as best-effort — if they're missing, carry on without them.
Per-episode thumbnails come from `--write-thumbnail`; rename to `<basename>-thumb.jpg`.
### Library refresh
`POST {jellyfin_url}/Library/Refresh` with header `X-Emby-Token: {api_key}`, triggered after:
- a download batch that produced at least one new file
- a reap that deleted at least one file
- an unsubscribe
Refreshing after deletions matters — without it Jellyfin shows ghost episodes until its own
scheduled scan.
---
## 8. Data model
SQLite at `/var/lib/youtube-automate/subs.db`. **WAL mode** — the cron job and the web server
both write.
```sql
PRAGMA journal_mode = WAL;
PRAGMA foreign_keys = ON;
CREATE TABLE channel (
id INTEGER PRIMARY KEY,
channel_id TEXT NOT NULL UNIQUE, -- UC...
handle TEXT, -- @handle, informational
title TEXT NOT NULL,
description TEXT,
dir_name TEXT NOT NULL UNIQUE, -- sanitised, immutable after creation
added_at TEXT NOT NULL,
backfilled INTEGER NOT NULL DEFAULT 0,
retention_days INTEGER, -- NULL = use global setting
last_polled_at TEXT,
last_poll_ok INTEGER,
consecutive_poll_failures INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE video (
id INTEGER PRIMARY KEY,
video_id TEXT NOT NULL UNIQUE,
channel_pk INTEGER NOT NULL REFERENCES channel(id) ON DELETE CASCADE,
title TEXT,
upload_date TEXT, -- YYYY-MM-DD
duration INTEGER,
season INTEGER,
episode INTEGER,
state TEXT NOT NULL,
discovery_source TEXT NOT NULL, -- 'uulf_feed' | 'uc_feed' | 'backfill'
rel_path TEXT, -- relative to media root, NULL unless downloaded
size_bytes INTEGER,
attempts INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
discovered_at TEXT NOT NULL,
downloaded_at TEXT,
deleted_at TEXT
);
CREATE INDEX idx_video_state ON video(state);
CREATE INDEX idx_video_upload_date ON video(upload_date);
CREATE INDEX idx_video_channel ON video(channel_pk);
CREATE TABLE setting (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
```
Schema changes are applied by an idempotent migration function keyed on `PRAGMA user_version`.
### `video.state` values
| State | Meaning | Retried? |
|---|---|---|
| `pending` | discovered, queued | — |
| `downloading` | claimed by a worker | recovered on startup → `pending` |
| `downloaded` | on disk, `rel_path` set | — |
| `deleted` | aged out; tombstone prevents re-download | **never** |
| `deferred` | premiere/upcoming, not yet available | yes, on later polls |
| `skipped_short` | rejected by match filter | only if later seen in `UULF` (§4) |
| `skipped_live` | rejected by match filter | never |
| `skipped_old` | discovered but already outside the window | never |
| `failed` | download error | yes, up to `max_attempts` |
The tombstone behaviour is the important part: a `deleted` row must never be re-discovered and
re-downloaded, and the `skipped_short` repair in §4 must never resurrect one. This is why we
keep our own state table rather than using yt-dlp's `--download-archive`.
### Settings
Store as strings; provide typed accessors with defaults so a missing key never crashes.
| Key | Default | Notes |
|---|---|---|
| `retention_days` | `14` | global default; per-channel override wins |
| `backfill_days` | `7` | window pulled when a channel is added |
| `max_height` | `720` | |
| `min_duration_seconds` | `120` | fallback-path Shorts filter |
| `sponsorblock_mark` | `true` | |
| `write_subs` | `true` | |
| `sub_langs` | `en.*` | |
| `jellyfin_url` | `http://127.0.0.1:8096` | |
| `jellyfin_api_key` | *(empty)* | masked in the UI; set via CLI |
| `pot_provider_url` | `http://127.0.0.1:4416` | |
| `max_attempts` | `5` | |
| `disk_cap_gb` | `0` | `0` = disabled; if set, evict oldest downloaded videos |
Validate on save (integers parse, URLs well-formed) and re-render the form with an inline error
rather than 500-ing.
Two further keys are **not** editable through the settings form and are never rendered:
`admin_password_hash` (see §11) and `session_secret` (generated on first run).
**No secret is committed to the repo.** The Jellyfin API key and the admin password are set with
CLI subcommands and live only in the DB.
---
## 9. Entry point and scheduling
`/usr/local/bin/youtube-automate` is a shim that execs the venv interpreter against the package.
Subcommands:
| Command | Purpose |
|---|---|
| `run [--channel ID]` | poll → download → reap. This is what cron calls. |
| `poll [--channel ID]` | discovery only |
| `download` | drain the pending queue |
| `reap` | retention pass |
| `serve` | admin HTTP server (systemd unit) |
| `subscribe <url>` / `unsubscribe <id>` | CLI equivalents, useful for debugging |
| `set-password` | prompt for and store the admin password (never on the command line) |
| `set-jellyfin-key` | store the Jellyfin API key |
| `setup-jellyfin-library` | create/verify the YouTube Shows library with the right options |
| `doctor` | yt-dlp version, POT provider reachable, DB writable, media root writable, Jellyfin reachable, permissions sane |
`run` takes a **non-blocking `flock`** on `/var/lib/youtube-automate/run.lock` and exits 0
silently if already held. On startup it resets any `downloading` rows to `pending` and clears
matching orphans out of `.work/` (crash recovery).
`run` accepts `--channel` so the web subscribe handler can spawn an immediate backfill rather
than waiting up to an hour.
### Cron (susan's crontab, alongside the existing entries)
```
17 * * * * runitor -uuid 41a4d61a-7743-43d9-9b5d-d37d536e4726 -- sg mediaserver "/usr/local/bin/youtube-automate run"
40 4 * * 1 runitor -uuid 721e4cf0-d796-48e7-a184-79d21e1ba373 -- /var/lib/youtube-automate/venv/bin/python -m uv pip install --upgrade yt-dlp
```
Hourly is ample for single-digit channels; both checks are registered in Healthchecks with
generous grace periods. The weekly updater is monitored separately so a stale yt-dlp alerts
distinctly from a failed run — §12 calls that the top operational risk.
### systemd unit for the admin server
`Type=simple`, `Restart=always`, `User=susan`, `Group=mediaserver`, `UMask=0002`. Binds
`127.0.0.1:8085` only.
### nginx
`tube.jihakuz.xyz` → `127.0.0.1:8085`, public, TLS via certbot. The DNS record and njal.la
update key already exist in `update-dns.sh`, so no DNS work is needed.
**The hostname was already claimed.** A leftover TubeArchivist server block inside
`sites-available/jihakuz.xyz` served `tube.jihakuz.xyz`, proxying to the now-dead
`127.0.0.1:8003`. nginx uses the *first* server block matching a name and that file loads before
any standalone `tube.jihakuz.xyz` file, so installing a second vhost changed nothing and every
request returned **502**. The old block also already owns the Let's Encrypt certificate for the
hostname, so the fix is to repoint it rather than duplicate it — `deploy/fix-nginx-tube.sh`
does that, and removes the redundant standalone vhost.
The forwarding headers are not cosmetic: the app throttles failed logins per client address and
reads `X-Forwarded-For`. Without it every attempt appears to come from nginx itself, so a single
attacker would lock out everyone.
Because the UI is public, the app carries real CSRF tokens and treats every input as hostile
(§11).
---
## 10. Retention and deletion
### Aging out (`reap`)
Effective retention for a video = its channel's `retention_days` if set, else the global
`retention_days`. Candidates: `state = 'downloaded'` and `upload_date < today - effective`.
Per-channel overrides exist because the subscription mix spans wildly different cadences.
EthosLab uploads roughly every two to three weeks (20 Jul, 1 Jul, 19 Jun, 1 Jun, 11 May), so at
a flat 14 days his show would sit empty in Jellyfin more often than not.
Deletion removes the media file plus its `.nfo`, `-thumb.jpg`, `.srt` and `.info.json` siblings,
then prunes the season directory if it's empty, and the channel directory if that leaves it bare
of seasons. Set `state='deleted'`, `deleted_at`, and `rel_path=NULL`. **Keep the row** — it's
the tombstone. Trigger a Jellyfin refresh if anything was deleted.
Reap is a purely local operation and does not require Jellyfin to be reachable. Watch-state
protection was considered and declined (§15), so a video can in principle vanish mid-watch;
the mitigation available is raising that channel's `retention_days`.
### Unsubscribe
Hard delete: `shutil.rmtree()` the channel directory, then `DELETE FROM channel` (cascades to
`video`). Re-subscribing later starts from scratch and re-downloads the backfill window. The UI
requires a confirmation step — this is destructive and irreversible.
### Disk cap (optional, `disk_cap_gb`)
If enabled, after each reap sum `size_bytes` over `downloaded` rows and evict oldest-by-
`upload_date` until under the cap.
---
## 11. Admin UI
Stdlib only — `http.server.ThreadingHTTPServer` + `BaseHTTPRequestHandler`. No Flask, no
FastAPI, no npm. Server-rendered HTML with a single embedded `<style>` block. The only
JavaScript is a `confirm()` on the unsubscribe button.
### Authentication
The UI is publicly reachable, so this is real auth, and it must not nag — the requirement is
"log in once per device, effectively never again".
- One shared password, stored as a **scrypt** hash (`n=2**14, r=8, p=1`, per-install salt) in
`setting.admin_password_hash`. Set only via `youtube-automate set-password`, which prompts —
never passed as an argument, never written to the repo.
- Login form posts to `/login`. On success, set a session cookie containing an HMAC-signed token
(`session_secret`, generated on first run), with attributes
`Secure; HttpOnly; SameSite=Lax; Path=/; Max-Age=31536000` — a 365-day expiry.
- Every state-changing POST carries a **CSRF token** bound to the session and verified
constant-time.
- Login throttling: after 5 consecutive failures from an IP, reject for 60 seconds. In-memory is
fine.
- All comparisons of secrets use `hmac.compare_digest`.
### Routes
| Method | Path | Behaviour |
|---|---|---|
| `GET` | `/login` | login form |
| `POST` | `/login` | authenticate, set cookie, 303 → `/` |
| `POST` | `/logout` | clear cookie, 303 → `/login` |
| `GET` | `/` | channel list + add form + settings form |
| `POST` | `/channels` | resolve URL → insert → spawn backfill → 303 → `/` |
| `POST` | `/channels/<id>/retention` | set or clear the per-channel override |
| `POST` | `/channels/<id>/delete` | confirm-guarded hard delete → 303 |
| `POST` | `/settings` | validate + persist → 303 |
| `POST` | `/channels/<id>/rescan` | re-queue `skipped_old` inside the current window → 303 |
| `GET` | `/health` | JSON: yt-dlp version, POT provider up, last run time, queue depth |
All POSTs redirect (303) so refresh doesn't resubmit. Everything except `/login` requires a
valid session — **including `/health`**, which the original spec left open. That was reasonable
when the UI was going to be tailnet-only; on a public hostname it is gratuitous fingerprinting
surface (yt-dlp version, queue depth, channel count) for no benefit, since the cron job is
monitored by Healthchecks rather than by polling this endpoint.
### Channel list should show, per channel
Title, `@handle`, video count on disk, disk usage, most recent upload date, last poll time, the
effective retention (and whether it's an override), and a warning badge if
`consecutive_poll_failures > 2`.
### Channel URL resolution
Accept: `https://www.youtube.com/@handle`, `/channel/UC...`, `/c/name`, `/user/name`, a bare
`@handle`, or a bare `UC...` ID.
```
yt-dlp --flat-playlist --playlist-items 0 -J <url>
```
`--playlist-items 0` fetches channel metadata **without enumerating the uploads**, which is the
cheap way to do this. Pull `channel_id`, `channel`, `description`, `thumbnails` from the result.
If resolution fails or returns no `channel_id`, re-render the form with an error — don't insert
a half-formed row.
Reject duplicates on `channel_id`, not on the submitted URL — the same channel has many URL
forms.
---
## 12. Known gotchas
- **yt-dlp churn is the top operational risk.** Extractor args, client names, and PO token
requirements change frequently. Weekly update under runitor. When something breaks, check the
yt-dlp issue tracker before debugging our code.
- **The `UULF` prefix is undocumented** and could vanish. The `channel_id` fallback path is not
optional. `UUSH`/`UULV` 404 means "none", not "broken".
- **RSS feeds cap at ~15 entries.** Fine for hourly polling, insufficient for backfill — hence
the separate `--flat-playlist` path in §4.
- **`.work` must share a filesystem with the media root**, or every completed download becomes a
full copy. Both are under `/disks` (`/dev/sdb2`).
- **Don't transcode.** If Jellyfin is transcoding these files, the format selection in §6 is
wrong — fix it there, not by throwing hardware at it.
- **Descriptions are hostile input** for XML generation. Use `xml.etree`'s serialiser.
- **WAL mode is required** — two writers.
- **Group and umask matter** (§2). Files that Jellyfin can't read are a silent failure.
- **No passwordless sudo** — root steps go in `deploy/deploy.sh` for the operator to run.
---
## 13. Acceptance criteria
Work through these in order; each is a real check, not a code-reading exercise.
1. `doctor` passes on a clean install.
2. Subscribing to a real, active channel creates the directory, `tvshow.nfo`, `poster.jpg`, and
queues the backfill window.
3. At least one real video downloads end to end and lands in the right season directory with a
correct filename, NFO, thumbnail, and subtitle sidecar.
4. `ffprobe` on the downloaded file shows **h264 video and AAC audio**.
5. Files and directories are readable by the `jellyfin` user (group `mediaserver`, `0664`/`0770`).
6. Jellyfin, after a library scan, shows the channel as a show, the year as a season, and the
video as an episode with the correct title, description, and air date — with all internet
metadata providers disabled.
7. The video **direct-plays** on a client with no transcoding — the Jellyfin dashboard's
active-streams panel must show neither video nor audio transcoding.
8. Re-running `run` immediately downloads nothing and errors on nothing (idempotency).
9. Manually backdating a video's `upload_date` past the retention window causes `reap` to delete
it and its sidecars, prune the empty directory, and leave a `deleted` tombstone.
10. Re-running `poll` after that does **not** re-download the deleted video.
11. A per-channel `retention_days` override visibly changes which videos that channel's reap
deletes, while other channels follow the global value.
12. Unsubscribing removes the directory and all rows.
13. Killing `run` mid-download leaves no orphan in `.work`, and the next run recovers the
`downloading` row to `pending`.
14. The admin page requires login, stays logged in across a browser restart, and rejects a POST
with a missing or wrong CSRF token.
15. The admin page renders, adds, removes, and persists settings; a bad settings value produces
an inline error rather than a traceback.
16. Editing `retention_days` in the UI visibly changes reap behaviour on the next run.
17. `pytest` passes with no network access.
---
## 14. Build order
1. Skeleton: package layout, DB schema + migrations, settings accessors, `doctor`
2. venv + POT provider + a hardcoded single-video download proving §3 and §6 work — **do this
before writing anything else of substance**
3. Channel resolution + `subscribe`/`unsubscribe` on the CLI
4. Discovery (`poll`), both feed paths, including the `skipped_short` repair
5. Download worker + NFO/artwork generation + move-into-place
6. Jellyfin library creation and verification (criteria 6 and 7) — a checkpoint, not a step
7. `reap`, including per-channel overrides
8. `run` orchestration, flock, crash recovery, cron + runitor
9. Admin server, including auth and CSRF
10. systemd unit, nginx vhost, certbot, `deploy.sh`
Steps 2 and 6 are the two places where this design could still turn out to be wrong. Hit them
early and report back rather than building on top of an unverified assumption.
### Tests
`pytest`, offline, no network and no real yt-dlp invocation. Fakes for yt-dlp and the Jellyfin
API. Coverage targets:
- filename sanitisation, including hostile titles and the 120-char truncation
- season/episode numbering and the ordinal-within-day logic, including the clamp at 9
- RSS parsing from fixture XML, both `UULF` and fallback shapes, including a 404
- the `skipped_short` → `pending` repair, and proof it never resurrects a `deleted` tombstone
- NFO generation with descriptions full of `&`, `<`, emoji and newlines
- state machine transitions, including crash recovery and `deferred`
- reap candidate selection with and without per-channel overrides
- settings validation and typed accessors with missing keys
- auth: scrypt round-trip, cookie signing, CSRF verification, throttling
---
## 15. Changes from the original handover
| Area | Change | Why |
|---|---|---|
| Naming | `ytsubs` → `youtube-automate` throughout | The original contradicted itself; the bare repo, checkout and state dir already existed under `youtube-automate` |
| Media root | `/disks/Plex/YouTubeSubs` → `/disks/Plex/YouTube`; `.ingest` → `.work` | TubeArchivist is not installed, the directory is empty and no Jellyfin library references it — verified |
| Admin UI | Tailnet-only + basic auth → **public HTTPS at `tube.jihakuz.xyz`** with app login, scrypt, 365-day signed cookie, CSRF | Requester's brother has no tailnet device; basic auth re-prompts on some mobile browsers |
| Watch state | Removed | Declined. There are five Jellyfin users, so a single `jellyfin_user_id` would protect one person's progress; per-channel retention is the mitigation instead |
| Retention | Added per-channel `retention_days` override | EthosLab's 2–3 week cadence would leave his show empty at a flat 14 days |
| Shorts filter | `min_duration_seconds` stays 120, but `skipped_short` is repaired on a later `UULF` sighting | DolanDarkest posts 122–167s videos; the repair removes the transient-outage data-loss path |
| Premieres | `is_upcoming` now `deferred` and retried, not permanently skipped | A premiere becomes downloadable once it airs |
| Subtitles | Added `--write-subs` alongside `--write-auto-subs` | Channels with real uploaded subs were being skipped |
| Match filter | `!=` → `!=?` on `live_status` | An unknown `live_status` shouldn't disqualify a video, same logic as `duration>?` |
| Download | Record `size_bytes` | `disk_cap_gb` was unimplementable without it |
| Refresh | Also refresh Jellyfin after reap and unsubscribe | Otherwise ghost episodes linger |
| Episode format | `E08110` → `E8110` | Consistency with the `<episode>` value in the NFO |
| Permissions | Added `Group=mediaserver`, `UMask=0002`, setgid on media root | Original was silent on it; Jellyfin reads the tree only via that group |
| `.work` | Added an `.ignore` file as well as the dot prefix | Not worth relying on undocumented behaviour for the partial-download directory |
| Versions | Pinned bgutil to `1.3.1` / `1.3.1-deno` | Floating `latest` on both halves invites version skew |
| yt-dlp | Install in the venv; leave `/usr/local/bin/yt-dlp` (2023.11.16) alone | Nothing on the box references it, so replacing it buys nothing and risks other jobs |
| `run` | Accepts `--channel` | §9 spawned `run --channel` but only defined the flag on `poll` |
| Monitoring | Two Healthchecks UUIDs, one per job | Matches the existing one-UUID-per-cron-entry convention on susan |
| Tests | Added §14 test plan | Requested |
### Found during build-order step 2 (the PO token proof)
These were discovered by actually running the pipeline, not by reading the doc.
| Area | Change | Why |
|---|---|---|
| Format sort | `-S "vcodec:h264,acodec:aac,res:720"` → `-S "vcodec:h264,res:720,acodec:aac"` | **The original selects 360p.** Ranking `acodec` above `res` makes the combined 360p format (which has AAC) outrank the 720p video-only stream (`acodec=none`) under `bv*`. Verified: original gives `18 \| 640x360`, corrected gives `298+140 \| 1280x720` |
| JS runtime | Added Deno into the venv `bin/`, plus `PATH` requirement | New yt-dlp requirement. Without it, `n challenge solving failed` — 22 formats and zero `mweb` formats vs 29 with it. Deno's V8 runs fine without AVX |
| yt-dlp extras | `yt-dlp` → `yt-dlp[default]` | The `[default]` extra is what ships `yt-dlp-ejs`, the challenge solver scripts. Plain `yt-dlp` does not include it |
| Impersonation | Added `curl-cffi<0.16` | Unpinned installs get 0.16.0, which yt-dlp rejects as "unsupported"; the supported range is `0.5.10` or `0.10.x`–`0.15.x`. Removes a per-download warning and helps against the bot-detection interstitial |
| Subtitles | De-duplicate `en` vs `en-orig` on move | `--sub-langs "en.*"` matches both, producing two identical English tracks in Jellyfin |
| `doctor` | Also compare `/ping` reported version against the installed plugin version | Version skew between container and plugin is silent otherwise |
### Found during the rest of the build
| Area | Change | Why |
|---|---|---|
| Backfill | `--flat-playlist` has no dates; RSS is now the primary source | Verified: every flat-playlist entry returns `timestamp: None`, so the specced client-side date filter was impossible |
| `skipped_old` | Added explicit `rescan` | Otherwise raising a channel's retention appears to do nothing. EthosLab's 7-day backfill queued zero videos |
| `/health` | Now requires a session | The UI moved from tailnet-only to public |
| Reap | Channel directory is kept when its last season is pruned | Deleting it would make an active subscription vanish from Jellyfin and reappear later, and would bin the artwork |
| Premieres | `is_upcoming` → `deferred`, retried | A premiere becomes downloadable once it airs |
### Verified end to end on 2026-08-11
`doctor` passes clean. All three channels subscribe from three different URL forms, with
`tvshow.nfo` and real JPEG artwork. Backfill queued the right windows. A real video downloaded to
`h264 Main 1280x720 / aac LC` in mp4 with SponsorBlock chapters, landed as
`EthosLab - S2026E7010 - ... [j5jhOZvm5Vo].mp4` with all four sidecars and `susan:mediaserver`
`0664`. Jellyfin resolved it from the local NFO alone — correct title, season 2026, episode 7010,
aired date, overview — and reports `SupportsDirectPlay: true`. Reap deleted all five artefacts,
pruned the season directory, kept the channel artwork and left a tombstone that survives both
poll and rescan. The lock refuses a second run instantly. 240 offline tests pass.
tests/conftest.py
+87
View File
@@ -0,0 +1,87 @@
"""Test fixtures.
Every path the application uses is redirected into a tmpdir. No test touches
the network, the real media tree, or a real yt-dlp.
"""
from __future__ import annotations
import os
import tempfile
from pathlib import Path
# config resolves its paths at import time, so the environment has to be set
# before anything from the package is imported.
_SANDBOX = Path(tempfile.mkdtemp(prefix="yta-tests-"))
os.environ.setdefault("YTA_STATE_DIR", str(_SANDBOX / "state"))
os.environ.setdefault("YTA_MEDIA_ROOT", str(_SANDBOX / "media"))
os.environ.setdefault("YTA_DB_PATH", str(_SANDBOX / "state" / "subs.db"))
os.environ.setdefault("YTA_LOCK_PATH", str(_SANDBOX / "state" / "run.lock"))
os.environ.setdefault("YTA_VENV_BIN", str(_SANDBOX / "venv" / "bin"))
import pytest # noqa: E402
from youtube_automate import config, db, util, videos # noqa: E402
from youtube_automate.settings import Settings # noqa: E402
FIXTURES = Path(__file__).parent / "fixtures"
@pytest.fixture()
def media_root(tmp_path, monkeypatch):
"""Point the media root and work dir at a per-test tmpdir."""
root = tmp_path / "media"
work = root / ".work"
work.mkdir(parents=True)
(work / ".ignore").touch()
monkeypatch.setattr(config, "MEDIA_ROOT", root)
monkeypatch.setattr(config, "WORK_DIR", work)
return root
@pytest.fixture()
def conn(tmp_path):
connection = db.connect(tmp_path / "subs.db")
yield connection
connection.close()
@pytest.fixture()
def settings(conn):
return Settings(conn)
@pytest.fixture()
def channel(conn):
"""One subscribed channel, returned as a row."""
with conn:
conn.execute(
"INSERT INTO channel (channel_id, handle, title, description, dir_name, added_at) "
"VALUES (?, ?, ?, ?, ?, ?)",
(
"UCW7jUEpYT_t0Gsf632d6_wQ",
"@clabretro",
"clabretro",
"Retro computing",
"clabretro",
util.utcnow_iso(),
),
)
return conn.execute("SELECT * FROM channel WHERE dir_name = 'clabretro'").fetchone()
def add_video(conn, channel_pk, video_id, **kwargs):
"""Insert a video row with sensible defaults."""
defaults = {
"title": f"Video {video_id}",
"upload_date": "2026-08-01",
"state": videos.PENDING,
"discovery_source": videos.SOURCE_UULF,
}
defaults.update(kwargs)
videos.insert(conn, channel_pk=channel_pk, video_id=video_id, **defaults)
return videos.get(conn, video_id)
def feed_bytes(name: str) -> bytes:
return (FIXTURES / name).read_bytes()
tests/fixtures/uulf_feed.xml
Vendored
+37
View File
@@ -0,0 +1,37 @@
<?xml version="1.0" encoding="UTF-8"?>
<feed xmlns:yt="http://www.youtube.com/xml/schemas/2015"
xmlns:media="http://search.yahoo.com/mrss/"
xmlns="http://www.w3.org/2005/Atom">
<id>yt:playlist:UULFW7jUEpYT_t0Gsf632d6_wQ</id>
<title>Uploads from clabretro</title>
<entry>
<id>yt:video:08Ajr5fP52I</id>
<yt:videoId>08Ajr5fP52I</yt:videoId>
<yt:channelId>UCW7jUEpYT_t0Gsf632d6_wQ</yt:channelId>
<title>Learning to Design 3D Prints</title>
<published>2026-08-07T15:00:11+00:00</published>
<media:group>
<media:description>Tinkercad &amp; a cheap printer. Part 1/3 &lt;of a series&gt;.</media:description>
</media:group>
</entry>
<entry>
<id>yt:video:8k8nAQq0s_s</id>
<yt:videoId>8k8nAQq0s_s</yt:videoId>
<yt:channelId>UCW7jUEpYT_t0Gsf632d6_wQ</yt:channelId>
<title>Trying to use a Nortel PBX: part two</title>
<published>2026-08-02T14:30:00+00:00</published>
<media:group>
<media:description>Telephony experiments.</media:description>
</media:group>
</entry>
<entry>
<id>yt:video:vcYYcQyecNQ</id>
<yt:videoId>vcYYcQyecNQ</yt:videoId>
<yt:channelId>UCW7jUEpYT_t0Gsf632d6_wQ</yt:channelId>
<title>IBM Director on an xSeries 346 from 2004</title>
<published>2026-06-17T12:00:00+00:00</published>
<media:group>
<media:description>Old enterprise management software.</media:description>
</media:group>
</entry>
</feed>
tests/test_auth.py
+146
View File
@@ -0,0 +1,146 @@
"""Password hashing, session cookies, CSRF tokens and login throttling."""
import time
from youtube_automate.web import auth
class TestPasswords:
def test_round_trip(self):
stored = auth.hash_password("correct horse battery staple")
assert auth.verify_password(stored, "correct horse battery staple")
def test_wrong_password_rejected(self):
stored = auth.hash_password("secret")
assert not auth.verify_password(stored, "Secret")
assert not auth.verify_password(stored, "")
def test_salt_makes_hashes_unique(self):
assert auth.hash_password("same") != auth.hash_password("same")
def test_hash_is_not_the_plaintext(self):
assert "secret" not in auth.hash_password("secret")
def test_empty_stored_hash_rejects_everything(self):
assert not auth.verify_password("", "anything")
def test_malformed_stored_hash_does_not_raise(self):
for junk in ("nonsense", "scrypt$bad", "a$b$c$d$e$f", "scrypt$x$y$z$q$r"):
assert auth.verify_password(junk, "anything") is False
def test_unicode_password(self):
stored = auth.hash_password("pässwörd🎬")
assert auth.verify_password(stored, "pässwörd🎬")
class TestSessions:
def test_issue_and_verify(self):
secret = auth.new_secret()
token = auth.issue_session(secret)
assert auth.verify_session(secret, token)
def test_a_different_secret_rejects(self):
token = auth.issue_session(auth.new_secret())
assert not auth.verify_session(auth.new_secret(), token)
def test_tampered_payload_rejected(self):
secret = auth.new_secret()
token = auth.issue_session(secret)
payload, signature = token.split(".", 1)
assert not auth.verify_session(secret, f"{payload}x.{signature}")
def test_tampered_signature_rejected(self):
secret = auth.new_secret()
payload, _ = auth.issue_session(secret).split(".", 1)
assert not auth.verify_session(secret, f"{payload}.deadbeef")
def test_garbage_rejected(self):
secret = auth.new_secret()
for junk in ("", "no-dot", "a.b.c", "...."):
assert auth.verify_session(secret, junk) is False
def test_expires_after_a_year(self):
secret = auth.new_secret()
issued = time.time() - auth.SESSION_MAX_AGE - 10
token = auth.issue_session(secret, issued_at=issued)
assert not auth.verify_session(secret, token)
def test_still_valid_just_inside_a_year(self):
secret = auth.new_secret()
issued = time.time() - auth.SESSION_MAX_AGE + 60
token = auth.issue_session(secret, issued_at=issued)
assert auth.verify_session(secret, token)
def test_a_token_from_the_future_is_rejected(self):
secret = auth.new_secret()
token = auth.issue_session(secret, issued_at=time.time() + 3600)
assert not auth.verify_session(secret, token)
class TestCookie:
def test_carries_the_hardening_flags(self):
header = auth.cookie_header("abc")
for flag in ("HttpOnly", "Secure", "SameSite=Lax", "Path=/"):
assert flag in header
assert f"Max-Age={auth.SESSION_MAX_AGE}" in header
def test_secure_can_be_omitted_for_local_http_testing(self):
assert "Secure" not in auth.cookie_header("abc", secure=False)
def test_clear_cookie_expires_immediately(self):
assert "Max-Age=0" in auth.clear_cookie_header()
class TestCsrf:
def test_token_verifies(self):
secret, session = auth.new_secret(), auth.issue_session(auth.new_secret())
token = auth.csrf_token(secret, session)
assert auth.verify_csrf(secret, session, token)
def test_token_is_bound_to_the_session(self):
secret = auth.new_secret()
one = auth.issue_session(secret, issued_at=1000)
two = auth.issue_session(secret, issued_at=2000)
assert not auth.verify_csrf(secret, two, auth.csrf_token(secret, one))
def test_empty_token_rejected(self):
secret, session = auth.new_secret(), "sess"
assert not auth.verify_csrf(secret, session, "")
def test_wrong_token_rejected(self):
secret, session = auth.new_secret(), "sess"
assert not auth.verify_csrf(secret, session, "deadbeef")
class TestThrottle:
def test_allows_up_to_the_limit(self):
throttle = auth.LoginThrottle(max_failures=3, lockout=60)
for _ in range(2):
throttle.record_failure("1.2.3.4", now=1000)
assert not throttle.locked("1.2.3.4", now=1000)
def test_locks_after_the_limit(self):
throttle = auth.LoginThrottle(max_failures=3, lockout=60)
for _ in range(3):
throttle.record_failure("1.2.3.4", now=1000)
assert throttle.locked("1.2.3.4", now=1000)
def test_lock_expires(self):
throttle = auth.LoginThrottle(max_failures=3, lockout=60)
for _ in range(3):
throttle.record_failure("1.2.3.4", now=1000)
assert not throttle.locked("1.2.3.4", now=1061)
def test_success_clears_the_counter(self):
throttle = auth.LoginThrottle(max_failures=3, lockout=60)
for _ in range(2):
throttle.record_failure("1.2.3.4", now=1000)
throttle.record_success("1.2.3.4")
assert not throttle.locked("1.2.3.4", now=1000)
def test_addresses_are_tracked_separately(self):
throttle = auth.LoginThrottle(max_failures=2, lockout=60)
for _ in range(2):
throttle.record_failure("1.1.1.1", now=1000)
assert throttle.locked("1.1.1.1", now=1000)
assert not throttle.locked("2.2.2.2", now=1000)
tests/test_channels.py
+109
View File
@@ -0,0 +1,109 @@
"""Channel URL resolution and the subscribe/unsubscribe lifecycle."""
import pytest
from youtube_automate import channels, config
class TestNormaliseUrl:
@pytest.mark.parametrize(
"raw, expected",
[
("https://www.youtube.com/@clabretro", "https://www.youtube.com/@clabretro"),
("http://youtube.com/c/name", "http://youtube.com/c/name"),
("@clabretro", "https://www.youtube.com/@clabretro"),
("clabretro", "https://www.youtube.com/@clabretro"),
(
"UCW7jUEpYT_t0Gsf632d6_wQ",
"https://www.youtube.com/channel/UCW7jUEpYT_t0Gsf632d6_wQ",
),
("www.youtube.com/@x", "https://www.youtube.com/@x"),
(" @spaced ", "https://www.youtube.com/@spaced"),
],
)
def test_accepted_forms(self, raw, expected):
assert channels.normalise_url(raw) == expected
@pytest.mark.parametrize("raw", ["", " ", "not a channel!!", "@@@"])
def test_rejected_forms(self, raw):
with pytest.raises(channels.ResolutionError):
channels.normalise_url(raw)
def test_channel_id_must_be_the_right_shape(self):
# Too short to be a real UC id, so it is treated as a handle instead.
assert channels.normalise_url("UCshort") == "https://www.youtube.com/@UCshort"
class TestUulfPlaylistId:
def test_swaps_the_uc_prefix_for_uulf(self):
assert (
channels.uulf_playlist_id("UCW7jUEpYT_t0Gsf632d6_wQ")
== "UULFW7jUEpYT_t0Gsf632d6_wQ"
)
def test_length_is_preserved(self):
channel_id = "UCW7jUEpYT_t0Gsf632d6_wQ"
assert len(channels.uulf_playlist_id(channel_id)) == len(channel_id) + 2
class TestThumbnailPicking:
def test_finds_the_requested_id(self):
thumbs = [
{"id": "avatar_uncropped", "url": "http://a/avatar.jpg"},
{"id": "banner_uncropped", "url": "http://a/banner.jpg"},
]
assert channels._pick_thumbnail(thumbs, "banner_uncropped") == "http://a/banner.jpg"
def test_returns_none_when_absent(self):
assert channels._pick_thumbnail([{"id": "other", "url": "u"}], "avatar_uncropped") is None
def test_ignores_entries_without_a_url(self):
assert channels._pick_thumbnail([{"id": "avatar_uncropped"}], "avatar_uncropped") is None
def test_empty_list(self):
assert channels._pick_thumbnail([], "avatar_uncropped") is None
class TestUniqueDirName:
def test_first_use_is_unchanged(self, conn):
assert channels._unique_dir_name(conn, "clabretro") == "clabretro"
def test_collision_gets_a_suffix(self, conn, channel):
assert channels._unique_dir_name(conn, "clabretro") == "clabretro (2)"
def test_repeated_collisions_keep_counting(self, conn, channel):
with conn:
conn.execute(
"INSERT INTO channel (channel_id, title, dir_name, added_at) "
"VALUES ('UCx', 'clabretro', 'clabretro (2)', '2026-01-01')"
)
assert channels._unique_dir_name(conn, "clabretro") == "clabretro (3)"
class TestUnsubscribe:
def test_removes_the_directory_and_the_rows(self, conn, channel, media_root, monkeypatch):
monkeypatch.setattr(config, "MEDIA_ROOT", media_root)
channel_dir = media_root / channel["dir_name"]
(channel_dir / "Season 2026").mkdir(parents=True)
(channel_dir / "tvshow.nfo").write_text("<tvshow/>")
from conftest import add_video
add_video(conn, channel["id"], "v1")
title = channels.unsubscribe(conn, channel["id"])
assert title == "clabretro"
assert not channel_dir.exists()
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 0
# ON DELETE CASCADE must take the videos with it.
assert conn.execute("SELECT COUNT(*) FROM video").fetchone()[0] == 0
def test_missing_channel_raises(self, conn):
with pytest.raises(LookupError):
channels.unsubscribe(conn, 999)
def test_tolerates_a_missing_directory(self, conn, channel, media_root, monkeypatch):
monkeypatch.setattr(config, "MEDIA_ROOT", media_root)
channels.unsubscribe(conn, channel["id"])
assert conn.execute("SELECT COUNT(*) FROM channel").fetchone()[0] == 0
tests/test_discovery.py
+259
View File
@@ -0,0 +1,259 @@
"""Discovery: feed parsing, the fallback path, and the two repair mechanisms."""
from datetime import date, timedelta
import pytest
from conftest import add_video, feed_bytes
from youtube_automate import discovery, util, videos
class TestParseEntries:
def test_parses_all_entries(self):
entries = discovery.parse_entries(feed_bytes("uulf_feed.xml"))
assert len(entries) == 3
assert entries[0]["video_id"] == "08Ajr5fP52I"
assert entries[0]["published"] == date(2026, 8, 7)
def test_unescapes_description_entities(self):
entries = discovery.parse_entries(feed_bytes("uulf_feed.xml"))
assert "&" in entries[0]["description"]
assert "<of a series>" in entries[0]["description"]
def test_keeps_characters_the_filename_would_strip(self):
entries = discovery.parse_entries(feed_bytes("uulf_feed.xml"))
assert entries[1]["title"] == "Trying to use a Nortel PBX: part two"
def test_empty_feed_yields_nothing(self):
empty = b'<?xml version="1.0"?><feed xmlns="http://www.w3.org/2005/Atom"/>'
assert discovery.parse_entries(empty) == []
def test_unparseable_feed_raises(self):
with pytest.raises(discovery.FeedUnavailable):
discovery.parse_entries(b"<not xml")
def test_entry_without_video_id_is_skipped(self):
payload = (
b'<?xml version="1.0"?><feed xmlns="http://www.w3.org/2005/Atom">'
b"<entry><title>no id</title></entry></feed>"
)
assert discovery.parse_entries(payload) == []
class TestFeedUrls:
def test_uulf_strips_the_uc_prefix(self):
url = discovery.uulf_feed_url("UCW7jUEpYT_t0Gsf632d6_wQ")
assert "playlist_id=UULFW7jUEpYT_t0Gsf632d6_wQ" in url
def test_uc_feed_uses_channel_id(self):
url = discovery.uc_feed_url("UCW7jUEpYT_t0Gsf632d6_wQ")
assert "channel_id=UCW7jUEpYT_t0Gsf632d6_wQ" in url
class TestPollChannel:
def test_queues_recent_and_skips_old(self, conn, settings, channel, monkeypatch):
recent = util.today() - timedelta(days=2)
stale = util.today() - timedelta(days=400)
monkeypatch.setattr(
discovery,
"fetch_feed",
lambda url, timeout=30.0: b"ignored",
)
monkeypatch.setattr(
discovery,
"parse_entries",
lambda payload: [
{"video_id": "new1", "title": "new", "published": recent, "description": ""},
{"video_id": "old1", "title": "old", "published": stale, "description": ""},
],
)
stats = discovery.poll_channel(conn, settings, channel)
assert stats["queued"] == 1
assert stats["old"] == 1
assert videos.get(conn, "new1")["state"] == videos.PENDING
assert videos.get(conn, "old1")["state"] == videos.SKIPPED_OLD
def test_falls_back_to_channel_feed_when_uulf_is_empty(
self, conn, settings, channel, monkeypatch
):
seen_urls = []
def fake_fetch(url, timeout=30.0):
seen_urls.append(url)
return None if "playlist_id" in url else b"feed"
monkeypatch.setattr(discovery, "fetch_feed", fake_fetch)
monkeypatch.setattr(
discovery,
"parse_entries",
lambda payload: [
{
"video_id": "fb1",
"title": "fallback",
"published": util.today(),
"description": "",
}
],
)
stats = discovery.poll_channel(conn, settings, channel)
assert stats["source"] == videos.SOURCE_UC
assert any("playlist_id" in url for url in seen_urls)
assert any("channel_id" in url for url in seen_urls)
assert videos.get(conn, "fb1")["discovery_source"] == videos.SOURCE_UC
def test_feed_failure_increments_the_counter(self, conn, settings, channel, monkeypatch):
def boom(url, timeout=30.0):
raise discovery.FeedUnavailable("HTTP 503")
monkeypatch.setattr(discovery, "fetch_feed", boom)
stats = discovery.poll_channel(conn, settings, channel)
assert "error" in stats
row = conn.execute(
"SELECT * FROM channel WHERE id = ?", (channel["id"],)
).fetchone()
assert row["consecutive_poll_failures"] == 1
assert row["last_poll_ok"] == 0
def test_success_resets_the_failure_counter(self, conn, settings, channel, monkeypatch):
with conn:
conn.execute(
"UPDATE channel SET consecutive_poll_failures = 4 WHERE id = ?",
(channel["id"],),
)
monkeypatch.setattr(discovery, "fetch_feed", lambda url, timeout=30.0: b"x")
monkeypatch.setattr(discovery, "parse_entries", lambda payload: [])
discovery.poll_channel(conn, settings, channel)
row = conn.execute(
"SELECT * FROM channel WHERE id = ?", (channel["id"],)
).fetchone()
assert row["consecutive_poll_failures"] == 0
assert row["last_poll_ok"] == 1
class TestSkippedShortRepair:
"""A fallback-discovered video wrongly rejected as a Short must come back
once the authoritative UULF feed lists it."""
def _poll_with(self, monkeypatch, entry):
monkeypatch.setattr(discovery, "fetch_feed", lambda url, timeout=30.0: b"x")
monkeypatch.setattr(discovery, "parse_entries", lambda payload: [entry])
def test_repairs_a_uc_discovered_skipped_short(
self, conn, settings, channel, monkeypatch
):
add_video(
conn,
channel["id"],
"short1",
state=videos.SKIPPED_SHORT,
discovery_source=videos.SOURCE_UC,
upload_date=util.today().isoformat(),
)
self._poll_with(
monkeypatch,
{
"video_id": "short1",
"title": "t",
"published": util.today(),
"description": "",
},
)
stats = discovery.poll_channel(conn, settings, channel)
assert stats["repaired"] == 1
row = videos.get(conn, "short1")
assert row["state"] == videos.PENDING
assert row["discovery_source"] == videos.SOURCE_UULF
def test_does_not_repair_one_discovered_via_uulf(
self, conn, settings, channel, monkeypatch
):
add_video(
conn,
channel["id"],
"short2",
state=videos.SKIPPED_SHORT,
discovery_source=videos.SOURCE_UULF,
)
self._poll_with(
monkeypatch,
{"video_id": "short2", "title": "t", "published": util.today(), "description": ""},
)
discovery.poll_channel(conn, settings, channel)
assert videos.get(conn, "short2")["state"] == videos.SKIPPED_SHORT
def test_never_resurrects_a_deleted_tombstone(
self, conn, settings, channel, monkeypatch
):
add_video(
conn,
channel["id"],
"gone1",
state=videos.DELETED,
discovery_source=videos.SOURCE_UC,
)
self._poll_with(
monkeypatch,
{"video_id": "gone1", "title": "t", "published": util.today(), "description": ""},
)
discovery.poll_channel(conn, settings, channel)
assert videos.get(conn, "gone1")["state"] == videos.DELETED
class TestRescan:
def test_requeues_skipped_old_inside_the_window(self, conn, settings, channel):
inside = (util.today() - timedelta(days=5)).isoformat()
add_video(conn, channel["id"], "v1", state=videos.SKIPPED_OLD, upload_date=inside)
assert discovery.rescan_channel(conn, settings, channel) == 1
assert videos.get(conn, "v1")["state"] == videos.PENDING
def test_leaves_videos_outside_the_window_alone(self, conn, settings, channel):
outside = (util.today() - timedelta(days=200)).isoformat()
add_video(conn, channel["id"], "v2", state=videos.SKIPPED_OLD, upload_date=outside)
assert discovery.rescan_channel(conn, settings, channel) == 0
assert videos.get(conn, "v2")["state"] == videos.SKIPPED_OLD
def test_honours_a_per_channel_override(self, conn, settings, channel):
age = (util.today() - timedelta(days=30)).isoformat()
add_video(conn, channel["id"], "v3", state=videos.SKIPPED_OLD, upload_date=age)
# Default retention is 14 days, so nothing moves.
assert discovery.rescan_channel(conn, settings, channel) == 0
with conn:
conn.execute(
"UPDATE channel SET retention_days = 60 WHERE id = ?", (channel["id"],)
)
widened = conn.execute(
"SELECT * FROM channel WHERE id = ?", (channel["id"],)
).fetchone()
assert discovery.rescan_channel(conn, settings, widened) == 1
def test_never_resurrects_a_tombstone(self, conn, settings, channel):
add_video(
conn,
channel["id"],
"v4",
state=videos.DELETED,
upload_date=util.today().isoformat(),
)
assert discovery.rescan_channel(conn, settings, channel) == 0
assert videos.get(conn, "v4")["state"] == videos.DELETED
class TestEffectiveRetention:
def test_override_wins(self, settings, channel, conn):
with conn:
conn.execute(
"UPDATE channel SET retention_days = 60 WHERE id = ?", (channel["id"],)
)
row = conn.execute("SELECT * FROM channel WHERE id = ?", (channel["id"],)).fetchone()
assert discovery.effective_retention_days(settings, row) == 60
def test_falls_back_to_the_global_default(self, settings, channel):
assert discovery.effective_retention_days(settings, channel) == 14
tests/test_download.py
+264
View File
@@ -0,0 +1,264 @@
"""Download worker: argument construction, rejection classification, moves."""
import json
from conftest import add_video
from youtube_automate import config, download, videos
class TestBuildArgs:
def _row(self, conn, channel, **kwargs):
add_video(conn, channel["id"], "vid1", **kwargs)
return conn.execute(
"SELECT v.*, c.dir_name, c.title AS channel_title, c.channel_id "
"FROM video v JOIN channel c ON c.id = v.channel_pk "
"WHERE v.video_id = 'vid1'"
).fetchone()
def test_sort_puts_vcodec_before_res_and_res_before_acodec(
self, conn, settings, channel
):
"""The original spec ordering selected 360p — see specs.md §6."""
args = download.build_args(settings, self._row(conn, channel))
sort = args[args.index("-S") + 1]
assert sort == "vcodec:h264,res:720,acodec:aac"
assert sort.index("vcodec") < sort.index("res") < sort.index("acodec")
def test_format_selector_caps_height(self, conn, settings, channel):
args = download.build_args(settings, self._row(conn, channel))
assert args[args.index("-f") + 1] == "bv*[height<=720]+ba/b[height<=720]"
def test_max_height_setting_is_honoured(self, conn, settings, channel):
settings.set("max_height", "480")
args = download.build_args(settings, self._row(conn, channel))
assert "height<=480" in args[args.index("-f") + 1]
assert "res:480" in args[args.index("-S") + 1]
def test_merges_to_mp4(self, conn, settings, channel):
args = download.build_args(settings, self._row(conn, channel))
assert args[args.index("--merge-output-format") + 1] == "mp4"
def test_no_match_filter_for_uulf_rows(self, conn, settings, channel):
args = download.build_args(
settings, self._row(conn, channel, discovery_source=videos.SOURCE_UULF)
)
assert "--match-filter" not in args
def test_match_filter_applied_to_fallback_rows(self, conn, settings, channel):
args = download.build_args(
settings, self._row(conn, channel, discovery_source=videos.SOURCE_UC)
)
assert "--match-filter" in args
expression = args[args.index("--match-filter") + 1]
assert "duration>?120" in expression
# The `?` forms must be used so unknown values pass rather than reject.
assert "live_status!=?is_live" in expression
assert "live_status!=?is_upcoming" in expression
assert "!was_live" in expression
def test_min_duration_setting_flows_into_the_filter(self, conn, settings, channel):
settings.set("min_duration_seconds", "60")
args = download.build_args(
settings, self._row(conn, channel, discovery_source=videos.SOURCE_UC)
)
assert "duration>?60" in args[args.index("--match-filter") + 1]
def test_subtitles_can_be_disabled(self, conn, settings, channel):
settings.set("write_subs", "false")
args = download.build_args(settings, self._row(conn, channel))
assert "--write-subs" not in args
def test_sponsorblock_marks_rather_than_removes(self, conn, settings, channel):
args = download.build_args(settings, self._row(conn, channel))
assert "--sponsorblock-mark" in args
assert "--sponsorblock-remove" not in args
assert "--embed-chapters" in args
def test_targets_the_right_video(self, conn, settings, channel):
args = download.build_args(settings, self._row(conn, channel))
assert args[-1] == "https://www.youtube.com/watch?v=vid1"
class TestRejectionClassification:
def test_not_a_rejection_when_no_marker(self):
assert download._classify_rejection({}, "downloading", "") is None
def test_upcoming_premiere_is_deferred_not_skipped(self):
outcome = download._classify_rejection(
{"live_status": "is_upcoming"}, "does not pass filter", ""
)
assert outcome == videos.DEFERRED
def test_live_is_skipped_permanently(self):
assert (
download._classify_rejection(
{"live_status": "is_live"}, "does not pass filter", ""
)
== videos.SKIPPED_LIVE
)
def test_past_livestream_is_skipped(self):
assert (
download._classify_rejection(
{"was_live": True}, "does not pass filter", ""
)
== videos.SKIPPED_LIVE
)
def test_otherwise_it_was_too_short(self):
assert (
download._classify_rejection({"duration": 30}, "does not pass filter", "")
== videos.SKIPPED_SHORT
)
def test_missing_info_json_still_classifies(self):
assert (
download._classify_rejection(None, "does not pass filter", "")
== videos.SKIPPED_SHORT
)
class TestSubtitleChoice:
def test_prefers_plain_en(self, media_root):
(config.WORK_DIR / "v.en.srt").write_text("a")
(config.WORK_DIR / "v.en-orig.srt").write_text("b")
assert download._choose_subtitle("v").name == "v.en.srt"
def test_promotes_en_orig_when_alone(self, media_root):
(config.WORK_DIR / "v.en-orig.srt").write_text("b")
assert download._choose_subtitle("v").name == "v.en-orig.srt"
def test_none_when_no_subtitles(self, media_root):
assert download._choose_subtitle("v") is None
class TestWorkDir:
def test_cleanup_removes_only_that_video(self, media_root):
(config.WORK_DIR / "keep.mp4").write_bytes(b"x")
(config.WORK_DIR / "drop.mp4").write_bytes(b"x")
(config.WORK_DIR / "drop.info.json").write_text("{}")
download.cleanup_work("drop")
assert (config.WORK_DIR / "keep.mp4").exists()
assert not (config.WORK_DIR / "drop.mp4").exists()
assert not (config.WORK_DIR / "drop.info.json").exists()
def test_recover_orphans_clears_everything_but_the_ignore_marker(self, media_root):
(config.WORK_DIR / "a.part").write_bytes(b"x")
(config.WORK_DIR / "b.mp4").write_bytes(b"x")
assert download.recover_orphans() == 2
assert [p.name for p in config.WORK_DIR.iterdir()] == [".ignore"]
class TestMoveIntoPlace:
def _row(self, conn, channel):
return conn.execute(
"SELECT v.*, c.dir_name, c.title AS channel_title, c.channel_id "
"FROM video v JOIN channel c ON c.id = v.channel_pk "
"WHERE v.video_id = 'vid1'"
).fetchone()
def _artefacts(self, video_id="vid1"):
(config.WORK_DIR / f"{video_id}.mp4").write_bytes(b"video-bytes")
(config.WORK_DIR / f"{video_id}.info.json").write_text("{}")
(config.WORK_DIR / f"{video_id}.jpg").write_bytes(b"jpg")
(config.WORK_DIR / f"{video_id}.en.srt").write_text("1\n")
def test_places_every_artefact_with_the_shared_stem(
self, conn, channel, media_root
):
add_video(conn, channel["id"], "vid1", upload_date="2026-08-11")
self._artefacts()
info = {
"upload_date": "20260811",
"title": "A Title: with colon",
"description": "plot",
"duration": 600,
}
rel_path, size = download._move_into_place(conn, self._row(conn, channel), info)
placed = (media_root / rel_path).parent
stem = "clabretro - S2026E8110 - A Title with colon [vid1]"
assert {p.name for p in placed.iterdir()} == {
f"{stem}.mp4",
f"{stem}.nfo",
f"{stem}.info.json",
f"{stem}-thumb.jpg",
f"{stem}.en.srt",
}
assert size == len(b"video-bytes")
def test_work_dir_is_emptied_of_that_video(self, conn, channel, media_root):
add_video(conn, channel["id"], "vid1", upload_date="2026-08-11")
self._artefacts()
download._move_into_place(
conn, self._row(conn, channel), {"upload_date": "20260811", "title": "t"}
)
assert list(config.WORK_DIR.glob("vid1.*")) == []
def test_database_row_records_the_result(self, conn, channel, media_root):
add_video(conn, channel["id"], "vid1", upload_date="2026-08-11")
self._artefacts()
download._move_into_place(
conn,
self._row(conn, channel),
{"upload_date": "20260811", "title": "t", "duration": 600},
)
row = videos.get(conn, "vid1")
assert row["state"] == videos.DOWNLOADED
assert row["season"] == 2026
assert row["episode"] == 8110
assert row["size_bytes"] == len(b"video-bytes")
assert row["rel_path"].endswith(".mp4")
def test_info_json_upload_date_beats_the_feed_date(
self, conn, channel, media_root
):
add_video(conn, channel["id"], "vid1", upload_date="2026-01-01")
self._artefacts()
download._move_into_place(
conn, self._row(conn, channel), {"upload_date": "20260811", "title": "t"}
)
row = videos.get(conn, "vid1")
assert row["upload_date"] == "2026-08-11"
assert row["episode"] == 8110
def test_missing_media_file_raises(self, conn, channel, media_root):
add_video(conn, channel["id"], "vid1", upload_date="2026-08-11")
(config.WORK_DIR / "vid1.info.json").write_text("{}")
try:
download._move_into_place(
conn, self._row(conn, channel), {"upload_date": "20260811", "title": "t"}
)
except FileNotFoundError:
pass
else: # pragma: no cover
raise AssertionError("expected FileNotFoundError")
def test_written_nfo_matches_the_episode_number(self, conn, channel, media_root):
add_video(conn, channel["id"], "vid1", upload_date="2026-08-11")
self._artefacts()
rel_path, _ = download._move_into_place(
conn,
self._row(conn, channel),
{"upload_date": "20260811", "title": "t", "description": "d", "duration": 60},
)
nfo_path = (media_root / rel_path).with_suffix(".nfo")
content = nfo_path.read_text()
assert "<episode>8110</episode>" in content
assert "S2026E8110" in nfo_path.name
class TestReadInfoJson:
def test_returns_none_when_absent(self, media_root):
assert download._read_info_json("nope") is None
def test_returns_none_on_corrupt_json(self, media_root):
(config.WORK_DIR / "v.info.json").write_text("{not json")
assert download._read_info_json("v") is None
def test_parses_valid_json(self, media_root):
(config.WORK_DIR / "v.info.json").write_text(json.dumps({"title": "x"}))
assert download._read_info_json("v")["title"] == "x"
tests/test_naming.py
+128
View File
@@ -0,0 +1,128 @@
from datetime import date
import pytest
from youtube_automate import naming
class TestSanitise:
@pytest.mark.parametrize(
"raw, expected",
[
("Hermitcraft S11#11: Expanding Business", "Hermitcraft S11#11 Expanding Business"),
("A/B", "A B"),
('Say "hello" <now>', "Say hello now"),
("path\\to\\thing", "path to thing"),
("what? really * | yes", "what really yes"),
(" .leading and trailing. ", "leading and trailing"),
("line one\nline two", "line one line two"),
("tabs\tand\r\nnewlines", "tabs and newlines"),
],
)
def test_removes_illegal_characters(self, raw, expected):
assert naming.sanitize_component(raw) == expected
def test_empty_input_is_empty(self):
assert naming.sanitize_component("") == ""
assert naming.sanitize_component(None) == ""
def test_title_that_is_only_illegal_characters_collapses_to_empty(self):
assert naming.sanitize_component("///???") == ""
def test_truncates_to_120_characters(self):
long = "word " * 60
result = naming.sanitize_component(long)
assert len(result) <= naming.MAX_TITLE_LEN
def test_truncation_prefers_a_word_boundary(self):
text = "alpha bravo charlie delta echo foxtrot golf hotel india juliet " * 3
result = naming.sanitize_component(text)
assert not result.endswith(" ")
# Should not cut mid-word when a boundary is available late enough.
assert result == result.rstrip()
assert " " in result
def test_hard_cuts_when_no_late_word_boundary_exists(self):
text = "a" + "b" * 400
result = naming.sanitize_component(text)
assert len(result) == naming.MAX_TITLE_LEN
class TestEpisodeNumbering:
@pytest.mark.parametrize(
"day, ordinal, expected",
[
(date(2026, 8, 11), 0, 8110),
(date(2026, 8, 11), 1, 8111),
(date(2026, 8, 12), 0, 8120),
(date(2026, 1, 1), 0, 1010),
(date(2026, 12, 31), 0, 12310),
(date(2026, 12, 31), 9, 12319),
],
)
def test_episode_number(self, day, ordinal, expected):
assert naming.episode_number(day, ordinal) == expected
def test_ordinal_is_clamped_at_nine(self):
assert naming.episode_number(date(2026, 8, 11), 12) == 8119
def test_negative_ordinal_clamps_to_zero(self):
assert naming.episode_number(date(2026, 8, 11), -3) == 8110
def test_numbers_sort_chronologically_across_the_year(self):
days = [date(2026, 1, 1), date(2026, 6, 15), date(2026, 8, 11), date(2026, 12, 31)]
numbers = [naming.episode_number(day, 0) for day in days]
assert numbers == sorted(numbers)
def test_episode_range_covers_ten_slots(self):
low, high = naming.episode_range(date(2026, 8, 11))
assert (low, high) == (8110, 8119)
def test_season_is_the_upload_year(self):
assert naming.season_for(date(2026, 8, 11)) == 2026
class TestParseUploadDate:
def test_accepts_ytdlp_compact_form(self):
assert naming.parse_upload_date("20260811") == date(2026, 8, 11)
def test_accepts_iso_form(self):
assert naming.parse_upload_date("2026-08-11") == date(2026, 8, 11)
def test_accepts_iso_timestamp(self):
assert naming.parse_upload_date("2026-08-11T12:00:00+00:00") == date(2026, 8, 11)
def test_passes_through_a_date(self):
assert naming.parse_upload_date(date(2026, 8, 11)) == date(2026, 8, 11)
def test_rejects_nonsense(self):
with pytest.raises(ValueError):
naming.parse_upload_date("not a date")
class TestBasename:
def test_includes_video_id_for_uniqueness(self):
stem = naming.basename("clabretro", 2026, 8110, "A Title", "dQw4w9WgXcQ")
assert stem == "clabretro - S2026E8110 - A Title [dQw4w9WgXcQ]"
def test_episode_is_unpadded_to_match_the_nfo(self):
stem = naming.basename("c", 2026, 8110, "t", "id")
assert "S2026E8110" in stem
assert "E08110" not in stem
def test_falls_back_to_video_id_when_the_title_sanitises_away(self):
stem = naming.basename("c", 2026, 8110, "///", "dQw4w9WgXcQ")
assert stem.endswith("dQw4w9WgXcQ [dQw4w9WgXcQ]")
def test_two_videos_same_title_differ_by_id(self):
one = naming.basename("c", 2026, 8110, "Same", "aaaaaaaaaaa")
two = naming.basename("c", 2026, 8111, "Same", "bbbbbbbbbbb")
assert one != two
class TestChannelDirName:
def test_sanitises_the_title(self):
assert naming.channel_dir_name("Tom / Jerry", "UCabc") == "Tom Jerry"
def test_falls_back_to_channel_id_when_title_is_unusable(self):
assert naming.channel_dir_name("???", "UCabc") == "UCabc"
tests/test_nfo.py
+94
View File
@@ -0,0 +1,94 @@
import xml.etree.ElementTree as ET
from youtube_automate import nfo
HOSTILE = (
"Ampersands & angle <brackets> and \"quotes\"\n"
"control chars: \x00\x07\x1b\n"
"emoji 🎬 and em-dash — and links https://example.com/?a=1&b=2"
)
class TestEpisodeNfo:
def build(self, **overrides):
kwargs = dict(
title="Video Title",
show_title="Some Channel",
season=2026,
episode=8110,
plot="A plot.",
aired="2026-08-11",
duration_seconds=762,
video_id="dQw4w9WgXcQ",
)
kwargs.update(overrides)
return nfo.episode_nfo(**kwargs)
def test_is_well_formed_xml(self):
root = ET.fromstring(self.build())
assert root.tag == "episodedetails"
def test_hostile_description_still_parses(self):
root = ET.fromstring(self.build(plot=HOSTILE))
plot = root.findtext("plot")
assert "&" in plot and "<brackets>" in plot
assert "🎬" in plot
def test_control_characters_are_stripped(self):
plot = ET.fromstring(self.build(plot=HOSTILE)).findtext("plot")
for bad in ("\x00", "\x07", "\x1b"):
assert bad not in plot
def test_newlines_are_preserved(self):
plot = ET.fromstring(self.build(plot="one\ntwo")).findtext("plot")
assert plot == "one\ntwo"
def test_runtime_is_rounded_minutes(self):
assert ET.fromstring(self.build(duration_seconds=762)).findtext("runtime") == "13"
def test_short_video_still_gets_at_least_one_minute(self):
assert ET.fromstring(self.build(duration_seconds=20)).findtext("runtime") == "1"
def test_runtime_omitted_when_duration_unknown(self):
assert ET.fromstring(self.build(duration_seconds=None)).find("runtime") is None
def test_unique_id_marks_youtube_as_default(self):
unique = ET.fromstring(self.build()).find("uniqueid")
assert unique.get("type") == "youtube"
assert unique.get("default") == "true"
assert unique.text == "dQw4w9WgXcQ"
def test_season_and_episode_are_present(self):
root = ET.fromstring(self.build())
assert root.findtext("season") == "2026"
assert root.findtext("episode") == "8110"
def test_title_keeps_characters_that_the_filename_strips(self):
root = ET.fromstring(self.build(title="Hermitcraft S11#11: Expanding Business"))
assert root.findtext("title") == "Hermitcraft S11#11: Expanding Business"
def test_empty_plot_does_not_break(self):
assert ET.fromstring(self.build(plot=None)).find("plot") is not None
class TestTvshowNfo:
def test_well_formed_and_carries_channel_id(self):
root = ET.fromstring(nfo.tvshow_nfo("clabretro", HOSTILE, "UCabc123"))
assert root.tag == "tvshow"
assert root.findtext("title") == "clabretro"
assert root.findtext("studio") == "YouTube"
assert root.find("uniqueid").text == "UCabc123"
class TestWrite:
def test_write_is_atomic_and_leaves_no_temp_file(self, tmp_path):
target = tmp_path / "sub" / "tvshow.nfo"
nfo.write(target, b"<tvshow/>")
assert target.read_bytes() == b"<tvshow/>"
assert list(tmp_path.rglob("*.tmp")) == []
def test_overwrites_existing(self, tmp_path):
target = tmp_path / "tvshow.nfo"
nfo.write(target, b"<a/>")
nfo.write(target, b"<b/>")
assert target.read_bytes() == b"<b/>"
tests/test_reap.py
+186
View File
@@ -0,0 +1,186 @@
"""Retention: candidate selection, artefact deletion, pruning, disk cap."""
from datetime import timedelta
from conftest import add_video
from youtube_automate import config, reap, util, videos
def _place(media_root, channel_dir, season, stem):
"""Create a downloaded video's full set of artefacts on disk."""
season_dir = media_root / channel_dir / f"Season {season}"
season_dir.mkdir(parents=True, exist_ok=True)
(season_dir / f"{stem}.mp4").write_bytes(b"video-bytes")
(season_dir / f"{stem}.nfo").write_text("<episodedetails/>")
(season_dir / f"{stem}.info.json").write_text("{}")
(season_dir / f"{stem}-thumb.jpg").write_bytes(b"jpg")
(season_dir / f"{stem}.en.srt").write_text("1\n")
return f"{channel_dir}/Season {season}/{stem}.mp4"
def _downloaded(conn, channel, media_root, video_id, upload_date, size=1024):
"""Place artefacts and record the row.
`size` is only ever read back out of the database (the disk-cap arithmetic
uses `size_bytes`), so the files on disk stay tiny no matter how large a
size the test claims.
"""
stem = f"clabretro - S2026E8110 - Title [{video_id}]"
rel = _place(media_root, channel["dir_name"], 2026, stem)
add_video(conn, channel["id"], video_id, upload_date=upload_date)
videos.mark_downloaded(
conn, video_id, rel_path=rel, size_bytes=size, season=2026, episode=8110,
upload_date=upload_date, duration=100, title="Title",
)
return rel
class TestCandidates:
def test_selects_only_videos_past_the_window(self, conn, settings, channel, media_root):
fresh = (util.today() - timedelta(days=3)).isoformat()
stale = (util.today() - timedelta(days=30)).isoformat()
_downloaded(conn, channel, media_root, "fresh", fresh)
_downloaded(conn, channel, media_root, "stale", stale)
due = [row["video_id"] for row in reap.candidates(conn, settings)]
assert due == ["stale"]
def test_per_channel_override_widens_the_window(
self, conn, settings, channel, media_root
):
age = (util.today() - timedelta(days=30)).isoformat()
_downloaded(conn, channel, media_root, "v1", age)
assert len(reap.candidates(conn, settings)) == 1
with conn:
conn.execute(
"UPDATE channel SET retention_days = 60 WHERE id = ?", (channel["id"],)
)
assert reap.candidates(conn, settings) == []
def test_per_channel_override_narrows_the_window(
self, conn, settings, channel, media_root
):
age = (util.today() - timedelta(days=5)).isoformat()
_downloaded(conn, channel, media_root, "v1", age)
assert reap.candidates(conn, settings) == []
with conn:
conn.execute(
"UPDATE channel SET retention_days = 2 WHERE id = ?", (channel["id"],)
)
assert len(reap.candidates(conn, settings)) == 1
def test_ignores_videos_that_are_not_downloaded(self, conn, settings, channel):
old = (util.today() - timedelta(days=99)).isoformat()
add_video(conn, channel["id"], "p", upload_date=old, state=videos.PENDING)
add_video(conn, channel["id"], "d", upload_date=old, state=videos.DELETED)
assert reap.candidates(conn, settings) == []
class TestDeletion:
def test_removes_every_sidecar(self, conn, settings, channel, media_root, monkeypatch):
monkeypatch.setattr(config, "MEDIA_ROOT", media_root)
old = (util.today() - timedelta(days=30)).isoformat()
rel = _downloaded(conn, channel, media_root, "v1", old)
season_dir = (media_root / rel).parent
reap.run(conn, settings)
assert not season_dir.exists()
def test_leaves_a_tombstone(self, conn, settings, channel, media_root, monkeypatch):
monkeypatch.setattr(config, "MEDIA_ROOT", media_root)
old = (util.today() - timedelta(days=30)).isoformat()
_downloaded(conn, channel, media_root, "v1", old)
reap.run(conn, settings)
row = videos.get(conn, "v1")
assert row["state"] == videos.DELETED
assert row["rel_path"] is None
def test_keeps_the_channel_directory_and_its_artwork(
self, conn, settings, channel, media_root, monkeypatch
):
"""Deleting the channel dir would make an active subscription vanish
from Jellyfin and reappear later."""
monkeypatch.setattr(config, "MEDIA_ROOT", media_root)
channel_dir = media_root / channel["dir_name"]
channel_dir.mkdir(parents=True, exist_ok=True)
(channel_dir / "tvshow.nfo").write_text("<tvshow/>")
(channel_dir / "poster.jpg").write_bytes(b"jpg")
old = (util.today() - timedelta(days=30)).isoformat()
_downloaded(conn, channel, media_root, "v1", old)
reap.run(conn, settings)
assert channel_dir.is_dir()
assert (channel_dir / "tvshow.nfo").exists()
assert (channel_dir / "poster.jpg").exists()
def test_does_not_prune_a_season_that_still_has_videos(
self, conn, settings, channel, media_root, monkeypatch
):
monkeypatch.setattr(config, "MEDIA_ROOT", media_root)
old = (util.today() - timedelta(days=30)).isoformat()
fresh = (util.today() - timedelta(days=1)).isoformat()
rel = _downloaded(conn, channel, media_root, "old1", old)
_downloaded(conn, channel, media_root, "new1", fresh)
reap.run(conn, settings)
season_dir = (media_root / rel).parent
assert season_dir.is_dir()
assert list(season_dir.glob("*new1*"))
assert not list(season_dir.glob("*old1*"))
def test_only_deletes_files_matching_the_stem(
self, conn, settings, channel, media_root, monkeypatch
):
monkeypatch.setattr(config, "MEDIA_ROOT", media_root)
old = (util.today() - timedelta(days=30)).isoformat()
rel = _downloaded(conn, channel, media_root, "v1", old)
season_dir = (media_root / rel).parent
bystander = season_dir / "unrelated file.txt"
bystander.write_text("keep me")
reap.run(conn, settings)
assert bystander.exists()
class TestDiskCap:
def test_disabled_by_default(self, conn, settings, channel, media_root):
_downloaded(conn, channel, media_root, "v1", util.today().isoformat(), size=10**6)
assert reap.disk_cap_evictions(conn, settings) == []
def test_evicts_oldest_first_until_under_the_cap(
self, conn, settings, channel, media_root
):
gigabyte = 1024**3
for index, day in enumerate((10, 5, 1)):
_downloaded(
conn,
channel,
media_root,
f"v{index}",
(util.today() - timedelta(days=day)).isoformat(),
size=gigabyte,
)
settings.set("disk_cap_gb", "2")
evicted = [row["video_id"] for row in reap.disk_cap_evictions(conn, settings)]
assert evicted == ["v0"]
def test_nothing_evicted_when_under_the_cap(self, conn, settings, channel, media_root):
_downloaded(conn, channel, media_root, "v1", util.today().isoformat(), size=1024)
settings.set("disk_cap_gb", "5")
assert reap.disk_cap_evictions(conn, settings) == []
class TestEffectiveRetention:
def test_override_wins(self, settings):
assert reap.effective_retention(settings, 60) == 60
def test_none_falls_back_to_global(self, settings):
assert reap.effective_retention(settings, None) == 14
def test_zero_falls_back_to_global(self, settings):
assert reap.effective_retention(settings, 0) == 14
tests/test_runner.py
+96
View File
@@ -0,0 +1,96 @@
"""Locking and crash recovery."""
import multiprocessing
import pytest
from conftest import add_video
from youtube_automate import config, download, runner, videos
def _hold_lock(path, started, release):
from youtube_automate import runner as runner_module
with runner_module.exclusive_lock(path):
started.set()
release.wait(timeout=30)
class TestExclusiveLock:
def test_acquires_when_free(self, tmp_path):
with runner.exclusive_lock(tmp_path / "run.lock"):
pass # no exception is the assertion
def test_can_be_reacquired_after_release(self, tmp_path):
path = tmp_path / "run.lock"
with runner.exclusive_lock(path):
pass
with runner.exclusive_lock(path):
pass
def test_second_holder_is_refused(self, tmp_path):
path = tmp_path / "run.lock"
started = multiprocessing.Event()
release = multiprocessing.Event()
holder = multiprocessing.Process(
target=_hold_lock, args=(path, started, release)
)
holder.start()
try:
assert started.wait(timeout=15), "helper never acquired the lock"
with pytest.raises(runner.AlreadyRunning):
with runner.exclusive_lock(path):
pass
finally:
release.set()
holder.join(timeout=15)
def test_creates_the_parent_directory(self, tmp_path):
path = tmp_path / "nested" / "deeper" / "run.lock"
with runner.exclusive_lock(path):
assert path.exists()
class TestRecover:
def test_requeues_downloading_rows(self, conn, channel, media_root):
add_video(conn, channel["id"], "a", state=videos.DOWNLOADING)
result = runner.recover(conn)
assert result["requeued"] == 1
assert videos.get(conn, "a")["state"] == videos.PENDING
def test_clears_work_dir_orphans(self, conn, media_root):
(config.WORK_DIR / "half.part").write_bytes(b"x")
(config.WORK_DIR / "half.mp4").write_bytes(b"x")
result = runner.recover(conn)
assert result["orphans"] == 2
assert [p.name for p in config.WORK_DIR.iterdir()] == [".ignore"]
def test_is_a_no_op_on_a_clean_state(self, conn, media_root):
assert runner.recover(conn) == {"requeued": 0, "orphans": 0}
def test_leaves_the_ignore_marker_in_place(self, conn, media_root):
download.recover_orphans()
assert (config.WORK_DIR / ".ignore").exists()
class TestSummarise:
def test_reports_each_stage(self):
text = runner.summarise(
{
"poll": {"queued": 3, "repaired": 1, "failed": 0},
"download": {videos.DOWNLOADED: 2, videos.FAILED: 1},
"reap": {"deleted": 4, "evicted": 0},
}
)
assert "discovered=3" in text
assert "repaired=1" in text
assert "downloaded=2" in text
assert "failed=1" in text
assert "reaped=4" in text
def test_surfaces_a_download_error(self):
text = runner.summarise({"download": {"error": "provider down"}})
assert "ERROR=provider down" in text
def test_handles_empty_input(self):
assert "downloaded=0" in runner.summarise({})
tests/test_settings.py
+110
View File
@@ -0,0 +1,110 @@
"""Typed settings accessors and form validation."""
import pytest
from youtube_automate import settings as settings_module
from youtube_automate.settings import DEFAULTS, Settings, validate, validate_all
class TestAccessors:
def test_missing_key_returns_the_default(self, settings):
assert settings.get_int("retention_days") == 14
assert settings.get_str("sub_langs") == "en.*"
assert settings.get_bool("write_subs") is True
def test_unknown_key_never_raises(self, settings):
assert settings.get_str("no_such_key") == ""
assert settings.get_int("no_such_key") == 0
def test_set_then_get(self, settings):
settings.set("retention_days", "30")
assert settings.get_int("retention_days") == 30
def test_set_overwrites(self, settings):
settings.set("retention_days", "30")
settings.set("retention_days", "45")
assert settings.get_int("retention_days") == 45
def test_corrupt_integer_falls_back_to_the_default(self, settings):
settings.set("retention_days", "not a number")
assert settings.get_int("retention_days") == 14
@pytest.mark.parametrize("truthy", ["true", "True", "1", "yes", "on"])
def test_bool_truthy_forms(self, settings, truthy):
settings.set("write_subs", truthy)
assert settings.get_bool("write_subs") is True
@pytest.mark.parametrize("falsy", ["false", "False", "0", "no", "off", ""])
def test_bool_falsy_forms(self, settings, falsy):
settings.set("write_subs", falsy)
assert settings.get_bool("write_subs") is False
def test_corrupt_bool_falls_back_to_the_default(self, settings):
settings.set("write_subs", "maybe")
assert settings.get_bool("write_subs") is True
def test_all_editable_covers_every_default(self, settings):
values = settings.all_editable()
assert set(values) == set(DEFAULTS)
def test_secrets_are_not_editable(self):
for key in settings_module.SECRET_KEYS:
assert key not in settings_module.EDITABLE
class TestValidation:
@pytest.mark.parametrize(
"key, value",
[
("retention_days", "14"),
("backfill_days", "0"),
("max_height", "720"),
("min_duration_seconds", "120"),
("disk_cap_gb", "0"),
("write_subs", "true"),
("sponsorblock_mark", "false"),
("jellyfin_url", "http://127.0.0.1:8096"),
("pot_provider_url", "https://example.com:4416"),
("sub_langs", "en.*"),
],
)
def test_accepts_good_values(self, key, value):
ok, _ = validate(key, value)
assert ok
@pytest.mark.parametrize(
"key, value",
[
("retention_days", "abc"),
("retention_days", "0"),
("retention_days", "-5"),
("max_height", "10"),
("min_duration_seconds", "-1"),
("write_subs", "maybe"),
("jellyfin_url", "not-a-url"),
("jellyfin_url", "ftp://host/"),
("jellyfin_url", "http://"),
("sub_langs", ""),
],
)
def test_rejects_bad_values(self, key, value):
ok, message = validate(key, value)
assert not ok
assert message
def test_validate_all_reports_each_bad_field(self):
errors = validate_all(
{"retention_days": "abc", "max_height": "720", "jellyfin_url": "nope"}
)
assert set(errors) == {"retention_days", "jellyfin_url"}
def test_validate_all_ignores_unknown_keys(self):
assert validate_all({"not_a_setting": "x"}) == {}
def test_empty_api_key_is_acceptable(self):
ok, _ = validate("jellyfin_api_key", "")
assert ok
def test_whitespace_is_tolerated(self):
ok, _ = validate("retention_days", " 21 ")
assert ok
tests/test_videos.py
+189
View File
@@ -0,0 +1,189 @@
"""The video state machine and episode assignment."""
from datetime import date
from conftest import add_video
from youtube_automate import videos
class TestNextEpisode:
def test_first_video_of_the_day_gets_the_base_number(self, conn, channel):
add_video(conn, channel["id"], "a")
season, episode = videos.next_episode(
conn, channel["id"], date(2026, 8, 11), "a"
)
assert (season, episode) == (2026, 8110)
def test_second_video_increments_the_ordinal(self, conn, channel):
add_video(conn, channel["id"], "a")
add_video(conn, channel["id"], "b")
videos.mark_downloaded(
conn, "a", rel_path="x", size_bytes=1, season=2026, episode=8110,
upload_date="2026-08-11", duration=1, title="t",
)
season, episode = videos.next_episode(
conn, channel["id"], date(2026, 8, 11), "b"
)
assert episode == 8111
def test_ordinal_is_computed_from_the_database_not_the_batch(self, conn, channel):
"""Stability across runs: a video keeps its slot even if others are
assigned in a different order later."""
for name, ep in (("a", 8110), ("b", 8111)):
add_video(conn, channel["id"], name)
videos.mark_downloaded(
conn, name, rel_path="x", size_bytes=1, season=2026, episode=ep,
upload_date="2026-08-11", duration=1, title="t",
)
add_video(conn, channel["id"], "c")
_, episode = videos.next_episode(conn, channel["id"], date(2026, 8, 11), "c")
assert episode == 8112
def test_a_different_day_starts_fresh(self, conn, channel):
add_video(conn, channel["id"], "a")
videos.mark_downloaded(
conn, "a", rel_path="x", size_bytes=1, season=2026, episode=8110,
upload_date="2026-08-11", duration=1, title="t",
)
add_video(conn, channel["id"], "b")
_, episode = videos.next_episode(conn, channel["id"], date(2026, 8, 12), "b")
assert episode == 8120
def test_another_channel_does_not_share_the_numbering(self, conn, channel):
with conn:
conn.execute(
"INSERT INTO channel (channel_id, title, dir_name, added_at) "
"VALUES ('UCother', 'Other', 'Other', '2026-01-01')"
)
other = conn.execute(
"SELECT id FROM channel WHERE dir_name = 'Other'"
).fetchone()["id"]
add_video(conn, channel["id"], "a")
videos.mark_downloaded(
conn, "a", rel_path="x", size_bytes=1, season=2026, episode=8110,
upload_date="2026-08-11", duration=1, title="t",
)
add_video(conn, other, "b")
_, episode = videos.next_episode(conn, other, date(2026, 8, 11), "b")
assert episode == 8110
def test_clamps_at_the_tenth_upload_of_a_day(self, conn, channel):
for index in range(10):
name = f"v{index}"
add_video(conn, channel["id"], name)
videos.mark_downloaded(
conn, name, rel_path="x", size_bytes=1, season=2026,
episode=8110 + index, upload_date="2026-08-11", duration=1, title="t",
)
add_video(conn, channel["id"], "overflow")
_, episode = videos.next_episode(
conn, channel["id"], date(2026, 8, 11), "overflow"
)
assert episode == 8119
def test_reassigning_the_same_video_is_stable(self, conn, channel):
add_video(conn, channel["id"], "a")
videos.mark_downloaded(
conn, "a", rel_path="x", size_bytes=1, season=2026, episode=8110,
upload_date="2026-08-11", duration=1, title="t",
)
# Its own row must be excluded, so it gets the same slot back.
_, episode = videos.next_episode(conn, channel["id"], date(2026, 8, 11), "a")
assert episode == 8110
class TestQueue:
def test_claim_returns_pending(self, conn, channel):
add_video(conn, channel["id"], "a")
assert [row["video_id"] for row in videos.claim_pending(conn, 5)] == ["a"]
def test_claim_includes_failed_with_attempts_left(self, conn, channel):
add_video(conn, channel["id"], "a", state=videos.FAILED)
with conn:
conn.execute("UPDATE video SET attempts = 2 WHERE video_id = 'a'")
assert len(videos.claim_pending(conn, 5)) == 1
def test_claim_excludes_exhausted_failures(self, conn, channel):
add_video(conn, channel["id"], "a", state=videos.FAILED)
with conn:
conn.execute("UPDATE video SET attempts = 5 WHERE video_id = 'a'")
assert videos.claim_pending(conn, 5) == []
def test_claim_excludes_terminal_states(self, conn, channel):
for index, state in enumerate(
(videos.DELETED, videos.SKIPPED_LIVE, videos.SKIPPED_OLD,
videos.SKIPPED_SHORT, videos.DOWNLOADED)
):
add_video(conn, channel["id"], f"v{index}", state=state)
assert videos.claim_pending(conn, 5) == []
def test_claim_is_oldest_first(self, conn, channel):
add_video(conn, channel["id"], "new", upload_date="2026-08-10")
add_video(conn, channel["id"], "old", upload_date="2026-08-01")
assert [row["video_id"] for row in videos.claim_pending(conn, 5)] == ["old", "new"]
def test_limit_is_respected(self, conn, channel):
for index in range(5):
add_video(conn, channel["id"], f"v{index}")
assert len(videos.claim_pending(conn, 5, limit=2)) == 2
class TestCrashRecovery:
def test_downloading_rows_return_to_pending(self, conn, channel):
add_video(conn, channel["id"], "a", state=videos.DOWNLOADING)
assert videos.recover_downloading(conn) == 1
assert videos.get(conn, "a")["state"] == videos.PENDING
def test_other_states_are_untouched(self, conn, channel):
add_video(conn, channel["id"], "a", state=videos.DOWNLOADED)
add_video(conn, channel["id"], "b", state=videos.DELETED)
assert videos.recover_downloading(conn) == 0
assert videos.get(conn, "a")["state"] == videos.DOWNLOADED
assert videos.get(conn, "b")["state"] == videos.DELETED
class TestFailures:
def test_attempts_accumulate(self, conn, channel):
add_video(conn, channel["id"], "a")
videos.record_failure(conn, "a", "boom", 5)
videos.record_failure(conn, "a", "boom", 5)
row = videos.get(conn, "a")
assert row["attempts"] == 2
assert row["state"] == videos.FAILED
assert row["last_error"] == "boom"
def test_exhaustion_is_reported(self, conn, channel):
add_video(conn, channel["id"], "a")
for _ in range(4):
videos.record_failure(conn, "a", "boom", 5)
assert videos.record_failure(conn, "a", "boom", 5) == "exhausted"
class TestTombstone:
def test_mark_deleted_clears_the_path_but_keeps_the_row(self, conn, channel):
add_video(conn, channel["id"], "a")
videos.mark_downloaded(
conn, "a", rel_path="x/y.mp4", size_bytes=10, season=2026, episode=8110,
upload_date="2026-08-11", duration=1, title="t",
)
videos.mark_deleted(conn, "a")
row = videos.get(conn, "a")
assert row is not None
assert row["state"] == videos.DELETED
assert row["rel_path"] is None
assert row["deleted_at"]
def test_insert_never_overwrites_an_existing_row(self, conn, channel):
add_video(conn, channel["id"], "a", state=videos.DELETED)
add_video(conn, channel["id"], "a", state=videos.PENDING)
assert videos.get(conn, "a")["state"] == videos.DELETED
class TestQueueDepth:
def test_counts_only_work_in_progress(self, conn, channel):
add_video(conn, channel["id"], "a", state=videos.PENDING)
add_video(conn, channel["id"], "b", state=videos.DOWNLOADING)
add_video(conn, channel["id"], "c", state=videos.FAILED)
add_video(conn, channel["id"], "d", state=videos.DOWNLOADED)
assert videos.queue_depth(conn) == 3
youtube_automate/__init__.py
+3
View File
@@ -0,0 +1,3 @@
"""youtube-automate — a DVR for YouTube subscriptions, laid out for Jellyfin."""
__version__ = "1.0.0"
youtube_automate/__main__.py
+4
View File
@@ -0,0 +1,4 @@
from .cli import main
if __name__ == "__main__":
raise SystemExit(main())
youtube_automate/channels.py
+241
View File
@@ -0,0 +1,241 @@
"""Channel resolution, subscribe and unsubscribe."""
from __future__ import annotations
import logging
import re
import shutil
import sqlite3
import subprocess
import tempfile
import urllib.request
from pathlib import Path
from . import config, naming, nfo, util, ytdlp
from .settings import Settings
log = logging.getLogger(__name__)
_CHANNEL_ID = re.compile(r"^UC[A-Za-z0-9_-]{22}$")
_HANDLE = re.compile(r"^@[A-Za-z0-9._-]+$")
# Artwork we try to pull at subscribe time. Best effort — a channel without them
# still works, it just looks plainer in Jellyfin.
_ARTWORK = (("avatar_uncropped", "poster.jpg"), ("banner_uncropped", "fanart.jpg"))
class ResolutionError(RuntimeError):
pass
def normalise_url(text: str) -> str:
"""Turn any accepted channel reference into a URL yt-dlp understands."""
text = (text or "").strip()
if not text:
raise ResolutionError("no channel given")
if text.startswith(("http://", "https://")):
return text
if _CHANNEL_ID.match(text):
return f"https://www.youtube.com/channel/{text}"
if _HANDLE.match(text):
return f"https://www.youtube.com/{text}"
if text.startswith("www.youtube.com") or text.startswith("youtube.com"):
return "https://" + text
# Bare word: assume it's a handle without the @.
if re.match(r"^[A-Za-z0-9._-]+$", text):
return f"https://www.youtube.com/@{text}"
raise ResolutionError(f"could not interpret {text!r} as a channel")
def resolve(settings: Settings, text: str) -> dict:
"""Fetch channel metadata without enumerating the uploads."""
url = normalise_url(text)
args = [
"--flat-playlist",
"--playlist-items",
"0",
"-J",
"--no-warnings",
"--ignore-config",
*ytdlp.extractor_args(settings.get_str("pot_provider_url")),
url,
]
try:
data = ytdlp.run_json(args, timeout=180)
except ytdlp.YtdlpError as exc:
raise ResolutionError(str(exc)) from exc
channel_id = data.get("channel_id") or data.get("id") or ""
if not _CHANNEL_ID.match(channel_id):
raise ResolutionError(f"no channel id found for {url}")
handle = data.get("uploader_id") or ""
if handle and not handle.startswith("@"):
handle = ""
return {
"channel_id": channel_id,
"title": (data.get("channel") or data.get("title") or channel_id).strip(),
"description": data.get("description") or "",
"handle": handle,
"thumbnails": data.get("thumbnails") or [],
}
def uulf_playlist_id(channel_id: str) -> str:
"""Long-form-only uploads playlist for a channel (specs.md §4)."""
return "UULF" + channel_id[2:]
# --------------------------------------------------------------------------
# artwork
def _pick_thumbnail(thumbnails: list[dict], wanted_id: str) -> str | None:
for thumb in thumbnails:
if str(thumb.get("id", "")) == wanted_id and thumb.get("url"):
return thumb["url"]
return None
def _download_image(url: str, destination: Path) -> bool:
"""Fetch an image and normalise it to JPEG via ffmpeg.
YouTube serves avatars as webp as often as jpeg, and naming a webp file
.jpg would be a lie some clients notice.
"""
try:
request = urllib.request.Request(
url, headers={"User-Agent": config.USER_AGENT}
)
with urllib.request.urlopen(request, timeout=60) as response:
payload = response.read()
except OSError as exc:
log.warning("artwork download failed (%s): %s", url, exc)
return False
with tempfile.NamedTemporaryFile(suffix=".img", delete=True) as raw:
raw.write(payload)
raw.flush()
result = subprocess.run(
["ffmpeg", "-y", "-loglevel", "error", "-i", raw.name, str(destination)],
capture_output=True,
text=True,
timeout=120,
)
if result.returncode != 0:
log.warning("artwork conversion failed: %s", result.stderr.strip()[:200])
return False
return True
def write_channel_metadata(channel_dir: Path, info: dict) -> None:
"""Write tvshow.nfo and best-effort artwork into the channel directory."""
channel_dir.mkdir(parents=True, exist_ok=True)
nfo.write(
channel_dir / "tvshow.nfo",
nfo.tvshow_nfo(info["title"], info.get("description"), info["channel_id"]),
)
for thumb_id, filename in _ARTWORK:
url = _pick_thumbnail(info.get("thumbnails") or [], thumb_id)
if url:
_download_image(url, channel_dir / filename)
# --------------------------------------------------------------------------
# subscribe / unsubscribe
def get(conn: sqlite3.Connection, pk: int) -> sqlite3.Row | None:
return conn.execute("SELECT * FROM channel WHERE id = ?", (pk,)).fetchone()
def get_by_channel_id(conn: sqlite3.Connection, channel_id: str) -> sqlite3.Row | None:
return conn.execute(
"SELECT * FROM channel WHERE channel_id = ?", (channel_id,)
).fetchone()
def all_channels(conn: sqlite3.Connection) -> list[sqlite3.Row]:
return conn.execute("SELECT * FROM channel ORDER BY title COLLATE NOCASE").fetchall()
def _unique_dir_name(conn: sqlite3.Connection, base: str) -> str:
"""dir_name is UNIQUE; two channels can legitimately share a title."""
candidate = base
suffix = 2
while conn.execute(
"SELECT 1 FROM channel WHERE dir_name = ?", (candidate,)
).fetchone():
candidate = f"{base} ({suffix})"
suffix += 1
return candidate
def subscribe(conn: sqlite3.Connection, settings: Settings, text: str) -> sqlite3.Row:
"""Resolve, insert and lay down on-disk metadata. Raises ResolutionError."""
info = resolve(settings, text)
existing = get_by_channel_id(conn, info["channel_id"])
if existing:
raise ResolutionError(f"already subscribed to {existing['title']}")
dir_name = _unique_dir_name(
conn, naming.channel_dir_name(info["title"], info["channel_id"])
)
with conn:
cursor = conn.execute(
"INSERT INTO channel (channel_id, handle, title, description, dir_name, added_at) "
"VALUES (?, ?, ?, ?, ?, ?)",
(
info["channel_id"],
info["handle"],
info["title"],
info["description"],
dir_name,
util.utcnow_iso(),
),
)
pk = cursor.lastrowid
write_channel_metadata(config.MEDIA_ROOT / dir_name, info)
log.info("subscribed to %s (%s)", info["title"], info["channel_id"])
return get(conn, pk)
def refresh_metadata(conn: sqlite3.Connection, settings: Settings, pk: int) -> None:
"""Re-resolve a channel and rewrite tvshow.nfo if the title changed.
dir_name is deliberately never recomputed — channels rename themselves and
we do not want orphaned directories.
"""
row = get(conn, pk)
if row is None:
return
info = resolve(settings, row["channel_id"])
if info["title"] != row["title"] or info["description"] != (row["description"] or ""):
with conn:
conn.execute(
"UPDATE channel SET title = ?, description = ? WHERE id = ?",
(info["title"], info["description"], pk),
)
write_channel_metadata(config.MEDIA_ROOT / row["dir_name"], info)
def unsubscribe(conn: sqlite3.Connection, pk: int) -> str:
"""Hard delete: remove the directory tree, then the rows. Irreversible."""
row = get(conn, pk)
if row is None:
raise LookupError(f"no channel with id {pk}")
title = row["title"]
channel_dir = config.MEDIA_ROOT / row["dir_name"]
if channel_dir.is_dir():
shutil.rmtree(channel_dir, ignore_errors=True)
with conn:
conn.execute("DELETE FROM channel WHERE id = ?", (pk,))
log.info("unsubscribed from %s and removed %s", title, channel_dir)
return title
youtube_automate/cli.py
+372
View File
@@ -0,0 +1,372 @@
"""Command line entry point."""
from __future__ import annotations
import argparse
import getpass
import sys
from . import (
channels,
config,
db,
discovery,
doctor,
download,
jellyfin,
reap,
runner,
util,
)
from .settings import Settings
from .web import auth
def _open():
conn = db.connect()
return conn, Settings(conn)
# --------------------------------------------------------------------------
# commands
def cmd_doctor(args) -> int:
conn, settings = _open()
try:
text, code = doctor.report(doctor.run_checks(settings))
print(text)
return code
finally:
conn.close()
def cmd_set_password(args) -> int:
conn, settings = _open()
try:
password = getpass.getpass("New admin password: ")
if len(password) < 8:
print("Password must be at least 8 characters.", file=sys.stderr)
return 1
if password != getpass.getpass("Repeat: "):
print("Passwords did not match.", file=sys.stderr)
return 1
settings.set("admin_password_hash", auth.hash_password(password))
if not settings.raw("session_secret"):
settings.set("session_secret", auth.new_secret())
print("Admin password set.")
return 0
finally:
conn.close()
def cmd_set_jellyfin_key(args) -> int:
conn, settings = _open()
try:
key = args.key or getpass.getpass("Jellyfin API key: ")
key = key.strip()
if not key:
print("No key given.", file=sys.stderr)
return 1
client = jellyfin.Jellyfin(settings.get_str("jellyfin_url"), key)
try:
client.virtual_folders()
except jellyfin.JellyfinError as exc:
print(f"Key rejected by Jellyfin: {exc}", file=sys.stderr)
return 1
settings.set("jellyfin_api_key", key)
print("Jellyfin API key stored and verified.")
return 0
finally:
conn.close()
def cmd_setup_jellyfin_library(args) -> int:
conn, settings = _open()
try:
client = jellyfin.from_settings(settings)
if not client.configured:
print("Set jellyfin_url and the API key first.", file=sys.stderr)
return 1
existing = client.find_library(config.MEDIA_ROOT)
if existing:
print(
f"Library '{existing.get('Name')}' already covers {config.MEDIA_ROOT}."
)
return 0
client.create_library(config.MEDIA_ROOT)
created = client.find_library(config.MEDIA_ROOT)
if not created:
print("Library creation reported success but it is not present.",
file=sys.stderr)
return 1
print(f"Created library '{created.get('Name')}' for {config.MEDIA_ROOT}.")
return 0
finally:
conn.close()
def cmd_subscribe(args) -> int:
conn, settings = _open()
try:
try:
row = channels.subscribe(conn, settings, args.url)
except channels.ResolutionError as exc:
print(f"Could not subscribe: {exc}", file=sys.stderr)
return 1
print(f"Subscribed to {row['title']} ({row['channel_id']}) -> {row['dir_name']}/")
return 0
finally:
conn.close()
def cmd_unsubscribe(args) -> int:
conn, settings = _open()
try:
row = channels.get(conn, args.id)
if row is None:
print(f"No channel with id {args.id}.", file=sys.stderr)
return 1
if not args.yes:
print(f"This permanently deletes {config.MEDIA_ROOT / row['dir_name']} "
f"and all rows for {row['title']}.")
if input("Type the channel title to confirm: ").strip() != row["title"]:
print("Aborted.")
return 1
title = channels.unsubscribe(conn, args.id)
jellyfin.from_settings(settings).refresh()
print(f"Unsubscribed from {title}.")
return 0
finally:
conn.close()
def cmd_poll(args) -> int:
conn, settings = _open()
try:
if args.rescan:
rows = (
[channels.get(conn, args.channel)]
if args.channel
else channels.all_channels(conn)
)
total = sum(
discovery.rescan_channel(conn, settings, row) for row in rows if row
)
print(f"re-queued {total} previously-skipped video(s)")
totals = discovery.poll_all(conn, settings, args.channel)
print(
f"queued={totals['queued']} extended={totals['extended']} "
f"already-known={totals['known']} outside-window={totals['old']} "
f"repaired={totals['repaired']} poll-failures={totals['failed']}"
)
return 1 if totals["failed"] else 0
finally:
conn.close()
def cmd_download(args) -> int:
conn, settings = _open()
try:
videos_recovered = download.recover_orphans()
if videos_recovered:
print(f"cleared {videos_recovered} orphan file(s) from the work dir")
try:
counts = download.drain(conn, settings, args.limit)
except RuntimeError as exc:
print(str(exc), file=sys.stderr)
return 1
if not counts:
print("nothing to download")
return 0
print(" ".join(f"{state}={count}" for state, count in sorted(counts.items())))
return 0
finally:
conn.close()
def cmd_reap(args) -> int:
conn, settings = _open()
try:
result = reap.run(conn, settings)
print(f"deleted={result['deleted']} evicted={result['evicted']}")
return 0
finally:
conn.close()
def cmd_run(args) -> int:
try:
with runner.exclusive_lock():
conn, settings = _open()
try:
result = runner.run(conn, settings, args.channel)
print(runner.summarise(result))
return 1 if "error" in result.get("download", {}) else 0
finally:
conn.close()
except runner.AlreadyRunning:
# Expected when a long backfill outlasts the hourly cron tick.
return 0
def cmd_serve(args) -> int:
from .web import server
server.serve(args.host, args.port, secure_cookies=not args.insecure_cookies)
return 0
def cmd_set_retention(args) -> int:
conn, _ = _open()
try:
row = channels.get(conn, args.id)
if row is None:
print(f"No channel with id {args.id}.", file=sys.stderr)
return 1
if args.days.lower() in ("default", "none", "clear"):
value = None
else:
try:
value = int(args.days)
except ValueError:
print("days must be a whole number or 'default'.", file=sys.stderr)
return 1
if value < 1:
print("days must be at least 1.", file=sys.stderr)
return 1
with conn:
conn.execute(
"UPDATE channel SET retention_days = ? WHERE id = ?", (value, args.id)
)
shown = "the global default" if value is None else f"{value} days"
print(f"{row['title']} retention set to {shown}.")
return 0
finally:
conn.close()
def cmd_channels(args) -> int:
conn, _ = _open()
try:
rows = channels.all_channels(conn)
if not rows:
print("No channels subscribed.")
return 0
print(f"{'id':>3} {'title':<32} {'handle':<20} {'videos':>6} last poll")
for row in rows:
counts = conn.execute(
"SELECT COUNT(*) FROM video WHERE channel_pk = ? AND state = 'downloaded'",
(row["id"],),
).fetchone()[0]
print(
f"{row['id']:>3} {row['title'][:32]:<32} {(row['handle'] or ''):<20} "
f"{counts:>6} {row['last_polled_at'] or 'never'}"
)
return 0
finally:
conn.close()
# --------------------------------------------------------------------------
def build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
prog="youtube-automate",
description="A DVR for YouTube subscriptions, laid out for Jellyfin.",
)
parser.add_argument("-v", "--verbose", action="store_true", help="debug logging")
sub = parser.add_subparsers(dest="command", required=True)
sub.add_parser("doctor", help="check the installation").set_defaults(
func=cmd_doctor
)
sub.add_parser("set-password", help="set the admin UI password").set_defaults(
func=cmd_set_password
)
key = sub.add_parser("set-jellyfin-key", help="store and verify the Jellyfin API key")
key.add_argument("key", nargs="?", help="omit to be prompted (preferred)")
key.set_defaults(func=cmd_set_jellyfin_key)
sub.add_parser(
"setup-jellyfin-library", help="create the Shows library for the media root"
).set_defaults(func=cmd_setup_jellyfin_library)
subscribe = sub.add_parser("subscribe", help="subscribe to a channel")
subscribe.add_argument("url", help="channel URL, @handle, or UC... id")
subscribe.set_defaults(func=cmd_subscribe)
unsubscribe = sub.add_parser(
"unsubscribe", help="remove a channel and everything it downloaded"
)
unsubscribe.add_argument("id", type=int, help="channel id from `channels`")
unsubscribe.add_argument(
"--yes", action="store_true", help="skip the confirmation prompt"
)
unsubscribe.set_defaults(func=cmd_unsubscribe)
sub.add_parser("channels", help="list subscribed channels").set_defaults(
func=cmd_channels
)
poll = sub.add_parser("poll", help="discover new videos")
poll.add_argument("--channel", type=int, help="restrict to one channel id")
poll.add_argument(
"--rescan",
action="store_true",
help="also re-queue videos previously skipped as too old that the "
"current retention window now covers",
)
poll.set_defaults(func=cmd_poll)
serve = sub.add_parser("serve", help="run the admin web server")
serve.add_argument("--host", default="127.0.0.1")
serve.add_argument("--port", type=int, default=8085)
serve.add_argument(
"--insecure-cookies",
action="store_true",
help="omit the Secure cookie flag (local testing over plain http only)",
)
serve.set_defaults(func=cmd_serve)
run_cmd = sub.add_parser("run", help="poll, download and reap (what cron calls)")
run_cmd.add_argument("--channel", type=int, help="restrict discovery to one channel")
run_cmd.set_defaults(func=cmd_run)
sub.add_parser("reap", help="delete videos past the retention window").set_defaults(
func=cmd_reap
)
down = sub.add_parser("download", help="drain the pending queue")
down.add_argument("--limit", type=int, help="stop after this many videos")
down.set_defaults(func=cmd_download)
retention = sub.add_parser(
"set-retention", help="set or clear a channel's retention override"
)
retention.add_argument("id", type=int, help="channel id")
retention.add_argument(
"days", help="number of days, or 'default' to clear the override"
)
retention.set_defaults(func=cmd_set_retention)
return parser
def main(argv: list[str] | None = None) -> int:
parser = build_parser()
args = parser.parse_args(argv)
util.setup_logging(args.verbose)
util.apply_umask()
return args.func(args)
youtube_automate/config.py
+33
View File
@@ -0,0 +1,33 @@
"""Filesystem paths and process-level constants.
Every path is overridable through the environment so the test suite can point the
whole application at a tmpdir without touching the real media tree.
"""
from __future__ import annotations
import os
from pathlib import Path
def _path(env: str, default: str) -> Path:
return Path(os.environ.get(env, default))
STATE_DIR = _path("YTA_STATE_DIR", "/var/lib/youtube-automate")
MEDIA_ROOT = _path("YTA_MEDIA_ROOT", "/disks/Plex/YouTube")
DB_PATH = _path("YTA_DB_PATH", str(STATE_DIR / "subs.db"))
LOCK_PATH = _path("YTA_LOCK_PATH", str(STATE_DIR / "run.lock"))
VENV_BIN = _path("YTA_VENV_BIN", str(STATE_DIR / "venv" / "bin"))
WORK_DIR = MEDIA_ROOT / ".work"
# Files created by the service must stay group-readable by `mediaserver`, which is
# how Jellyfin reaches the tree. See specs.md §2.
UMASK = 0o002
# Media-adjacent sidecars we own and therefore may delete on reap.
SIDECAR_SUFFIXES = (".nfo", "-thumb.jpg", ".en.srt", ".info.json")
USER_AGENT = "youtube-automate/1.0"
youtube_automate/db.py
+87
View File
@@ -0,0 +1,87 @@
"""SQLite access and schema migrations.
WAL is mandatory: the hourly cron job and the long-running admin server both write.
"""
from __future__ import annotations
import sqlite3
from pathlib import Path
from . import config
SCHEMA_VERSION = 1
_SCHEMA_V1 = """
CREATE TABLE IF NOT EXISTS channel (
id INTEGER PRIMARY KEY,
channel_id TEXT NOT NULL UNIQUE,
handle TEXT,
title TEXT NOT NULL,
description TEXT,
dir_name TEXT NOT NULL UNIQUE,
added_at TEXT NOT NULL,
backfilled INTEGER NOT NULL DEFAULT 0,
retention_days INTEGER,
last_polled_at TEXT,
last_poll_ok INTEGER,
consecutive_poll_failures INTEGER NOT NULL DEFAULT 0
);
CREATE TABLE IF NOT EXISTS video (
id INTEGER PRIMARY KEY,
video_id TEXT NOT NULL UNIQUE,
channel_pk INTEGER NOT NULL REFERENCES channel(id) ON DELETE CASCADE,
title TEXT,
upload_date TEXT,
duration INTEGER,
season INTEGER,
episode INTEGER,
state TEXT NOT NULL,
discovery_source TEXT NOT NULL,
rel_path TEXT,
size_bytes INTEGER,
attempts INTEGER NOT NULL DEFAULT 0,
last_error TEXT,
discovered_at TEXT NOT NULL,
downloaded_at TEXT,
deleted_at TEXT
);
CREATE INDEX IF NOT EXISTS idx_video_state ON video(state);
CREATE INDEX IF NOT EXISTS idx_video_upload_date ON video(upload_date);
CREATE INDEX IF NOT EXISTS idx_video_channel ON video(channel_pk);
CREATE TABLE IF NOT EXISTS setting (
key TEXT PRIMARY KEY,
value TEXT NOT NULL
);
"""
def connect(path: Path | None = None) -> sqlite3.Connection:
"""Open the database, applying migrations if needed."""
path = Path(path) if path is not None else config.DB_PATH
path.parent.mkdir(parents=True, exist_ok=True)
conn = sqlite3.connect(path, timeout=30.0, isolation_level=None)
conn.row_factory = sqlite3.Row
conn.execute("PRAGMA journal_mode = WAL")
conn.execute("PRAGMA foreign_keys = ON")
conn.execute("PRAGMA busy_timeout = 30000")
migrate(conn)
return conn
def migrate(conn: sqlite3.Connection) -> int:
"""Bring the schema up to SCHEMA_VERSION. Idempotent."""
current = conn.execute("PRAGMA user_version").fetchone()[0]
if current >= SCHEMA_VERSION:
return current
with conn:
if current < 1:
conn.executescript(_SCHEMA_V1)
# Future migrations append here, each guarded by `if current < N`.
conn.execute(f"PRAGMA user_version = {SCHEMA_VERSION}")
return SCHEMA_VERSION
youtube_automate/discovery.py
+363
View File
@@ -0,0 +1,363 @@
"""Discovery: RSS polling and the subscribe-time backfill.
Primary path is the undocumented UULF uploads playlist feed, which excludes
Shorts and livestreams at the cheapest possible point (verified — see specs.md
§4). The channel_id feed is the fallback, and rows discovered that way carry
`discovery_source='uc_feed'` so the download step knows to apply the duration and
live-status match filter.
"""
from __future__ import annotations
import logging
import sqlite3
import urllib.error
import urllib.request
import xml.etree.ElementTree as ET
from datetime import date, timedelta
from . import channels, config, util, videos, ytdlp
from .settings import Settings
log = logging.getLogger(__name__)
NS = {
"atom": "http://www.w3.org/2005/Atom",
"yt": "http://www.youtube.com/xml/schemas/2015",
"media": "http://search.yahoo.com/mrss/",
}
FEED_BASE = "https://www.youtube.com/feeds/videos.xml"
class FeedUnavailable(Exception):
"""The feed could not be fetched at all (network/5xx). Not the same as 404."""
def uulf_feed_url(channel_id: str) -> str:
return f"{FEED_BASE}?playlist_id={channels.uulf_playlist_id(channel_id)}"
def uc_feed_url(channel_id: str) -> str:
return f"{FEED_BASE}?channel_id={channel_id}"
def fetch_feed(url: str, timeout: float = 30.0) -> bytes | None:
"""Return the feed body, or None if YouTube says it doesn't exist.
A 404 on UULF/UUSH/UULV means "no such playlist", i.e. the channel has none
of that kind of video — it is not an error.
"""
request = urllib.request.Request(url, headers={"User-Agent": config.USER_AGENT})
try:
with urllib.request.urlopen(request, timeout=timeout) as response:
return response.read()
except urllib.error.HTTPError as exc:
if exc.code == 404:
return None
raise FeedUnavailable(f"HTTP {exc.code}") from exc
except OSError as exc:
raise FeedUnavailable(str(exc)) from exc
def parse_entries(payload: bytes) -> list[dict]:
"""Parse an Atom feed into video dicts. The feed carries no duration."""
try:
root = ET.fromstring(payload)
except ET.ParseError as exc:
raise FeedUnavailable(f"unparseable feed: {exc}") from exc
entries = []
for entry in root.findall("atom:entry", NS):
video_id = entry.findtext("yt:videoId", "", NS)
if not video_id:
continue
published = entry.findtext("atom:published", "", NS)
try:
published_date = date.fromisoformat(published[:10])
except ValueError:
continue
description = entry.findtext("media:group/media:description", "", NS)
entries.append(
{
"video_id": video_id,
"title": (entry.findtext("atom:title", "", NS) or "").strip(),
"published": published_date,
"description": description or "",
}
)
return entries
def effective_retention_days(settings: Settings, channel: sqlite3.Row) -> int:
override = channel["retention_days"] if "retention_days" in channel.keys() else None
if override:
return int(override)
return settings.get_int("retention_days")
# --------------------------------------------------------------------------
def _record(
conn: sqlite3.Connection,
channel: sqlite3.Row,
entry: dict,
source: str,
cutoff: date,
) -> str:
"""Insert or repair one discovered video. Returns what happened."""
existing = videos.get(conn, entry["video_id"])
if existing is not None:
# The only repair we ever perform: a video the fallback path rejected as
# too short, later confirmed long-form by the authoritative UULF feed.
# Deleted rows are tombstones and are never touched here.
if (
source == videos.SOURCE_UULF
and existing["state"] == videos.SKIPPED_SHORT
and existing["discovery_source"] == videos.SOURCE_UC
):
with conn:
conn.execute(
"UPDATE video SET state = ?, discovery_source = ?, "
"last_error = NULL WHERE video_id = ?",
(videos.PENDING, videos.SOURCE_UULF, entry["video_id"]),
)
log.info(
"re-queued %s: UULF confirms it is long-form", entry["video_id"]
)
return "repaired"
return "known"
state = videos.PENDING if entry["published"] >= cutoff else videos.SKIPPED_OLD
videos.insert(
conn,
channel_pk=channel["id"],
video_id=entry["video_id"],
title=entry["title"],
upload_date=entry["published"].isoformat(),
state=state,
discovery_source=source,
)
return "queued" if state == videos.PENDING else "old"
def poll_channel(conn: sqlite3.Connection, settings: Settings, channel: sqlite3.Row) -> dict:
"""Poll one channel. Never raises for feed problems — records them instead."""
stats = {"queued": 0, "old": 0, "known": 0, "repaired": 0, "source": None}
source = videos.SOURCE_UULF
try:
payload = fetch_feed(uulf_feed_url(channel["channel_id"]))
entries = parse_entries(payload) if payload else []
if not entries:
# UULF 404'd or came back empty — fall back to the channel feed.
log.warning(
"UULF feed empty for %s, falling back to channel_id feed",
channel["title"],
)
source = videos.SOURCE_UC
payload = fetch_feed(uc_feed_url(channel["channel_id"]))
entries = parse_entries(payload) if payload else []
except FeedUnavailable as exc:
_record_poll_failure(conn, channel, str(exc))
stats["error"] = str(exc)
return stats
cutoff = util.today() - timedelta(days=effective_retention_days(settings, channel))
for entry in entries:
stats[_record(conn, channel, entry, source, cutoff)] += 1
stats["source"] = source
_record_poll_success(conn, channel)
return stats
def _record_poll_success(conn: sqlite3.Connection, channel: sqlite3.Row) -> None:
with conn:
conn.execute(
"UPDATE channel SET last_polled_at = ?, last_poll_ok = 1, "
"consecutive_poll_failures = 0 WHERE id = ?",
(util.utcnow_iso(), channel["id"]),
)
def _record_poll_failure(conn: sqlite3.Connection, channel: sqlite3.Row, error: str) -> None:
log.error("poll failed for %s: %s", channel["title"], error)
with conn:
conn.execute(
"UPDATE channel SET last_polled_at = ?, last_poll_ok = 0, "
"consecutive_poll_failures = consecutive_poll_failures + 1 WHERE id = ?",
(util.utcnow_iso(), channel["id"]),
)
# --------------------------------------------------------------------------
# backfill
def _flat_playlist(settings: Settings, channel_id: str, limit: int) -> list[dict]:
"""Reverse-chronological upload list. Entries carry no upload date."""
args = [
"--flat-playlist",
"--playlist-end",
str(limit),
"-J",
"--no-warnings",
"--ignore-config",
*ytdlp.extractor_args(settings.get_str("pot_provider_url")),
f"https://www.youtube.com/playlist?list={channels.uulf_playlist_id(channel_id)}",
]
try:
data = ytdlp.run_json(args, timeout=300)
except ytdlp.YtdlpError as exc:
log.warning("flat playlist failed for %s: %s", channel_id, exc)
return []
return [entry for entry in (data.get("entries") or []) if entry.get("id")]
def _upload_date(settings: Settings, video_id: str) -> date | None:
"""One extraction to learn a single video's upload date."""
args = [
"--skip-download",
"--no-warnings",
"--ignore-config",
"--no-playlist",
"--print",
"%(upload_date)s",
*ytdlp.extractor_args(settings.get_str("pot_provider_url")),
f"https://www.youtube.com/watch?v={video_id}",
]
result = ytdlp.run(args, timeout=180)
text = (result.stdout or "").strip().splitlines()
if result.returncode != 0 or not text:
return None
try:
from . import naming
return naming.parse_upload_date(text[-1])
except ValueError:
return None
def backfill_channel(
conn: sqlite3.Connection, settings: Settings, channel: sqlite3.Row
) -> dict:
"""Queue the last `backfill_days` for a freshly subscribed channel.
The RSS feed is the primary source because it is the only one that carries
upload dates — `--flat-playlist` reports `timestamp: None` for every entry.
RSS returns ~15 items, which covers the default 7-day window for any channel
uploading less than twice a day. Only when the feed's oldest entry is still
inside the window do we extend via the playlist, resolving those extra dates
one video at a time.
"""
days = settings.get_int("backfill_days")
cutoff = util.today() - timedelta(days=days)
stats = {"queued": 0, "old": 0, "known": 0, "repaired": 0, "extended": 0}
try:
payload = fetch_feed(uulf_feed_url(channel["channel_id"]))
entries = parse_entries(payload) if payload else []
source = videos.SOURCE_UULF
if not entries:
source = videos.SOURCE_UC
payload = fetch_feed(uc_feed_url(channel["channel_id"]))
entries = parse_entries(payload) if payload else []
except FeedUnavailable as exc:
_record_poll_failure(conn, channel, str(exc))
stats["error"] = str(exc)
return stats
seen = set()
for entry in entries:
seen.add(entry["video_id"])
stats[_record(conn, channel, entry, source, cutoff)] += 1
oldest = min((entry["published"] for entry in entries), default=None)
if oldest is not None and oldest >= cutoff:
# The feed did not reach past the window, so there may be more.
log.info(
"%s: RSS reaches only to %s, extending backfill via playlist",
channel["title"],
oldest.isoformat(),
)
for item in _flat_playlist(settings, channel["channel_id"], 50):
video_id = item["id"]
if video_id in seen or videos.exists(conn, video_id):
continue
upload_date = _upload_date(settings, video_id)
if upload_date is None:
continue
if upload_date < cutoff:
break # playlist is reverse-chronological; everything after is older
videos.insert(
conn,
channel_pk=channel["id"],
video_id=video_id,
title=(item.get("title") or "").strip(),
upload_date=upload_date.isoformat(),
duration=item.get("duration"),
state=videos.PENDING,
discovery_source=videos.SOURCE_BACKFILL,
)
stats["extended"] += 1
with conn:
conn.execute("UPDATE channel SET backfilled = 1 WHERE id = ?", (channel["id"],))
_record_poll_success(conn, channel)
return stats
def rescan_channel(
conn: sqlite3.Connection, settings: Settings, channel: sqlite3.Row
) -> int:
"""Re-queue `skipped_old` rows that the current retention window now covers.
`skipped_old` is judged against whatever window was in force at discovery
time, and it is otherwise terminal. Without this, raising a channel's
retention_days would appear to do nothing for an infrequent uploader —
every one of their videos is already marked old. This is deliberately an
explicit action rather than something poll does silently, because doing it
on every poll would make `backfill_days` meaningless: it would immediately
re-queue everything the initial backfill had deliberately left behind.
Tombstones (`deleted`) are never touched.
"""
cutoff = util.today() - timedelta(days=effective_retention_days(settings, channel))
with conn:
cursor = conn.execute(
"UPDATE video SET state = ?, last_error = NULL "
"WHERE channel_pk = ? AND state = ? AND upload_date >= ?",
(videos.PENDING, channel["id"], videos.SKIPPED_OLD, cutoff.isoformat()),
)
if cursor.rowcount:
log.info(
"%s: re-queued %d video(s) now inside the %s window",
channel["title"],
cursor.rowcount,
cutoff.isoformat(),
)
return cursor.rowcount
def poll_all(conn: sqlite3.Connection, settings: Settings, channel_pk: int | None = None) -> dict:
"""Backfill anything new, then poll everything. Returns aggregate counts."""
if channel_pk is not None:
rows = [row for row in [channels.get(conn, channel_pk)] if row is not None]
else:
rows = channels.all_channels(conn)
totals = {"queued": 0, "old": 0, "known": 0, "repaired": 0, "extended": 0, "failed": 0}
for channel in rows:
if not channel["backfilled"]:
stats = backfill_channel(conn, settings, channel)
else:
stats = poll_channel(conn, settings, channel)
if "error" in stats:
totals["failed"] += 1
for key in ("queued", "old", "known", "repaired", "extended"):
totals[key] += stats.get(key, 0)
log.info("%s: %s", channel["title"], stats)
return totals
youtube_automate/doctor.py
+203
View File
@@ -0,0 +1,203 @@
"""Preflight checks.
`doctor` is the first acceptance criterion and the thing to run when something
breaks. Every check returns a row rather than raising, so one failure doesn't
hide the others.
"""
from __future__ import annotations
import grp
import os
import stat
import subprocess
from dataclasses import dataclass
from pathlib import Path
from . import config, jellyfin, ytdlp
from .settings import Settings
MEDIA_GROUP = "mediaserver"
@dataclass
class Check:
name: str
ok: bool
detail: str
fatal: bool = True
def _deno() -> Check:
binary = config.VENV_BIN / "deno"
if not binary.exists():
return Check(
"js runtime",
False,
f"deno not found at {binary} — yt-dlp cannot solve n challenges (specs.md §3)",
)
try:
result = subprocess.run(
[str(binary), "--version"], capture_output=True, text=True, timeout=30
)
except OSError as exc:
return Check("js runtime", False, f"deno unusable: {exc}")
if result.returncode != 0:
return Check("js runtime", False, "deno --version failed")
return Check("js runtime", True, result.stdout.splitlines()[0])
def _ytdlp() -> Check:
try:
return Check("yt-dlp", True, ytdlp.version())
except (OSError, ytdlp.YtdlpError) as exc:
return Check("yt-dlp", False, str(exc))
def _ejs() -> Check:
try:
from importlib.metadata import version as pkg_version
return Check("yt-dlp-ejs", True, pkg_version("yt-dlp-ejs"))
except Exception:
return Check(
"yt-dlp-ejs",
False,
"not installed — reinstall with the yt-dlp[default] extra (specs.md §3)",
)
def _pot(settings: Settings) -> Check:
url = settings.get_str("pot_provider_url")
try:
info = ytdlp.pot_provider_ping(url)
except Exception as exc:
return Check("pot provider", False, f"{url}/ping unreachable: {exc}")
server_version = str(info.get("version", "?"))
installed = ytdlp.plugin_version()
if installed and installed != server_version:
return Check(
"pot provider",
False,
f"version skew: server {server_version} vs plugin {installed}",
)
return Check("pot provider", True, f"up, version {server_version}")
def _database() -> Check:
try:
from . import db
conn = db.connect()
version = conn.execute("PRAGMA user_version").fetchone()[0]
mode = conn.execute("PRAGMA journal_mode").fetchone()[0]
conn.close()
except Exception as exc:
return Check("database", False, str(exc))
if str(mode).lower() != "wal":
return Check("database", False, f"journal_mode is {mode}, expected wal")
return Check("database", True, f"{config.DB_PATH} (schema v{version}, {mode})")
def _media_root() -> Check:
root = config.MEDIA_ROOT
if not root.is_dir():
return Check("media root", False, f"{root} does not exist")
if not os.access(root, os.W_OK | os.X_OK):
return Check("media root", False, f"{root} is not writable")
info = root.stat()
problems = []
try:
group = grp.getgrgid(info.st_gid).gr_name
except KeyError:
group = str(info.st_gid)
if group != MEDIA_GROUP:
problems.append(f"group is {group}, expected {MEDIA_GROUP}")
if not info.st_mode & stat.S_ISGID:
problems.append("setgid bit not set (new dirs won't inherit the group)")
if problems:
return Check("media root", False, f"{root}: " + "; ".join(problems))
return Check("media root", True, f"{root} ({group}, setgid)")
def _work_dir() -> Check:
work = config.WORK_DIR
if not work.is_dir():
return Check("work dir", False, f"{work} does not exist")
if work.stat().st_dev != config.MEDIA_ROOT.stat().st_dev:
return Check(
"work dir",
False,
"not on the same filesystem as the media root — moves would be copies",
)
if not (work / ".ignore").exists():
return Check("work dir", False, f"{work}/.ignore missing", fatal=False)
return Check("work dir", True, f"{work} (same fs, .ignore present)")
def _jellyfin(settings: Settings) -> Check:
client = jellyfin.from_settings(settings)
if not client.base_url:
return Check("jellyfin", False, "jellyfin_url not set", fatal=False)
try:
info = client.public_info()
except jellyfin.JellyfinError as exc:
return Check("jellyfin", False, str(exc))
label = f"{info.get('ServerName', '?')} {info.get('Version', '?')}"
if not client.api_key:
return Check(
"jellyfin",
False,
f"{label} reachable but no API key set (run set-jellyfin-key)",
fatal=False,
)
try:
library = client.find_library(config.MEDIA_ROOT)
except jellyfin.JellyfinError as exc:
return Check("jellyfin", False, f"API key rejected: {exc}")
if library is None:
return Check(
"jellyfin",
False,
f"{label}, no library for {config.MEDIA_ROOT} (run setup-jellyfin-library)",
fatal=False,
)
return Check("jellyfin", True, f"{label}, library '{library.get('Name')}'")
def run_checks(settings: Settings) -> list[Check]:
return [
_ytdlp(),
_ejs(),
_deno(),
_pot(settings),
_database(),
_media_root(),
_work_dir(),
_jellyfin(settings),
]
def report(checks: list[Check]) -> tuple[str, int]:
"""Render the checks and return (text, exit_code)."""
lines = []
failed_fatal = 0
for check in checks:
if check.ok:
mark = "ok "
elif check.fatal:
mark = "FAIL"
failed_fatal += 1
else:
mark = "warn"
lines.append(f" [{mark}] {check.name:<14} {check.detail}")
if failed_fatal:
lines.append(f"\n{failed_fatal} fatal problem(s).")
else:
lines.append("\nAll fatal checks passed.")
return "\n".join(lines), (1 if failed_fatal else 0)
youtube_automate/download.py
+330
View File
@@ -0,0 +1,330 @@
"""The download worker.
One video at a time, into `.work/`, then everything moves into the season
directory with `os.rename()` — which is atomic because the work dir shares a
filesystem with the media root.
"""
from __future__ import annotations
import json
import logging
import shutil
import sqlite3
from pathlib import Path
from . import config, jellyfin, naming, nfo, videos, ytdlp
from .settings import Settings
log = logging.getLogger(__name__)
# yt-dlp exit code / message fragments that mean "the match filter rejected it",
# which is a decision rather than a failure.
_REJECT_MARKERS = (
"does not pass filter",
"skipping ..",
)
class DownloadOutcome:
QUEUED = "queued"
DONE = "downloaded"
SKIPPED_SHORT = videos.SKIPPED_SHORT
SKIPPED_LIVE = videos.SKIPPED_LIVE
DEFERRED = videos.DEFERRED
FAILED = videos.FAILED
def build_args(settings: Settings, video: sqlite3.Row) -> list[str]:
"""The yt-dlp invocation from specs.md §6."""
max_height = settings.get_int("max_height")
args = [
*ytdlp.extractor_args(settings.get_str("pot_provider_url")),
"-f",
f"bv*[height<={max_height}]+ba/b[height<={max_height}]",
# vcodec must outrank res: on hardware that cannot transcode, h264 at a
# lower resolution beats VP9 at 720p. And acodec must NOT outrank res,
# or `bv*` picks the combined 360p stream because it carries AAC.
"-S",
f"vcodec:h264,res:{max_height},acodec:aac",
"--merge-output-format",
"mp4",
"--no-playlist",
"--ignore-config",
"--write-info-json",
"--write-thumbnail",
"--convert-thumbnails",
"jpg",
"--retries",
"3",
"--fragment-retries",
"10",
"--sleep-requests",
"2",
"--sleep-interval",
"5",
"--max-sleep-interval",
"15",
"-P",
str(config.WORK_DIR),
"-o",
"%(id)s.%(ext)s",
]
if settings.get_bool("write_subs"):
args += [
"--write-subs",
"--write-auto-subs",
"--sub-langs",
settings.get_str("sub_langs"),
"--convert-subs",
"srt",
]
if settings.get_bool("sponsorblock_mark"):
args += ["--sponsorblock-mark", "all", "--embed-chapters"]
# The match filter only applies to rows the fallback feed produced. UULF has
# already excluded Shorts and livestreams for everything else.
if video["discovery_source"] == videos.SOURCE_UC:
minimum = settings.get_int("min_duration_seconds")
args += [
"--match-filter",
f"duration>?{minimum} & live_status!=?is_live "
f"& live_status!=?is_upcoming & !was_live",
]
args.append(f"https://www.youtube.com/watch?v={video['video_id']}")
return args
def _work_files(video_id: str) -> list[Path]:
return sorted(config.WORK_DIR.glob(f"{video_id}.*"))
def cleanup_work(video_id: str) -> None:
for path in _work_files(video_id):
try:
path.unlink()
except OSError as exc: # pragma: no cover - unusual fs state
log.warning("could not remove %s: %s", path, exc)
def _classify_rejection(info: dict | None, stdout: str, stderr: str) -> str | None:
"""Decide why the match filter rejected a video, if it did."""
blob = f"{stdout}\n{stderr}".lower()
if not any(marker in blob for marker in _REJECT_MARKERS):
return None
live_status = (info or {}).get("live_status")
if live_status == "is_upcoming":
return DownloadOutcome.DEFERRED
if live_status in ("is_live", "was_live") or (info or {}).get("was_live"):
return DownloadOutcome.SKIPPED_LIVE
# Duration is the only other condition in our filter.
return DownloadOutcome.SKIPPED_SHORT
def _read_info_json(video_id: str) -> dict | None:
path = config.WORK_DIR / f"{video_id}.info.json"
if not path.exists():
return None
try:
return json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
log.warning("unreadable info.json for %s: %s", video_id, exc)
return None
def _choose_subtitle(video_id: str) -> Path | None:
"""`--sub-langs en.*` matches both `en` and `en-orig`, yielding two identical
English tracks. Prefer `.en.srt`; otherwise promote `en-orig`."""
preferred = config.WORK_DIR / f"{video_id}.en.srt"
if preferred.exists():
return preferred
candidates = sorted(config.WORK_DIR.glob(f"{video_id}.en*.srt"))
return candidates[0] if candidates else None
def _move_into_place(
conn: sqlite3.Connection, video: sqlite3.Row, info: dict
) -> tuple[str, int]:
"""Rename every artefact into the season directory. Returns (rel_path, size)."""
upload_date = naming.parse_upload_date(
info.get("upload_date") or video["upload_date"]
)
season, episode = videos.next_episode(
conn, video["channel_pk"], upload_date, video["video_id"]
)
title = (info.get("title") or video["title"] or video["video_id"]).strip()
stem = naming.basename(
video["dir_name"], season, episode, title, video["video_id"]
)
season_dir = config.MEDIA_ROOT / video["dir_name"] / naming.season_dir_name(season)
season_dir.mkdir(parents=True, exist_ok=True)
media_source = config.WORK_DIR / f"{video['video_id']}.mp4"
if not media_source.exists():
raise FileNotFoundError(f"no mp4 produced for {video['video_id']}")
# The episode NFO is generated into the work dir first so that a failure
# here never leaves a half-populated season directory.
nfo.write(
config.WORK_DIR / f"{video['video_id']}.nfo",
nfo.episode_nfo(
title=title,
show_title=video["channel_title"],
season=season,
episode=episode,
plot=info.get("description"),
aired=upload_date.isoformat(),
duration_seconds=info.get("duration"),
video_id=video["video_id"],
),
)
moves: list[tuple[Path, Path]] = [(media_source, season_dir / f"{stem}.mp4")]
for suffix, target in (
(".nfo", f"{stem}.nfo"),
(".info.json", f"{stem}.info.json"),
(".jpg", f"{stem}-thumb.jpg"),
):
source = config.WORK_DIR / f"{video['video_id']}{suffix}"
if source.exists():
moves.append((source, season_dir / target))
subtitle = _choose_subtitle(video["video_id"])
if subtitle:
moves.append((subtitle, season_dir / f"{stem}.en.srt"))
for source, destination in moves:
source.replace(destination) # same filesystem: atomic rename
size = (season_dir / f"{stem}.mp4").stat().st_size
rel_path = str(
(season_dir / f"{stem}.mp4").relative_to(config.MEDIA_ROOT)
)
videos.mark_downloaded(
conn,
video["video_id"],
rel_path=rel_path,
size_bytes=size,
season=season,
episode=episode,
upload_date=upload_date.isoformat(),
duration=info.get("duration"),
title=title,
)
return rel_path, size
def download_one(conn: sqlite3.Connection, settings: Settings, video: sqlite3.Row) -> str:
"""Download a single video. Returns the resulting state."""
video_id = video["video_id"]
config.WORK_DIR.mkdir(parents=True, exist_ok=True)
cleanup_work(video_id)
videos.set_state(conn, video_id, videos.DOWNLOADING)
result = ytdlp.run(build_args(settings, video))
info = _read_info_json(video_id)
rejection = _classify_rejection(info, result.stdout, result.stderr)
if rejection is not None:
cleanup_work(video_id)
videos.set_state(conn, video_id, rejection)
log.info("%s rejected by match filter -> %s", video_id, rejection)
return rejection
if result.returncode != 0:
error = ytdlp.first_error(result.stderr) or f"exit {result.returncode}"
cleanup_work(video_id)
outcome = videos.record_failure(
conn, video_id, error, settings.get_int("max_attempts")
)
log.error("%s failed: %s (%s)", video_id, error, outcome)
return videos.FAILED
if info is None:
cleanup_work(video_id)
videos.record_failure(
conn, video_id, "no info.json produced", settings.get_int("max_attempts")
)
return videos.FAILED
try:
rel_path, size = _move_into_place(conn, video, info)
except Exception as exc: # noqa: BLE001 - any failure must clean up
cleanup_work(video_id)
videos.record_failure(
conn, video_id, str(exc), settings.get_int("max_attempts")
)
log.exception("moving %s into place failed", video_id)
return videos.FAILED
cleanup_work(video_id)
_warn_if_not_h264(video_id, info)
log.info("downloaded %s -> %s (%.1f MB)", video_id, rel_path, size / 1e6)
return videos.DOWNLOADED
def _warn_if_not_h264(video_id: str, info: dict) -> None:
"""§6: log the cases where no h264 rendition existed, since Jellyfin will
have to transcode them and this box cannot."""
vcodec = str(info.get("vcodec") or "")
acodec = str(info.get("acodec") or "")
if vcodec and not vcodec.startswith(("avc1", "h264")):
log.warning("%s has no h264 rendition (got %s) — will transcode", video_id, vcodec)
if acodec and not acodec.startswith(("mp4a", "aac")):
log.warning("%s has no aac audio (got %s) — will transcode", video_id, acodec)
def drain(
conn: sqlite3.Connection, settings: Settings, limit: int | None = None
) -> dict:
"""Work the queue, one video at a time. Returns per-outcome counts."""
if not _provider_healthy(settings):
raise RuntimeError(
"PO token provider is not reachable — refusing to download and "
"accumulate 403s. Check the bgutil-provider container."
)
counts: dict[str, int] = {}
queue = videos.claim_pending(conn, settings.get_int("max_attempts"), limit)
log.info("%d video(s) in the queue", len(queue))
for video in queue:
state = download_one(conn, settings, video)
counts[state] = counts.get(state, 0) + 1
if counts.get(videos.DOWNLOADED):
jellyfin.from_settings(settings).refresh()
return counts
def _provider_healthy(settings: Settings) -> bool:
try:
ytdlp.pot_provider_ping(settings.get_str("pot_provider_url"))
return True
except Exception as exc: # noqa: BLE001
log.error("POT provider health check failed: %s", exc)
return False
def recover_orphans() -> int:
"""Remove anything left in the work dir by a killed run."""
if not config.WORK_DIR.is_dir():
return 0
removed = 0
for path in config.WORK_DIR.iterdir():
if path.name == ".ignore":
continue
try:
if path.is_dir():
shutil.rmtree(path, ignore_errors=True)
else:
path.unlink()
removed += 1
except OSError: # pragma: no cover
pass
return removed
youtube_automate/jellyfin.py
+134
View File
@@ -0,0 +1,134 @@
"""Minimal Jellyfin API client.
Only three things are needed: check the server is alive, create the Shows library
with internet metadata providers switched off, and trigger a refresh after we
change the tree.
"""
from __future__ import annotations
import json
import logging
import urllib.error
import urllib.parse
import urllib.request
from pathlib import Path
from . import config
log = logging.getLogger(__name__)
LIBRARY_NAME = "YouTube"
COLLECTION_TYPE = "tvshows"
# Metadata is supplied entirely by our own NFO sidecars, so every fetcher is
# disabled for all three item types a Shows library resolves.
_ITEM_TYPES = ("Series", "Season", "Episode")
class JellyfinError(RuntimeError):
pass
class Jellyfin:
def __init__(self, base_url: str, api_key: str = "", timeout: float = 30.0):
self.base_url = (base_url or "").rstrip("/")
self.api_key = api_key or ""
self.timeout = timeout
@property
def configured(self) -> bool:
return bool(self.base_url and self.api_key)
def _request(
self,
method: str,
path: str,
params: dict | None = None,
body: dict | None = None,
):
url = self.base_url + path
if params:
url += "?" + urllib.parse.urlencode(params)
data = None
headers = {"User-Agent": config.USER_AGENT, "Accept": "application/json"}
if self.api_key:
headers["X-Emby-Token"] = self.api_key
if body is not None:
data = json.dumps(body).encode("utf-8")
headers["Content-Type"] = "application/json"
request = urllib.request.Request(url, data=data, headers=headers, method=method)
try:
with urllib.request.urlopen(request, timeout=self.timeout) as response:
payload = response.read()
except urllib.error.HTTPError as exc:
raise JellyfinError(f"{method} {path} -> HTTP {exc.code}") from exc
except OSError as exc:
raise JellyfinError(f"{method} {path} -> {exc}") from exc
if not payload:
return None
try:
return json.loads(payload)
except json.JSONDecodeError:
return None
def public_info(self) -> dict:
"""Unauthenticated liveness check."""
return self._request("GET", "/System/Info/Public") or {}
def virtual_folders(self) -> list[dict]:
return self._request("GET", "/Library/VirtualFolders") or []
def find_library(self, path: Path | str) -> dict | None:
target = str(path).rstrip("/")
for folder in self.virtual_folders():
for location in folder.get("Locations") or []:
if str(location).rstrip("/") == target:
return folder
return None
def create_library(self, path: Path | str, name: str = LIBRARY_NAME) -> None:
"""Create the Shows library with all internet providers disabled."""
options = {
"EnableInternetProviders": False,
"SaveLocalMetadata": True,
"EnableRealtimeMonitor": False,
"EnableChapterImageExtraction": False,
"PathInfos": [{"Path": str(path)}],
"TypeOptions": [
{
"Type": item_type,
"MetadataFetchers": [],
"MetadataFetcherOrder": [],
"ImageFetchers": [],
"ImageFetcherOrder": [],
}
for item_type in _ITEM_TYPES
],
}
self._request(
"POST",
"/Library/VirtualFolders",
params={
"name": name,
"collectionType": COLLECTION_TYPE,
"paths": str(path),
"refreshLibrary": "false",
},
body={"LibraryOptions": options},
)
def refresh(self) -> None:
"""Trigger a library scan. Best effort — never fatal to the caller."""
try:
self._request("POST", "/Library/Refresh")
except JellyfinError as exc:
log.warning("jellyfin refresh failed: %s", exc)
def from_settings(settings) -> Jellyfin:
return Jellyfin(
settings.get_str("jellyfin_url"), settings.get_str("jellyfin_api_key")
)
youtube_automate/naming.py
+104
View File
@@ -0,0 +1,104 @@
"""Filename sanitisation and season/episode numbering.
Season is the upload year; episode is ``MMDD * 10 + ordinal_within_day``. That
scheme sorts correctly across a whole year (1 Jan is 1010, 31 Dec is 12310) and
leaves room for ten uploads per channel per day.
"""
from __future__ import annotations
import logging
import re
from datetime import date
log = logging.getLogger(__name__)
# Characters that are illegal or awkward in filenames on the platforms Jellyfin
# clients run on. Replaced with a space rather than deleted so that "A/B" reads
# as "A B" instead of collapsing into "AB".
FORBIDDEN = '/\\:*?"<>|'
MAX_TITLE_LEN = 120
MAX_ORDINAL = 9
_CONTROL = re.compile(r"[\x00-\x1f\x7f]")
_WHITESPACE = re.compile(r"\s+")
def sanitize_component(text: str, max_len: int = MAX_TITLE_LEN) -> str:
"""Make one path component safe, collapsing whitespace and truncating."""
text = _CONTROL.sub(" ", text or "")
text = "".join(" " if char in FORBIDDEN else char for char in text)
text = _WHITESPACE.sub(" ", text).strip()
text = truncate_on_word_boundary(text, max_len)
# A component may not begin or end with a dot or space: leading dots hide the
# file from Jellyfin, trailing ones confuse some clients.
text = text.strip(" .")
return text
def truncate_on_word_boundary(text: str, max_len: int) -> str:
if len(text) <= max_len:
return text
cut = text[:max_len]
space = cut.rfind(" ")
# Only honour the word boundary if it doesn't throw away most of the name.
if space > max_len * 0.6:
cut = cut[:space]
return cut.rstrip()
def channel_dir_name(title: str, channel_id: str) -> str:
"""Directory name for a channel. Stored once and never recomputed."""
name = sanitize_component(title)
return name or channel_id
def parse_upload_date(value: str | date) -> date:
"""Accept yt-dlp's YYYYMMDD, ISO YYYY-MM-DD, or a date."""
if isinstance(value, date):
return value
text = str(value).strip()
if len(text) == 8 and text.isdigit():
return date(int(text[:4]), int(text[4:6]), int(text[6:8]))
return date.fromisoformat(text[:10])
def season_for(upload_date: date) -> int:
return upload_date.year
def episode_base(upload_date: date) -> int:
"""First episode number available on this date."""
return (upload_date.month * 100 + upload_date.day) * 10
def episode_number(upload_date: date, ordinal: int) -> int:
"""Episode number for the nth upload on a given date (n starting at 0)."""
if ordinal > MAX_ORDINAL:
log.warning(
"more than %d uploads on %s; clamping ordinal %d",
MAX_ORDINAL + 1,
upload_date.isoformat(),
ordinal,
)
return episode_base(upload_date) + min(max(ordinal, 0), MAX_ORDINAL)
def episode_range(upload_date: date) -> tuple[int, int]:
"""Inclusive (low, high) episode numbers belonging to this date."""
base = episode_base(upload_date)
return base, base + MAX_ORDINAL
def season_dir_name(season: int) -> str:
return f"Season {season}"
def basename(channel_dir: str, season: int, episode: int, title: str, video_id: str) -> str:
"""Filename stem shared by the media file and every sidecar.
The [video_id] suffix guarantees uniqueness regardless of title collisions.
"""
safe_title = sanitize_component(title) or video_id
return f"{channel_dir} - S{season}E{episode} - {safe_title} [{video_id}]"
youtube_automate/nfo.py
+81
View File
@@ -0,0 +1,81 @@
"""Kodi-style NFO sidecars.
Video descriptions are hostile input — they contain ampersands, angle brackets,
emoji, ASCII art and control characters — so these are always built with
ElementTree's serialiser and never by string formatting.
"""
from __future__ import annotations
import re
import xml.etree.ElementTree as ET
from pathlib import Path
# XML 1.0 forbids most control characters outright; ElementTree will happily
# serialise them and produce a document no parser will read back. Written as a
# raw string so `re` interprets the escapes, not Python.
_ILLEGAL_XML = re.compile(
r"[^\x09\x0a\x0d\x20-퟿-�\U00010000-\U0010ffff]"
)
def clean_text(value: str | None) -> str:
return _ILLEGAL_XML.sub("", value or "")
def _child(parent: ET.Element, tag: str, text: str | None) -> ET.Element:
element = ET.SubElement(parent, tag)
element.text = clean_text(text)
return element
def _serialise(root: ET.Element) -> bytes:
ET.indent(root, space=" ")
return ET.tostring(root, encoding="utf-8", xml_declaration=True)
def tvshow_nfo(title: str, plot: str | None, channel_id: str) -> bytes:
root = ET.Element("tvshow")
_child(root, "title", title)
_child(root, "plot", plot)
_child(root, "studio", "YouTube")
unique = _child(root, "uniqueid", channel_id)
unique.set("type", "youtube")
unique.set("default", "true")
return _serialise(root)
def episode_nfo(
*,
title: str,
show_title: str,
season: int,
episode: int,
plot: str | None,
aired: str,
duration_seconds: int | None,
video_id: str,
) -> bytes:
root = ET.Element("episodedetails")
_child(root, "title", title)
_child(root, "showtitle", show_title)
_child(root, "season", str(season))
_child(root, "episode", str(episode))
_child(root, "plot", plot)
_child(root, "aired", aired)
if duration_seconds:
# Kodi/Jellyfin expect <runtime> in whole minutes.
_child(root, "runtime", str(max(1, round(duration_seconds / 60))))
_child(root, "studio", "YouTube")
unique = _child(root, "uniqueid", video_id)
unique.set("type", "youtube")
unique.set("default", "true")
return _serialise(root)
def write(path: Path, payload: bytes) -> None:
"""Write atomically so a crash never leaves Jellyfin a half-written NFO."""
path.parent.mkdir(parents=True, exist_ok=True)
temporary = path.with_name(path.name + ".tmp")
temporary.write_bytes(payload)
temporary.replace(path)
youtube_automate/reap.py
+150
View File
@@ -0,0 +1,150 @@
"""Retention: delete videos that have aged out.
Reap is a purely local operation. Jellyfin watch-state protection was considered
and declined (specs.md §15), so nothing here needs the Jellyfin API except the
refresh at the end — and that is best effort.
"""
from __future__ import annotations
import logging
import sqlite3
from datetime import timedelta
from pathlib import Path
from . import config, jellyfin, util, videos
from .settings import Settings
log = logging.getLogger(__name__)
def effective_retention(settings: Settings, channel_override: int | None) -> int:
return int(channel_override) if channel_override else settings.get_int("retention_days")
def candidates(conn: sqlite3.Connection, settings: Settings) -> list[sqlite3.Row]:
"""Downloaded videos past their channel's effective retention window."""
rows = conn.execute(
"SELECT v.*, c.dir_name, c.retention_days AS channel_retention, "
"c.title AS channel_title "
"FROM video v JOIN channel c ON c.id = v.channel_pk "
"WHERE v.state = ? ORDER BY v.upload_date ASC",
(videos.DOWNLOADED,),
).fetchall()
today = util.today()
due = []
for row in rows:
days = effective_retention(settings, row["channel_retention"])
if row["upload_date"] and row["upload_date"] < (today - timedelta(days=days)).isoformat():
due.append(row)
return due
def _delete_artefacts(rel_path: str) -> int:
"""Remove the media file and every sidecar sharing its stem."""
media = config.MEDIA_ROOT / rel_path
season_dir = media.parent
stem = media.stem # includes the [videoid], so it cannot collide
removed = 0
if season_dir.is_dir():
for path in season_dir.iterdir():
if path.name.startswith(stem):
try:
path.unlink()
removed += 1
except OSError as exc:
log.warning("could not delete %s: %s", path, exc)
return removed
def _prune_empty_season(season_dir: Path) -> None:
"""Remove the season directory once nothing is left in it.
The channel directory is deliberately kept even when it holds no seasons:
it still carries tvshow.nfo and the artwork, and deleting it would make an
active subscription vanish from Jellyfin and come back later.
"""
if not season_dir.is_dir() or season_dir == config.MEDIA_ROOT:
return
try:
next(season_dir.iterdir())
except StopIteration:
try:
season_dir.rmdir()
log.info("pruned empty season directory %s", season_dir)
except OSError as exc: # pragma: no cover
log.warning("could not prune %s: %s", season_dir, exc)
except OSError: # pragma: no cover
pass
def delete_video(conn: sqlite3.Connection, video: sqlite3.Row) -> bool:
"""Delete one video's files and leave a tombstone row."""
rel_path = video["rel_path"]
if not rel_path:
videos.mark_deleted(conn, video["video_id"])
return False
season_dir = (config.MEDIA_ROOT / rel_path).parent
removed = _delete_artefacts(rel_path)
_prune_empty_season(season_dir)
videos.mark_deleted(conn, video["video_id"])
log.info(
"reaped %s (%s, uploaded %s): %d file(s)",
video["video_id"],
video["channel_title"],
video["upload_date"],
removed,
)
return True
def disk_cap_evictions(
conn: sqlite3.Connection, settings: Settings
) -> list[sqlite3.Row]:
"""Oldest-first list of videos to evict to get back under the cap."""
cap_gb = settings.get_int("disk_cap_gb")
if cap_gb <= 0:
return []
cap_bytes = cap_gb * 1024**3
rows = conn.execute(
"SELECT v.*, c.dir_name, c.title AS channel_title "
"FROM video v JOIN channel c ON c.id = v.channel_pk "
"WHERE v.state = ? ORDER BY v.upload_date ASC",
(videos.DOWNLOADED,),
).fetchall()
total = sum(row["size_bytes"] or 0 for row in rows)
if total <= cap_bytes:
return []
evict = []
for row in rows:
if total <= cap_bytes:
break
evict.append(row)
total -= row["size_bytes"] or 0
log.info("disk cap %d GB exceeded; evicting %d video(s)", cap_gb, len(evict))
return evict
def run(conn: sqlite3.Connection, settings: Settings) -> dict:
"""Age-out pass plus optional disk-cap eviction."""
deleted = 0
for video in candidates(conn, settings):
if delete_video(conn, video):
deleted += 1
evicted = 0
for video in disk_cap_evictions(conn, settings):
if delete_video(conn, video):
evicted += 1
if deleted or evicted:
# Without this, Jellyfin shows ghost episodes until its own scheduled scan.
jellyfin.from_settings(settings).refresh()
return {"deleted": deleted, "evicted": evicted}
youtube_automate/runner.py
+96
View File
@@ -0,0 +1,96 @@
"""`run` orchestration: poll, then download, then reap — under a lock."""
from __future__ import annotations
import contextlib
import fcntl
import logging
import sqlite3
from pathlib import Path
from . import config, discovery, download, reap, util, videos
from .settings import Settings
log = logging.getLogger(__name__)
class AlreadyRunning(Exception):
pass
@contextlib.contextmanager
def exclusive_lock(path: Path | None = None):
"""Non-blocking flock. Raises AlreadyRunning if another run holds it.
The cron schedule is hourly and a large backfill can outlast that, so
overlapping runs are expected and must be a silent no-op rather than two
workers fighting over the same queue.
"""
path = path or config.LOCK_PATH
path.parent.mkdir(parents=True, exist_ok=True)
handle = path.open("w")
try:
try:
fcntl.flock(handle, fcntl.LOCK_EX | fcntl.LOCK_NB)
except OSError as exc:
raise AlreadyRunning(f"another run holds {path}") from exc
yield
finally:
with contextlib.suppress(OSError):
fcntl.flock(handle, fcntl.LOCK_UN)
handle.close()
def recover(conn: sqlite3.Connection) -> dict:
"""Undo the effects of a killed run before doing anything else."""
requeued = videos.recover_downloading(conn)
orphans = download.recover_orphans()
if requeued or orphans:
log.info(
"crash recovery: %d row(s) back to pending, %d orphan file(s) cleared",
requeued,
orphans,
)
return {"requeued": requeued, "orphans": orphans}
def run(
conn: sqlite3.Connection, settings: Settings, channel_pk: int | None = None
) -> dict:
"""One full cycle. Assumes the caller holds the lock."""
result = {"recovered": recover(conn)}
result["poll"] = discovery.poll_all(conn, settings, channel_pk)
try:
result["download"] = download.drain(conn, settings)
except RuntimeError as exc:
# The POT provider being down is loud and fatal for this run, but the
# poll results are still worth keeping.
log.error("%s", exc)
result["download"] = {"error": str(exc)}
return result
result["reap"] = reap.run(conn, settings)
settings.set("last_run_at", util.utcnow_iso())
return result
def summarise(result: dict) -> str:
poll = result.get("poll", {})
down = result.get("download", {})
reaped = result.get("reap", {})
parts = [
f"discovered={poll.get('queued', 0)}",
f"repaired={poll.get('repaired', 0)}",
f"poll_failures={poll.get('failed', 0)}",
f"downloaded={down.get(videos.DOWNLOADED, 0)}",
f"failed={down.get(videos.FAILED, 0)}",
f"skipped={down.get(videos.SKIPPED_SHORT, 0) + down.get(videos.SKIPPED_LIVE, 0)}",
f"deferred={down.get(videos.DEFERRED, 0)}",
f"reaped={reaped.get('deleted', 0)}",
f"evicted={reaped.get('evicted', 0)}",
]
if "error" in down:
parts.append(f"ERROR={down['error']}")
return " ".join(parts)
youtube_automate/settings.py
+144
View File
@@ -0,0 +1,144 @@
"""Typed settings accessors backed by the `setting` key/value table.
A missing key must never crash anything, so every read falls back to the default
and every malformed stored value falls back to the default too.
"""
from __future__ import annotations
import sqlite3
from urllib.parse import urlparse
DEFAULTS: dict[str, str] = {
"retention_days": "14",
"backfill_days": "7",
"max_height": "720",
"min_duration_seconds": "120",
"sponsorblock_mark": "true",
"write_subs": "true",
"sub_langs": "en.*",
"jellyfin_url": "http://127.0.0.1:8096",
"jellyfin_api_key": "",
"pot_provider_url": "http://127.0.0.1:4416",
"max_attempts": "5",
"disk_cap_gb": "0",
}
# Editable through the settings form. Everything else in the table is internal.
EDITABLE = tuple(DEFAULTS)
# Never rendered, never settable through the web form.
SECRET_KEYS = ("admin_password_hash", "session_secret")
# Shown as a masked value rather than plaintext.
MASKED_KEYS = ("jellyfin_api_key",)
_INT_KEYS = (
"retention_days",
"backfill_days",
"max_height",
"min_duration_seconds",
"max_attempts",
"disk_cap_gb",
)
_BOOL_KEYS = ("sponsorblock_mark", "write_subs")
_URL_KEYS = ("jellyfin_url", "pot_provider_url")
_TRUE = {"1", "true", "yes", "on"}
_FALSE = {"0", "false", "no", "off", ""}
class Settings:
def __init__(self, conn: sqlite3.Connection):
self.conn = conn
def raw(self, key: str) -> str:
row = self.conn.execute(
"SELECT value FROM setting WHERE key = ?", (key,)
).fetchone()
if row is None:
return DEFAULTS.get(key, "")
return row["value"]
def get_str(self, key: str) -> str:
return self.raw(key)
def get_int(self, key: str) -> int:
try:
return int(self.raw(key))
except (TypeError, ValueError):
return int(DEFAULTS.get(key, "0") or 0)
def get_bool(self, key: str) -> bool:
value = self.raw(key).strip().lower()
if value in _TRUE:
return True
if value in _FALSE:
return False
return DEFAULTS.get(key, "false").lower() in _TRUE
def set(self, key: str, value: str) -> None:
with self.conn:
self.conn.execute(
"INSERT INTO setting (key, value) VALUES (?, ?) "
"ON CONFLICT(key) DO UPDATE SET value = excluded.value",
(key, str(value)),
)
def all_editable(self) -> dict[str, str]:
return {key: self.raw(key) for key in EDITABLE}
def validate(key: str, value: str) -> tuple[bool, str]:
"""Validate one submitted setting.
Returns (ok, message). On failure the caller re-renders the form with the
message inline rather than raising.
"""
value = value.strip()
if key in _INT_KEYS:
try:
number = int(value)
except ValueError:
return False, "must be a whole number"
if number < 0:
return False, "must be zero or greater"
if key == "retention_days" and number < 1:
return False, "must be at least 1 day"
if key == "max_height" and number < 144:
return False, "must be at least 144"
return True, ""
if key in _BOOL_KEYS:
if value.lower() not in _TRUE | _FALSE:
return False, "must be true or false"
return True, ""
if key in _URL_KEYS:
parsed = urlparse(value)
if parsed.scheme not in ("http", "https") or not parsed.netloc:
return False, "must be a http:// or https:// URL"
return True, ""
if key == "sub_langs":
if not value:
return False, "must not be empty"
return True, ""
if key == "jellyfin_api_key":
return True, ""
return key in DEFAULTS, "unknown setting"
def validate_all(submitted: dict[str, str]) -> dict[str, str]:
"""Return {key: error} for everything that failed validation."""
errors: dict[str, str] = {}
for key, value in submitted.items():
if key not in EDITABLE:
continue
ok, message = validate(key, value)
if not ok:
errors[key] = message
return errors
youtube_automate/util.py
+45
View File
@@ -0,0 +1,45 @@
"""Small shared helpers."""
from __future__ import annotations
import logging
import os
from datetime import date, datetime, timezone
from . import config
def utcnow() -> datetime:
return datetime.now(timezone.utc)
def utcnow_iso() -> str:
return utcnow().replace(microsecond=0).isoformat()
def today() -> date:
return utcnow().date()
def apply_umask() -> None:
"""Ensure files land group-writable so Jellyfin's group can read them."""
os.umask(config.UMASK)
def setup_logging(verbose: bool = False) -> None:
logging.basicConfig(
level=logging.DEBUG if verbose else logging.INFO,
format="%(asctime)s %(levelname)-7s %(name)s: %(message)s",
datefmt="%Y-%m-%d %H:%M:%S",
)
# yt-dlp and urllib are noisy at debug level and we drive them deliberately.
logging.getLogger("urllib3").setLevel(logging.WARNING)
def human_bytes(value: int | None) -> str:
size = float(value or 0)
for unit in ("B", "KB", "MB", "GB", "TB"):
if size < 1024 or unit == "TB":
return f"{size:.0f} {unit}" if unit == "B" else f"{size:.1f} {unit}"
size /= 1024
return f"{size:.1f} TB"
youtube_automate/videos.py
+209
View File
@@ -0,0 +1,209 @@
"""Video row helpers and the state machine.
States (specs.md §8):
pending discovered, queued
downloading claimed by a worker; recovered to pending on startup
downloaded on disk, rel_path set
deleted aged out — tombstone, never re-downloaded
deferred premiere/upcoming, retried by later polls
skipped_short below min_duration_seconds; repaired if later seen in UULF
skipped_live livestream, never retried
skipped_old already outside the window when discovered
failed download error, retried up to max_attempts
"""
from __future__ import annotations
import sqlite3
from datetime import date
from . import naming, util
PENDING = "pending"
DOWNLOADING = "downloading"
DOWNLOADED = "downloaded"
DELETED = "deleted"
DEFERRED = "deferred"
SKIPPED_SHORT = "skipped_short"
SKIPPED_LIVE = "skipped_live"
SKIPPED_OLD = "skipped_old"
FAILED = "failed"
# States that mean "we have made a final negative decision about this video".
# A deleted row is a tombstone and must never be resurrected by any code path.
TERMINAL = (DELETED, SKIPPED_LIVE, SKIPPED_OLD)
SOURCE_UULF = "uulf_feed"
SOURCE_UC = "uc_feed"
SOURCE_BACKFILL = "backfill"
def get(conn: sqlite3.Connection, video_id: str) -> sqlite3.Row | None:
return conn.execute(
"SELECT * FROM video WHERE video_id = ?", (video_id,)
).fetchone()
def exists(conn: sqlite3.Connection, video_id: str) -> bool:
return get(conn, video_id) is not None
def insert(
conn: sqlite3.Connection,
*,
channel_pk: int,
video_id: str,
title: str,
upload_date: str | None,
state: str,
discovery_source: str,
duration: int | None = None,
) -> None:
with conn:
conn.execute(
"INSERT OR IGNORE INTO video "
"(video_id, channel_pk, title, upload_date, duration, state, "
" discovery_source, discovered_at) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
(
video_id,
channel_pk,
title,
upload_date,
duration,
state,
discovery_source,
util.utcnow_iso(),
),
)
def set_state(
conn: sqlite3.Connection, video_id: str, state: str, *, error: str | None = None
) -> None:
with conn:
conn.execute(
"UPDATE video SET state = ?, last_error = ? WHERE video_id = ?",
(state, error, video_id),
)
def record_failure(conn: sqlite3.Connection, video_id: str, error: str, max_attempts: int) -> str:
"""Bump attempts and decide whether to keep retrying."""
with conn:
conn.execute(
"UPDATE video SET attempts = attempts + 1, last_error = ? WHERE video_id = ?",
(error[:500], video_id),
)
row = conn.execute(
"SELECT attempts FROM video WHERE video_id = ?", (video_id,)
).fetchone()
attempts = row["attempts"] if row else max_attempts
state = FAILED
conn.execute(
"UPDATE video SET state = ? WHERE video_id = ?", (state, video_id)
)
return "exhausted" if attempts >= max_attempts else state
def mark_downloaded(
conn: sqlite3.Connection,
video_id: str,
*,
rel_path: str,
size_bytes: int,
season: int,
episode: int,
upload_date: str,
duration: int | None,
title: str,
) -> None:
with conn:
conn.execute(
"UPDATE video SET state = ?, rel_path = ?, size_bytes = ?, season = ?, "
"episode = ?, upload_date = ?, duration = ?, title = ?, "
"downloaded_at = ?, last_error = NULL WHERE video_id = ?",
(
DOWNLOADED,
rel_path,
size_bytes,
season,
episode,
upload_date,
duration,
title,
util.utcnow_iso(),
video_id,
),
)
def mark_deleted(conn: sqlite3.Connection, video_id: str) -> None:
"""Keep the row — it is the tombstone that prevents re-download."""
with conn:
conn.execute(
"UPDATE video SET state = ?, rel_path = NULL, size_bytes = NULL, "
"deleted_at = ? WHERE video_id = ?",
(DELETED, util.utcnow_iso(), video_id),
)
def next_episode(
conn: sqlite3.Connection, channel_pk: int, upload_date: date, video_id: str
) -> tuple[int, int]:
"""Assign (season, episode) for a video.
The ordinal is computed against what is already in the database for this
channel and date — never against the current batch — so it stays stable
across runs and across crashes mid-batch.
"""
season = naming.season_for(upload_date)
low, high = naming.episode_range(upload_date)
row = conn.execute(
"SELECT MAX(episode) AS top FROM video "
"WHERE channel_pk = ? AND season = ? AND episode BETWEEN ? AND ? "
"AND video_id != ?",
(channel_pk, season, low, high, video_id),
).fetchone()
top = row["top"] if row and row["top"] is not None else None
if top is None:
return season, low
if top >= high:
# More than ten uploads in a day; naming.episode_number logs the clamp.
return season, high
return season, top + 1
def claim_pending(
conn: sqlite3.Connection, max_attempts: int, limit: int | None = None
) -> list[sqlite3.Row]:
"""Queue: pending rows, plus failed rows that still have attempts left."""
sql = (
"SELECT v.*, c.dir_name, c.title AS channel_title, c.channel_id "
"FROM video v JOIN channel c ON c.id = v.channel_pk "
"WHERE (v.state = ? OR (v.state = ? AND v.attempts < ?)) "
"ORDER BY v.upload_date ASC, v.discovered_at ASC"
)
params: list = [PENDING, FAILED, max_attempts]
if limit:
sql += " LIMIT ?"
params.append(limit)
return conn.execute(sql, params).fetchall()
def recover_downloading(conn: sqlite3.Connection) -> int:
"""Crash recovery: anything left claimed goes back on the queue."""
with conn:
cursor = conn.execute(
"UPDATE video SET state = ? WHERE state = ?", (PENDING, DOWNLOADING)
)
return cursor.rowcount
def queue_depth(conn: sqlite3.Connection) -> int:
return conn.execute(
"SELECT COUNT(*) FROM video WHERE state IN (?, ?, ?)",
(PENDING, FAILED, DOWNLOADING),
).fetchone()[0]
youtube_automate/web/__init__.py
+1
View File
@@ -0,0 +1 @@
"""Admin web UI."""
youtube_automate/web/auth.py
+173
View File
@@ -0,0 +1,173 @@
"""Password hashing, session cookies and CSRF tokens.
The admin UI is publicly reachable over HTTPS, so this has to be real. The design
goal from specs.md §11 is "log in once per device, effectively never again",
which means a long-lived signed cookie rather than HTTP basic auth.
"""
from __future__ import annotations
import base64
import hashlib
import hmac
import json
import secrets
import time
SCRYPT_N = 2**14
SCRYPT_R = 8
SCRYPT_P = 1
DKLEN = 32
SESSION_MAX_AGE = 365 * 24 * 3600 # one year
COOKIE_NAME = "yta_session"
# Login throttling: after this many consecutive failures from one address, refuse
# for LOCKOUT_SECONDS regardless of whether the password is right.
MAX_FAILURES = 5
LOCKOUT_SECONDS = 60
def _b64(raw: bytes) -> str:
return base64.urlsafe_b64encode(raw).decode("ascii").rstrip("=")
def _unb64(text: str) -> bytes:
padding = "=" * (-len(text) % 4)
return base64.urlsafe_b64decode(text + padding)
# --------------------------------------------------------------------------
# passwords
def hash_password(password: str, *, salt: bytes | None = None) -> str:
salt = salt if salt is not None else secrets.token_bytes(16)
derived = hashlib.scrypt(
password.encode("utf-8"),
salt=salt,
n=SCRYPT_N,
r=SCRYPT_R,
p=SCRYPT_P,
dklen=DKLEN,
)
return f"scrypt${SCRYPT_N}${SCRYPT_R}${SCRYPT_P}${_b64(salt)}${_b64(derived)}"
def verify_password(stored: str, password: str) -> bool:
if not stored:
return False
try:
scheme, n, r, p, salt_b64, hash_b64 = stored.split("$")
if scheme != "scrypt":
return False
derived = hashlib.scrypt(
password.encode("utf-8"),
salt=_unb64(salt_b64),
n=int(n),
r=int(r),
p=int(p),
dklen=len(_unb64(hash_b64)),
)
except (ValueError, TypeError):
return False
return hmac.compare_digest(derived, _unb64(hash_b64))
# --------------------------------------------------------------------------
# sessions
def new_secret() -> str:
return _b64(secrets.token_bytes(32))
def _sign(secret: str, payload: bytes) -> str:
return _b64(hmac.new(secret.encode("utf-8"), payload, hashlib.sha256).digest())
def issue_session(secret: str, *, issued_at: float | None = None) -> str:
payload = json.dumps(
{"iat": int(issued_at if issued_at is not None else time.time())},
separators=(",", ":"),
).encode("utf-8")
return f"{_b64(payload)}.{_sign(secret, payload)}"
def verify_session(secret: str, token: str, *, now: float | None = None) -> bool:
if not token or not secret:
return False
try:
payload_b64, signature = token.split(".", 1)
payload = _unb64(payload_b64)
except (ValueError, TypeError):
return False
if not hmac.compare_digest(_sign(secret, payload), signature):
return False
try:
issued_at = int(json.loads(payload)["iat"])
except (ValueError, KeyError, TypeError):
return False
age = (now if now is not None else time.time()) - issued_at
return 0 <= age <= SESSION_MAX_AGE
def cookie_header(token: str, *, secure: bool = True) -> str:
parts = [
f"{COOKIE_NAME}={token}",
"Path=/",
"HttpOnly",
"SameSite=Lax",
f"Max-Age={SESSION_MAX_AGE}",
]
if secure:
parts.insert(2, "Secure")
return "; ".join(parts)
def clear_cookie_header() -> str:
return f"{COOKIE_NAME}=; Path=/; HttpOnly; SameSite=Lax; Max-Age=0"
# --------------------------------------------------------------------------
# CSRF
def csrf_token(secret: str, session_token: str) -> str:
return _sign(secret, b"csrf:" + session_token.encode("utf-8"))
def verify_csrf(secret: str, session_token: str, submitted: str) -> bool:
if not submitted:
return False
return hmac.compare_digest(csrf_token(secret, session_token), submitted)
# --------------------------------------------------------------------------
# throttling
class LoginThrottle:
"""In-memory consecutive-failure tracker keyed by remote address."""
def __init__(self, max_failures: int = MAX_FAILURES, lockout: int = LOCKOUT_SECONDS):
self.max_failures = max_failures
self.lockout = lockout
self._state: dict[str, tuple[int, float]] = {}
def locked(self, key: str, *, now: float | None = None) -> bool:
failures, last = self._state.get(key, (0, 0.0))
if failures < self.max_failures:
return False
elapsed = (now if now is not None else time.time()) - last
if elapsed >= self.lockout:
self._state.pop(key, None)
return False
return True
def record_failure(self, key: str, *, now: float | None = None) -> None:
failures, _ = self._state.get(key, (0, 0.0))
self._state[key] = (failures + 1, now if now is not None else time.time())
def record_success(self, key: str) -> None:
self._state.pop(key, None)
youtube_automate/web/server.py
+400
View File
@@ -0,0 +1,400 @@
"""The admin HTTP server.
Stdlib only. Binds to localhost; nginx terminates TLS in front of it at
tube.jihakuz.xyz. Because that hostname is public, this carries real
authentication, CSRF tokens on every state-changing request, and login
throttling.
"""
from __future__ import annotations
import http.cookies
import json
import logging
import subprocess
import sys
import threading
import urllib.parse
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from .. import channels, config, db, discovery, jellyfin, util, videos, ytdlp
from ..settings import EDITABLE, MASKED_KEYS, Settings, validate_all
from . import auth, templates
log = logging.getLogger(__name__)
MAX_BODY = 64 * 1024
class AdminServer(ThreadingHTTPServer):
daemon_threads = True
allow_reuse_address = True
def __init__(self, address, handler, *, secure_cookies: bool = True):
super().__init__(address, handler)
self.throttle = auth.LoginThrottle()
self.secure_cookies = secure_cookies
self.db_lock = threading.Lock()
class Handler(BaseHTTPRequestHandler):
server_version = "youtube-automate"
protocol_version = "HTTP/1.1"
# ---------------------------------------------------------------- utils
def log_message(self, fmt, *args): # noqa: A003 - stdlib signature
log.debug("%s - %s", self.client_address[0], fmt % args)
def _client_key(self) -> str:
"""Real client address, since we always sit behind nginx."""
forwarded = self.headers.get("X-Forwarded-For", "")
if forwarded:
return forwarded.split(",")[0].strip()
return self.client_address[0]
def _send(self, status: int, body: bytes, headers: dict | None = None) -> None:
self.send_response(status)
self.send_header("Content-Type", "text/html; charset=utf-8")
self.send_header("Content-Length", str(len(body)))
self.send_header("X-Content-Type-Options", "nosniff")
self.send_header("Referrer-Policy", "same-origin")
self.send_header("X-Frame-Options", "DENY")
for key, value in (headers or {}).items():
self.send_header(key, value)
self.end_headers()
self.wfile.write(body)
def _redirect(self, location: str, headers: dict | None = None) -> None:
combined = {"Location": location}
combined.update(headers or {})
self._send(303, b"", combined)
def _json(self, status: int, payload: dict) -> None:
body = json.dumps(payload, indent=2).encode("utf-8")
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def _form(self) -> dict[str, str]:
length = int(self.headers.get("Content-Length") or 0)
if length <= 0 or length > MAX_BODY:
return {}
raw = self.rfile.read(length).decode("utf-8", "replace")
return {
key: values[-1]
for key, values in urllib.parse.parse_qs(raw, keep_blank_values=True).items()
}
def _cookie_token(self) -> str:
header = self.headers.get("Cookie")
if not header:
return ""
jar = http.cookies.SimpleCookie()
try:
jar.load(header)
except http.cookies.CookieError:
return ""
morsel = jar.get(auth.COOKIE_NAME)
return morsel.value if morsel else ""
# ------------------------------------------------------------- session
def _open(self):
conn = db.connect()
return conn, Settings(conn)
def _secret(self, settings: Settings) -> str:
secret = settings.raw("session_secret")
if not secret:
secret = auth.new_secret()
settings.set("session_secret", secret)
return secret
def _authenticated(self, settings: Settings) -> str | None:
"""Return the session token if the request is signed in, else None."""
token = self._cookie_token()
if token and auth.verify_session(self._secret(settings), token):
return token
return None
def _check_csrf(self, settings: Settings, token: str, form: dict) -> bool:
return auth.verify_csrf(self._secret(settings), token, form.get("csrf", ""))
# ---------------------------------------------------------------- GET
def do_GET(self) -> None: # noqa: N802 - stdlib signature
path = urllib.parse.urlparse(self.path).path.rstrip("/") or "/"
conn, settings = self._open()
try:
token = self._authenticated(settings)
# /health was specced as unauthenticated back when the UI was going
# to be tailnet-only. On a public hostname it is gratuitous
# fingerprinting surface (yt-dlp version, queue depth), so it needs
# a session like everything else.
if path == "/health":
if not token:
return self._json(401, {"error": "authentication required"})
return self._health(conn, settings)
if path == "/login":
if token:
return self._redirect("/")
return self._send(200, templates.login_page(self._login_hint(settings)))
if not token:
return self._redirect("/login")
if path == "/":
return self._send(200, self._render_index(conn, settings, token))
return self._send(404, templates.page("Not found", "<h1>Not found</h1>"))
finally:
conn.close()
def _login_hint(self, settings: Settings) -> str | None:
if not settings.raw("admin_password_hash"):
return "No password is set yet. Run `youtube-automate set-password` on susan."
return None
def _health(self, conn, settings: Settings) -> None:
try:
ytdlp_version = ytdlp.version()
except Exception as exc: # noqa: BLE001
ytdlp_version = f"error: {exc}"
try:
ytdlp.pot_provider_ping(settings.get_str("pot_provider_url"))
pot_up = True
except Exception: # noqa: BLE001
pot_up = False
self._json(
200,
{
"yt_dlp_version": ytdlp_version,
"pot_provider_up": pot_up,
"last_run_at": settings.raw("last_run_at") or None,
"queue_depth": videos.queue_depth(conn),
"channels": len(channels.all_channels(conn)),
},
)
# --------------------------------------------------------------- POST
def do_POST(self) -> None: # noqa: N802 - stdlib signature
path = urllib.parse.urlparse(self.path).path.rstrip("/") or "/"
conn, settings = self._open()
try:
form = self._form()
if path == "/login":
return self._login(settings, form)
token = self._authenticated(settings)
if not token:
return self._redirect("/login")
if not self._check_csrf(settings, token, form):
log.warning("CSRF check failed for %s from %s", path, self._client_key())
return self._send(
400,
templates.page(
"Bad request",
"<h1>Bad request</h1><p>Invalid form token. "
'<a href="/">Go back</a> and try again.</p>',
),
)
if path == "/logout":
return self._redirect("/login", {"Set-Cookie": auth.clear_cookie_header()})
if path == "/channels":
return self._add_channel(conn, settings, token, form)
if path == "/settings":
return self._save_settings(conn, settings, token, form)
parts = path.strip("/").split("/")
if len(parts) == 3 and parts[0] == "channels" and parts[1].isdigit():
pk = int(parts[1])
if parts[2] == "delete":
return self._delete_channel(conn, settings, pk)
if parts[2] == "retention":
return self._set_retention(conn, pk, form)
if parts[2] == "rescan":
return self._rescan(conn, settings, pk)
return self._send(404, templates.page("Not found", "<h1>Not found</h1>"))
finally:
conn.close()
def _login(self, settings: Settings, form: dict) -> None:
key = self._client_key()
if self.server.throttle.locked(key):
return self._send(
429, templates.login_page("Too many attempts. Wait a minute.")
)
stored = settings.raw("admin_password_hash")
if stored and auth.verify_password(stored, form.get("password", "")):
self.server.throttle.record_success(key)
token = auth.issue_session(self._secret(settings))
return self._redirect(
"/",
{
"Set-Cookie": auth.cookie_header(
token, secure=self.server.secure_cookies
)
},
)
self.server.throttle.record_failure(key)
log.warning("failed login from %s", key)
return self._send(
401, templates.login_page(self._login_hint(settings) or "Wrong password.")
)
# ------------------------------------------------------------ actions
def _add_channel(self, conn, settings: Settings, token: str, form: dict) -> None:
url = (form.get("url") or "").strip()
try:
row = channels.subscribe(conn, settings, url)
except channels.ResolutionError as exc:
body = self._render_index(conn, settings, token, add_error=str(exc))
return self._send(400, body)
self._spawn_backfill(row["id"])
return self._redirect("/")
def _spawn_backfill(self, channel_pk: int) -> None:
"""Kick off discovery immediately rather than waiting for the hourly cron."""
try:
subprocess.Popen( # noqa: S603
[sys.executable, "-m", "youtube_automate", "run", "--channel",
str(channel_pk)],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
start_new_session=True,
)
except OSError as exc: # pragma: no cover
log.warning("could not spawn backfill for channel %d: %s", channel_pk, exc)
def _delete_channel(self, conn, settings: Settings, pk: int) -> None:
try:
channels.unsubscribe(conn, pk)
except LookupError:
return self._send(404, templates.page("Not found", "<h1>Not found</h1>"))
jellyfin.from_settings(settings).refresh()
return self._redirect("/")
def _set_retention(self, conn, pk: int, form: dict) -> None:
raw = (form.get("days") or "").strip()
value: int | None
if not raw:
value = None
else:
try:
value = max(1, int(raw))
except ValueError:
return self._redirect("/")
with conn:
conn.execute(
"UPDATE channel SET retention_days = ? WHERE id = ?", (value, pk)
)
return self._redirect("/")
def _rescan(self, conn, settings: Settings, pk: int) -> None:
channel = channels.get(conn, pk)
if channel is None:
return self._send(404, templates.page("Not found", "<h1>Not found</h1>"))
discovery.rescan_channel(conn, settings, channel)
self._spawn_backfill(pk)
return self._redirect("/")
def _save_settings(self, conn, settings: Settings, token: str, form: dict) -> None:
submitted = {key: form.get(key, "") for key in EDITABLE if key in form}
# A blank masked field means "keep what is stored", not "clear it".
for key in MASKED_KEYS:
if key in submitted and not submitted[key].strip():
submitted.pop(key)
errors = validate_all(submitted)
if errors:
body = self._render_index(
conn, settings, token, settings_errors=errors, submitted=submitted
)
return self._send(400, body)
for key, value in submitted.items():
settings.set(key, value.strip())
return self._redirect("/")
# ------------------------------------------------------------- render
def _render_index(
self,
conn,
settings: Settings,
token: str,
*,
add_error: str | None = None,
settings_errors: dict | None = None,
submitted: dict | None = None,
) -> bytes:
global_retention = settings.get_int("retention_days")
rows = []
for channel in channels.all_channels(conn):
stats = conn.execute(
"SELECT COUNT(*) AS n, COALESCE(SUM(size_bytes), 0) AS bytes, "
"MAX(upload_date) AS latest FROM video "
"WHERE channel_pk = ? AND state = ?",
(channel["id"], videos.DOWNLOADED),
).fetchone()
latest_any = conn.execute(
"SELECT MAX(upload_date) AS latest FROM video WHERE channel_pk = ?",
(channel["id"],),
).fetchone()
rows.append(
{
"id": channel["id"],
"title": channel["title"],
"handle": channel["handle"],
"channel_id": channel["channel_id"],
"retention_days": channel["retention_days"],
"global_retention": global_retention,
"last_polled_at": channel["last_polled_at"],
"last_poll_ok": channel["last_poll_ok"],
"consecutive_poll_failures": channel["consecutive_poll_failures"],
"downloaded": stats["n"],
"bytes": stats["bytes"],
"latest": latest_any["latest"],
}
)
values = settings.all_editable()
if submitted:
values.update(submitted)
return templates.index_page(
channels=rows,
settings_values=values,
settings_errors=settings_errors or {},
csrf=auth.csrf_token(self._secret(settings), token),
add_error=add_error,
queue_depth=videos.queue_depth(conn),
)
def serve(host: str = "127.0.0.1", port: int = 8085, *, secure_cookies: bool = True) -> None:
util.apply_umask()
server = AdminServer((host, port), Handler, secure_cookies=secure_cookies)
log.info("admin server listening on http://%s:%d", host, port)
try:
server.serve_forever()
except KeyboardInterrupt: # pragma: no cover
pass
finally:
server.server_close()
youtube_automate/web/templates.py
+246
View File
@@ -0,0 +1,246 @@
"""Server-rendered HTML. One embedded stylesheet, no JavaScript beyond a
confirm() on the destructive buttons."""
from __future__ import annotations
import html
from datetime import date
from .. import util
from ..settings import DEFAULTS, EDITABLE, MASKED_KEYS
STYLE = """
:root {
--bg: #14161a; --panel: #1c1f26; --line: #2c313b; --text: #e6e8ec;
--muted: #99a0ae; --accent: #6aa9ff; --warn: #ffb454; --bad: #ff6b6b;
--good: #6ade9b;
}
* { box-sizing: border-box; }
body { margin: 0; background: var(--bg); color: var(--text);
font: 15px/1.5 ui-sans-serif, system-ui, -apple-system, "Segoe UI", sans-serif; }
main { max-width: 62rem; margin: 0 auto; padding: 1.5rem 1rem 4rem; }
h1 { font-size: 1.4rem; margin: 0; }
h2 { font-size: 1.05rem; margin: 2rem 0 .75rem; color: var(--muted);
text-transform: uppercase; letter-spacing: .06em; }
header { display: flex; align-items: baseline; justify-content: space-between;
gap: 1rem; border-bottom: 1px solid var(--line); padding-bottom: .75rem; }
header .sub { color: var(--muted); font-size: .85rem; }
a { color: var(--accent); }
.panel { background: var(--panel); border: 1px solid var(--line);
border-radius: 10px; padding: 1rem; }
table { width: 100%; border-collapse: collapse; }
th, td { text-align: left; padding: .55rem .5rem; border-bottom: 1px solid var(--line);
vertical-align: middle; }
th { color: var(--muted); font-weight: 600; font-size: .78rem;
text-transform: uppercase; letter-spacing: .05em; }
td.num { text-align: right; font-variant-numeric: tabular-nums; }
.muted { color: var(--muted); }
.badge { display: inline-block; padding: .1rem .45rem; border-radius: 999px;
font-size: .75rem; border: 1px solid currentColor; }
.badge.warn { color: var(--warn); }
.badge.good { color: var(--good); }
.badge.bad { color: var(--bad); }
input[type=text], input[type=password], input[type=number], select {
background: #12141a; color: var(--text); border: 1px solid var(--line);
border-radius: 7px; padding: .45rem .55rem; font: inherit; width: 100%; }
button { background: var(--accent); color: #0b1017; border: 0; border-radius: 7px;
padding: .5rem .9rem; font: inherit; font-weight: 600; cursor: pointer; }
button.secondary { background: #2b3140; color: var(--text); }
button.danger { background: transparent; color: var(--bad);
border: 1px solid var(--bad); font-weight: 500; padding: .3rem .6rem; }
button.link { background: transparent; color: var(--accent); border: 0;
padding: .3rem .4rem; font-weight: 500; }
form.inline { display: inline; }
.row { display: flex; gap: .6rem; align-items: center; }
.grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(15rem, 1fr));
gap: .85rem; }
label { display: block; font-size: .82rem; color: var(--muted);
margin-bottom: .25rem; }
.field { margin-bottom: .3rem; }
.error { color: var(--bad); font-size: .8rem; margin-top: .2rem; }
.flash { border-radius: 8px; padding: .6rem .8rem; margin-bottom: 1rem;
border: 1px solid; }
.flash.ok { color: var(--good); border-color: var(--good); }
.flash.bad { color: var(--bad); border-color: var(--bad); }
.login { max-width: 21rem; margin: 6rem auto; }
footer { margin-top: 2.5rem; color: var(--muted); font-size: .8rem; }
@media (max-width: 40rem) {
th.hide, td.hide { display: none; }
}
"""
def _e(value) -> str:
return html.escape("" if value is None else str(value), quote=True)
def page(title: str, body: str) -> bytes:
return f"""<!doctype html>
<html lang="en"><head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>{_e(title)}</title>
<style>{STYLE}</style>
</head><body><main>{body}</main></body></html>""".encode("utf-8")
def login_page(error: str | None = None) -> bytes:
alert = f'<div class="flash bad">{_e(error)}</div>' if error else ""
body = f"""
<div class="login">
<h1>youtube-automate</h1>
<p class="muted">Sign in to manage subscriptions.</p>
{alert}
<form method="post" action="/login" class="panel">
<div class="field">
<label for="password">Password</label>
<input type="password" id="password" name="password" autofocus
autocomplete="current-password">
</div>
<div style="margin-top:.8rem"><button type="submit">Sign in</button></div>
</form>
<footer>You will stay signed in on this device for a year.</footer>
</div>"""
return page("Sign in — youtube-automate", body)
def _channel_row(channel: dict, csrf: str) -> str:
failures = channel["consecutive_poll_failures"]
if failures > 2:
badge = f'<span class="badge bad">{failures} failed polls</span>'
elif channel["last_poll_ok"] == 0:
badge = '<span class="badge warn">last poll failed</span>'
else:
badge = ""
retention = channel["retention_days"]
retention_value = "" if retention is None else str(retention)
placeholder = f"default ({channel['global_retention']})"
return f"""
<tr>
<td>
<strong>{_e(channel['title'])}</strong> {badge}<br>
<span class="muted">{_e(channel['handle'] or channel['channel_id'])}</span>
</td>
<td class="num">{channel['downloaded']}</td>
<td class="num hide">{_e(util.human_bytes(channel['bytes']))}</td>
<td class="hide muted">{_e(channel['latest'] or '—')}</td>
<td class="hide muted">{_e(channel['last_polled_at'] or 'never')}</td>
<td>
<form method="post" action="/channels/{channel['id']}/retention" class="row">
<input type="hidden" name="csrf" value="{_e(csrf)}">
<input type="number" name="days" min="1" style="width:6.5rem"
value="{_e(retention_value)}" placeholder="{_e(placeholder)}">
<button class="link" type="submit">save</button>
</form>
</td>
<td>
<form method="post" action="/channels/{channel['id']}/rescan" class="inline">
<input type="hidden" name="csrf" value="{_e(csrf)}">
<button class="link" type="submit"
title="Re-queue videos previously skipped as too old that the current
retention window now covers">rescan</button>
</form>
<form method="post" action="/channels/{channel['id']}/delete" class="inline"
onsubmit="return confirm('Permanently delete {_e(channel['title'])} and every video downloaded for it? This cannot be undone.');">
<input type="hidden" name="csrf" value="{_e(csrf)}">
<button class="danger" type="submit">remove</button>
</form>
</td>
</tr>"""
def _settings_form(values: dict, errors: dict, csrf: str) -> str:
fields = []
for key in EDITABLE:
value = values.get(key, DEFAULTS[key])
error = errors.get(key)
if key in MASKED_KEYS and value:
shown, placeholder = "", "stored — leave blank to keep"
else:
shown, placeholder = value, ""
input_type = "password" if key in MASKED_KEYS else "text"
fields.append(
f"""<div class="field">
<label for="{_e(key)}">{_e(key.replace('_', ' '))}</label>
<input type="{input_type}" id="{_e(key)}" name="{_e(key)}"
value="{_e(shown)}" placeholder="{_e(placeholder)}" autocomplete="off">
{f'<div class="error">{_e(error)}</div>' if error else ''}
</div>"""
)
return f"""
<form method="post" action="/settings" class="panel">
<input type="hidden" name="csrf" value="{_e(csrf)}">
<div class="grid">{''.join(fields)}</div>
<div style="margin-top:1rem"><button type="submit">Save settings</button></div>
</form>"""
def index_page(
*,
channels: list[dict],
settings_values: dict,
settings_errors: dict,
csrf: str,
flash: tuple[str, str] | None = None,
add_error: str | None = None,
queue_depth: int = 0,
) -> bytes:
flash_html = ""
if flash:
kind, message = flash
flash_html = f'<div class="flash {kind}">{_e(message)}</div>'
if channels:
rows = "".join(_channel_row(channel, csrf) for channel in channels)
table = f"""
<div class="panel">
<table>
<thead><tr>
<th>Channel</th><th class="num">On disk</th><th class="num hide">Size</th>
<th class="hide">Latest upload</th><th class="hide">Last poll</th>
<th>Retention (days)</th><th></th>
</tr></thead>
<tbody>{rows}</tbody>
</table>
</div>"""
else:
table = '<div class="panel muted">No channels yet. Add one below.</div>'
add_error_html = f'<div class="error">{_e(add_error)}</div>' if add_error else ""
body = f"""
<header>
<h1>youtube-automate</h1>
<div class="sub">
{len(channels)} channel(s) · {queue_depth} queued
· <form method="post" action="/logout" class="inline">
<input type="hidden" name="csrf" value="{_e(csrf)}">
<button class="link" type="submit">sign out</button>
</form>
</div>
</header>
{flash_html}
<h2>Channels</h2>
{table}
<h2>Add a channel</h2>
<form method="post" action="/channels" class="panel">
<input type="hidden" name="csrf" value="{_e(csrf)}">
<div class="row">
<input type="text" name="url" placeholder="https://www.youtube.com/@handle, @handle, or UC..." autocomplete="off">
<button type="submit">Add</button>
</div>
{add_error_html}
</form>
<h2>Settings</h2>
{_settings_form(settings_values, settings_errors, csrf)}
<footer>Downloads run hourly. Videos are deleted once they pass the retention
window for their channel — this is a DVR, not an archive.</footer>"""
return page("youtube-automate", body)
youtube_automate/ytdlp.py
+109
View File
@@ -0,0 +1,109 @@
"""Thin wrapper around the venv's yt-dlp binary.
Everything that shells out to yt-dlp goes through here so the PATH handling (Deno
must be discoverable — see specs.md §3) and the shared extractor args live in one
place.
"""
from __future__ import annotations
import json
import logging
import os
import subprocess
import urllib.request
from pathlib import Path
from . import config
log = logging.getLogger(__name__)
class YtdlpError(RuntimeError):
pass
def binary() -> Path:
return config.VENV_BIN / "yt-dlp"
def environment() -> dict[str, str]:
"""Env for a yt-dlp subprocess.
yt-dlp locates the JS runtime by searching PATH, so the venv's bin directory
must come first — that is where Deno lives.
"""
env = dict(os.environ)
env["PATH"] = f"{config.VENV_BIN}:{env.get('PATH', '')}"
return env
def version() -> str:
result = subprocess.run(
[str(binary()), "--version"],
capture_output=True,
text=True,
env=environment(),
timeout=60,
)
if result.returncode != 0:
raise YtdlpError(result.stderr.strip() or "yt-dlp --version failed")
return result.stdout.strip()
def extractor_args(pot_provider_url: str) -> list[str]:
return [
"--extractor-args",
"youtube:player_client=default,mweb",
"--extractor-args",
f"youtubepot-bgutilhttp:base_url={pot_provider_url}",
]
def run_json(args: list[str], timeout: int = 300) -> dict:
"""Run yt-dlp with -J and parse the single JSON document it prints."""
cmd = [str(binary()), *args]
log.debug("yt-dlp %s", " ".join(args))
result = subprocess.run(
cmd, capture_output=True, text=True, env=environment(), timeout=timeout
)
if result.returncode != 0:
raise YtdlpError(first_error(result.stderr) or "yt-dlp failed")
try:
return json.loads(result.stdout)
except json.JSONDecodeError as exc:
raise YtdlpError(f"could not parse yt-dlp output: {exc}") from exc
def run(args: list[str], timeout: int = 7200) -> subprocess.CompletedProcess:
"""Run yt-dlp for its side effects, returning the completed process."""
cmd = [str(binary()), *args]
log.debug("yt-dlp %s", " ".join(args))
return subprocess.run(
cmd, capture_output=True, text=True, env=environment(), timeout=timeout
)
def first_error(stderr: str) -> str:
for line in (stderr or "").splitlines():
if line.startswith("ERROR:"):
return line[len("ERROR:") :].strip()
return (stderr or "").strip().splitlines()[-1] if stderr.strip() else ""
def pot_provider_ping(base_url: str, timeout: float = 5.0) -> dict:
"""GET /ping on the bgutil provider. Raises on any failure."""
url = base_url.rstrip("/") + "/ping"
request = urllib.request.Request(url, headers={"User-Agent": config.USER_AGENT})
with urllib.request.urlopen(request, timeout=timeout) as response:
return json.loads(response.read().decode("utf-8"))
def plugin_version() -> str | None:
"""Installed bgutil plugin version, for comparison against the server's."""
try:
from importlib.metadata import version as pkg_version
return pkg_version("bgutil-ytdlp-pot-provider")
except Exception: # pragma: no cover - only when the plugin is absent
return None