youtube-automate/specs.md

youtube-automate — implementation spec

Target machine: susan Status: design agreed and verified against the live machine on 2026-08-11. Ready to build. Supersedes: specs.handover-original.md (the original design handover). Section 15 lists every change and why.

---

1. What we're building

A self-hosted pipeline that keeps a rolling retention window of recent YouTube uploads from a handful of subscribed channels, laid out on disk so Jellyfin presents each channel as a TV show and each video as an episode. A small web UI lets a non-operator (the requester's brother) subscribe, unsubscribe, and tweak settings.

The mental model is a DVR for YouTube subscriptions, not an archiver. Videos are disposable. If a video ages out or a channel is unsubscribed, it goes away permanently.

Explicitly in scope

• Poll subscribed channels for new uploads
• Download at ≤720p, h264/AAC, into a Jellyfin-friendly layout
• Generate Kodi-style .nfo metadata + artwork (no Jellyfin plugin)
• Delete videos older than the retention window
• Web admin over public HTTPS: add/remove channels, edit settings
• Cron-driven, idempotent, monitored via Healthchecks
• Offline unit tests

Explicitly out of scope — do not build these

• Cookie handling of any kind. No browser extension, no cookie-receiving API, no cookies.txt. See §3.
• Members-only, age-restricted, or private content (follows from the above)
• Transcoding of any kind (see §6 — this matters on this hardware)
• Multi-user support. One shared credential, one admin UI.
• Jellyfin watch-state protection (declined; see §10 and §15)
• CSV / Takeout importers

---

2. Environment facts

All of the following was verified on the live machine on 2026-08-11.

• susan: Debian 12, kernel 6.1.0-41. Dual Xeon X5675 (Westmere), 2×6 cores = 12 threads, 47 GB RAM. No AVX — confirmed absent from /proc/cpuinfo. Software video transcoding is effectively off the table.
• Jellyfin 10.11.4, running as a native systemd service (User=jellyfin, Group=jellyfin, UMask=0022) on 0.0.0.0:8096, enabled at boot. Not containerised.
• Docker running and enabled at boot, so --restart unless-stopped containers do come back after a reboot. No extra unit needed for the POT provider.
• Healthchecks container on 127.0.0.1:8001, published at hc.jihakuz.xyz.
• nginx active, Let's Encrypt via certbot, DNS at njal.la driven by ~/.local/bin/update-dns.sh (dynamic-DNS update keys, run every 15 min from cron).
• runitor v1.4.1 at /usr/local/bin/runitor. Cron convention: entries live in susan's crontab with HC_API_URL=https://hc.jihakuz.xyz/ping set at the top and one UUID per job.
• Tailscale 1.98.2, susan is 100.64.0.2. Headscale runs on the Linode, not here.
• susan is on a residential IP. This matters — YouTube flags datacenter IPs far more aggressively. Do not move any part of this to victoria (the Linode VPS).
• Toolchain: uv 0.11.23, Python 3.11.2, ffmpeg/ffprobe 5.1.7, sqlite 3.40.1, flock 2.38.1.
• /usr/local/bin/yt-dlp is version 2023.11.16 — a standalone binary from Dec 2023. It is far too old to work against YouTube today. Nothing on the machine references it (checked /usr/local/bin, /var/lib/radio, ~/.local/bin, and all crontabs), so it is left alone and we use our own venv copy instead.
• susan has no passwordless sudo. Anything touching /usr/local/bin, /etc/systemd/system or /etc/nginx must be run by the operator via deploy/deploy.sh.

TubeArchivist is gone

The original handover said /disks/Plex/YouTube was in use by TubeArchivist and must not be touched. This is no longer true. Verified: no TubeArchivist container exists (running or stopped), the directory is empty, and no Jellyfin library references it — the libraries are Audiobooks, Anime, Recordings, TV Shows, Music and Films. tube.jihakuz.xyz still has a DNS record in update-dns.sh but nothing serves it, so we reuse that hostname for the admin UI.

Paths

Purpose	Path
Source (bare repo)	/disks/git-repos/youtube-automate.git
Checkout	/opt/youtube-automate
Entry point	/usr/local/bin/youtube-automate
Virtualenv	/var/lib/youtube-automate/venv
Deno (JS runtime, §3)	/var/lib/youtube-automate/venv/bin/deno
State DB	/var/lib/youtube-automate/subs.db
Lock file	/var/lib/youtube-automate/run.lock
Media root	/disks/Plex/YouTube/
Scratch/work dir	/disks/Plex/YouTube/.work/

/disks is a single 5.5 TB ext4 volume (/dev/sdb2) with 1.3 TB free, so the work dir and the media root share a filesystem and finished downloads move into place with an atomic rename().

Estimated steady-state usage for the three test channels is 5–10 GB. Disk is a non-issue.

Ownership and permissions — do not skip this

The media tree convention on susan is susan:mediaserver, directories 0770, files 0664. /disks/Plex is 0770 with no setgid bit. Jellyfin reaches the tree only via its mediaserver supplementary group.

If the service runs as susan with the default umask 0022, new directories land susan:susan and only work by the o+rx bit — which breaks the moment the unit is hardened. Therefore:

• systemd unit runs User=susan, Group=mediaserver, UMask=0002
• the media root gets the setgid bit so new directories inherit mediaserver
• cron entries that call the CLI directly go through sg mediaserver "...", matching the existing radio entries in susan's crontab

---

3. YouTube access: PO tokens, not cookies

Verified against the yt-dlp wiki on 2026-08-11: the current recommendation is still the mweb client plus a PO Token Provider plugin supplying tokens for GVS (Google Video Server) requests. No account, no cookies, no browser. Cookies only unlock age-restricted / members-only / private content, which is out of scope, and carry a real risk of the Google account being banned.

Setup

1. Run the provider as a Docker sidecar. Docker is enabled at boot and unless-stopped survives a reboot, so no systemd unit is required:

   docker run --name bgutil-provider -d --restart unless-stopped --init \
     -p 127.0.0.1:4416:4416 brainicism/bgutil-ytdlp-pot-provider:1.3.1-deno
   

   Pin the tag. Floating latest on both halves is how you get a silent version skew.

2. Build the venv. All four of these are required — see "JS challenge solving" below for why the last two are not optional:

   uv venv --python 3.11 /var/lib/youtube-automate/venv
   VIRTUAL_ENV=/var/lib/youtube-automate/venv uv pip install \
     'yt-dlp[default]' \
     'bgutil-ytdlp-pot-provider==1.3.1' \
     'curl-cffi<0.16'
   

   • yt-dlp[default] — the [default] extra is what pulls in yt-dlp-ejs, the challenge solver scripts. Plain pip install yt-dlp does not include it.
   • bgutil-ytdlp-pot-provider==1.3.1 — must match the container tag.
   • curl-cffi<0.16 — enables request impersonation. yt-dlp 2026.7.4 accepts 0.5.10 or 0.10.x–0.15.x only; the current release (0.16.0) is rejected as "unsupported", and an unpinned install silently gets you that. Verify with yt-dlp --list-impersonate-targets, which must list Chrome/Safari targets rather than only (unavailable) lines.

3. Install a JavaScript runtime. Deno is the recommended runtime and there is none on susan. Install it inside the venv's bin/ so it needs no root and no PATH changes beyond the venv directory the CLI already uses:

   curl -fsSL -o deno.zip \
     https://github.com/denoland/deno/releases/latest/download/deno-x86_64-unknown-linux-gnu.zip
   unzip -q deno.zip && install -m 0755 deno /var/lib/youtube-automate/venv/bin/deno
   

   Deno 2.9.5 (V8 15.0) runs fine on Westmere despite the absent AVX — verified. Minimum supported version is 2.3.0. yt-dlp locates it on PATH, so every yt-dlp invocation must run with /var/lib/youtube-automate/venv/bin prepended to PATH.

JS challenge solving is mandatory — this is new since the original handover

Without a JS runtime and yt-dlp-ejs, yt-dlp emits:

WARNING: n challenge solving failed: Some formats may be missing. Ensure you have a
supported JavaScript runtime and challenge solver script distribution installed.

That's the nsig throttling signature. Unsolved, formats go missing and downloads can be throttled. Measured on susan: without the runtime, 22 formats and no mweb formats at all (everything came from the android client); with Deno + yt-dlp-ejs, 29 formats including the 12 mweb ones, and a 44 MB download completed at 7.7 MiB/s with no throttling.

Confirm the good state in -v output — all three lines must be present:

[debug] JS runtimes: deno-2.9.5
[debug] [youtube] [jsc] JS Challenge Providers: ... deno ...
[youtube] [jsc:deno] Solving JS challenges using deno

A benign line you can ignore: [pot:bgutil:script-deno] Script path doesn't exist: /home/susan/bgutil-ytdlp-pot-provider/server/.... That is the script variant of the bgutil plugin probing for a local checkout we deliberately don't have; the http variant is the one in use and it reports as available.

Verification

Before writing any other code:

PATH=/var/lib/youtube-automate/venv/bin:$PATH yt-dlp -v '<any watch URL>' -F

You must see:

[debug] [youtube] [pot] PO Token Providers: bgutil:http-1.3.1 (external), ...
[youtube] [pot:bgutil:http] Generating a gvs PO Token for mweb client via bgutil HTTP server
[debug] [youtube] <id>: Retrieved a gvs PO Token for mweb client

If you don't, stop and fix this first — everything downstream depends on it.

doctor and every run health-check the provider with a plain GET against http://127.0.0.1:4416/ping before a download batch, and fail loudly if it's down rather than silently accumulating 403s. The endpoint is confirmed to exist and returns {"server_uptime": <n>, "version": "<s>"} — doctor should also compare that version against the installed plugin version and warn on a mismatch.

The exact extractor-arg key youtubepot-bgutilhttp:base_url was confirmed against the current plugin README.

Status: proven on susan on 2026-08-11. Video cBwBMyJ_bxU downloaded end to end and ffprobe reports h264 (Main) 1280x720 @60fps + aac (LC) stereo in an mp4 container, with 3 SponsorBlock chapter marks embedded and no re-encoding. Files landed susan:mediaserver 0664 via the setgid bit and umask 0002.

Caveats

• The bgutil README warns that PO tokens no longer bypass the "Sign in to confirm you're not a bot" interstitial in most cases. That is a different failure mode (IP reputation) from the 403s on format URLs that this solves. Residential IP + low request volume + sleep intervals should keep us clear of it. If it starts firing, that's the point at which cookies get reconsidered — not before.
• yt-dlp's YouTube handling churns every few weeks. Client recommendations, extractor-arg names, and which clients need tokens all move. Treat the flags in §6 as the shape of the answer, not as gospel, and re-read the wiki when something breaks.
• Keep yt-dlp updated weekly under runitor (§9). A stale yt-dlp is the single most likely cause of "everything broke" — the 2023 binary already on this box is the cautionary example.

---

4. Discovery: RSS feeds

YouTube publishes an unauthenticated Atom feed per channel. Use the undocumented UULF playlist variant, which returns long-form videos only — no Shorts, no livestreams:

https://www.youtube.com/feeds/videos.xml?playlist_id=UULF<channel_id without the UC prefix>

So UCabc123... → https://www.youtube.com/feeds/videos.xml?playlist_id=UULFabc123...

Related prefixes: UU all uploads, UUSH shorts, UULV livestreams, UUMF/UUMO members-only.

The filtering is real — verified

Tested on 2026-08-11 against the three subscription targets:

Channel	UULF	UUSH	UULV	Shorts leaking into UULF
clabretro	200, 15 entries	200, 2 entries	200, 1 entry	0
EthosLab	200, 15 entries	404	404	n/a
DolanDarkest	200, 15 entries	404	404	n/a

clabretro is the useful case: it genuinely has 2 Shorts and 1 livestream, and none of them appear in the UULF feed. The primary discovery path does what it claims.

Note also that UUSH/UULV return HTTP 404 when a channel has none — that means "empty", not "broken". Only a UULF 404 (or an empty UULF result) triggers the fallback.

Robustness

These prefixes are undocumented and there have been reports of intermittent failures and missing entries in YouTube's native feeds. So:

• If the UULF feed 404s or returns zero entries, fall back to ?channel_id=UC... and record discovery_source = 'uc_feed' on the resulting rows.
• Rows discovered via the fallback path get the duration/live filter applied at download time (§6). Rows from UULF don't need it.
• A skipped_short row from the fallback path is re-queued if a later UULF poll lists that video. UULF is authoritative about what is and isn't a Short, so this repairs the case where a transient feed outage would otherwise permanently drop a legitimate short-but-not-Short video. See §6 for why this matters concretely.
• Track consecutive poll failures per channel and surface the count in the admin UI. Two channels quietly failing for a month is the realistic failure mode here.

Parsing

xml.etree.ElementTree, no dependencies. Namespaces:

NS = {
    "atom":  "http://www.w3.org/2005/Atom",
    "yt":    "http://www.youtube.com/xml/schemas/2015",
    "media": "http://search.yahoo.com/mrss/",
}

Per atom:entry: yt:videoId, atom:title, atom:published, media:group/media:description. The feed does not carry duration.

Backfill on subscribe

New subscriptions pull the last backfill_days (default 7).

--flat-playlist carries no upload dates. The original handover said to pull 50 entries with --flat-playlist -J and "date-filter the entries client-side". Verified against clabretro: every entry comes back with timestamp: None. There is no date to filter on. Entries do carry id, title, duration and live_status.

So the backfill is RSS-first:

1. Read the UULF feed (falling back to channel_id), which is the only source that carries dates. It returns ~15 entries — enough to cover a 7-day window for any channel uploading less than twice a day, which is all three subscription targets.
2. Only if the feed's oldest entry is still inside the backfill window might there be more, in which case extend via --flat-playlist --playlist-end 50 and resolve those extra dates one video at a time with --print "%(upload_date)s". The playlist is reverse-chronological, so stop at the first video outside the window.

In practice step 2 never runs for the current subscriptions. It exists so that raising backfill_days doesn't silently truncate at 15 videos.

skipped_old and rescan

A video discovered outside the window is skipped_old, which is terminal. That creates a trap: raising a channel's retention_days afterwards appears to do nothing, because every one of that channel's videos is already marked old. Observed with EthosLab — his most recent upload was 22 days before subscribe, so a 7-day backfill queued nothing at all for him.

rescan re-queues skipped_old rows whose upload_date now falls inside the channel's current effective retention window. It is an explicit action (poll --rescan, or a button in the admin UI), deliberately not something poll does silently — doing it on every poll would make backfill_days meaningless, since it would immediately re-queue everything the initial backfill had just decided to leave behind.

Tombstones are never touched: only skipped_old is eligible, never deleted.

Verified: setting EthosLab to 60 days and rescanning queued exactly the 3 videos inside that window and left the other 12 alone.

Getting the subscription list in

Manual entry via the admin page, one channel URL at a time. With single-digit channels that is enough. No CSV importer.

---

5. On-disk layout

Media root: /disks/Plex/YouTube/

/disks/Plex/YouTube/
├── .work/                                   # scratch, ignored by Jellyfin
│   └── .ignore                              # belt-and-braces, see below
└── Some Channel/
    ├── tvshow.nfo
    ├── poster.jpg
    ├── fanart.jpg
    └── Season 2026/
        ├── Some Channel - S2026E8110 - Video Title [dQw4w9WgXcQ].mp4
        ├── Some Channel - S2026E8110 - Video Title [dQw4w9WgXcQ].nfo
        ├── Some Channel - S2026E8110 - Video Title [dQw4w9WgXcQ]-thumb.jpg
        ├── Some Channel - S2026E8110 - Video Title [dQw4w9WgXcQ].en.srt
        └── Some Channel - S2026E8110 - Video Title [dQw4w9WgXcQ].info.json

The work dir is dot-prefixed (which Jellyfin skips) and contains an .ignore file, which Jellyfin also honours. Two independent mechanisms, because relying on undocumented hidden-folder behaviour for the directory holding partial downloads is not worth the risk.

Season / episode numbering

• Season = upload year (2026)
• Episode = MMDD * 10 + ordinal_within_day

So the first video uploaded on 11 August is 8110, the second that day is 8111, and the first on 12 August is 8120. This sorts correctly across the whole year — 1 Jan is 1010 and 31 Dec is 12310 — and tolerates up to 10 uploads per channel per day. Clamp the ordinal at 9 and log a warning if exceeded.

The ordinal must be computed against what's already in the DB for that channel+date, not against the current batch, so it's stable across runs.

Write the episode number unpadded (E8110), matching the <episode> value in the NFO exactly.

Filename sanitisation

Strip / \ : * ? " < > |, collapse runs of whitespace, strip leading/trailing dots and spaces, truncate the title component to 120 chars on a word boundary. The [videoid] suffix guarantees uniqueness regardless.

Store the channel directory name in the DB (channel.dir_name) at subscribe time and never recompute it — channels rename themselves and we don't want orphaned directories.

---

6. Download

Format selection — read this bit carefully

susan has no AVX and cannot realistically transcode. YouTube serves 720p as VP9 or AV1 by default. If a client can't direct-play those, Jellyfin will try to transcode and it will be miserable. So we force h264 + AAC in mp4 at download time:

-f "bv*[height<=720]+ba/b[height<=720]"
-S "vcodec:h264,res:720,acodec:aac"
--merge-output-format mp4

The sort field order matters and the original handover had it wrong. With -S "vcodec:h264,acodec:aac,res:720" — acodec ranked above res — this selects the legacy combined 360p format 18 rather than 720p. bv* means "best video, possibly with audio", so ranking acodec:aac before res makes a combined 360p stream (which has AAC) outrank a 720p video-only stream (whose acodec is none). Verified empirically: the original ordering yields 18 | 640x360, the corrected ordering yields 298+140 | 1280x720 | avc1 | mp4a.

Keep vcodec first, ahead of res. If a video has no h264 at 720p but does at 480p, this picks h264 at 480p — which direct-plays — rather than 720p VP9, which would force a transcode. On hardware that cannot transcode, codec fidelity beats resolution. Log any case that still falls back to VP9/AV1; they should be rare.

Do not use --sponsorblock-remove — it requires cutting and re-encoding. Use --sponsorblock-mark all with --embed-chapters, which writes chapter markers only and is free.

Full invocation

yt-dlp \
  --extractor-args "youtube:player_client=default,mweb" \
  --extractor-args "youtubepot-bgutilhttp:base_url=http://127.0.0.1:4416" \
  -f "bv*[height<=720]+ba/b[height<=720]" \
  -S "vcodec:h264,res:720,acodec:aac" \
  --merge-output-format mp4 \
  --no-playlist \
  --write-info-json \
  --write-thumbnail --convert-thumbnails jpg \
  --write-subs --write-auto-subs --sub-langs "en.*" --convert-subs srt \
  --sponsorblock-mark all --embed-chapters \
  --retries 3 --fragment-retries 10 \
  --sleep-requests 2 --sleep-interval 5 --max-sleep-interval 15 \
  -P "/disks/Plex/YouTube/.work" \
  -o "%(id)s.%(ext)s" \
  "https://www.youtube.com/watch?v=<VIDEO_ID>"

--write-subs is included alongside --write-auto-subs so channels with real uploaded subtitles aren't silently skipped in favour of nothing.

Subtitle de-duplication. --sub-langs "en.*" matches both en and en-orig, so a typical video yields two sidecars (<id>.en.srt and <id>.en-orig.srt) which Jellyfin would present as two identical "English" tracks. Observed on the test download. In the move-into-place step, keep .en.srt if present and discard .en-orig.srt; if only en-orig exists, rename it to .en.srt.

Match filter (fallback-discovered videos only)

For rows with discovery_source = 'uc_feed', append:

--match-filter "duration>?{min_duration_seconds} & live_status!=?is_live & live_status!=?is_upcoming & !was_live"

The >? and !=? forms allow videos with unknown values through rather than rejecting them — an absent live_status should not disqualify a video any more than an absent duration does.

min_duration_seconds defaults to 120.

Why 120 is a knowingly tight default

DolanDarkest — one of the three subscription targets — uploads daily videos of 122–167 seconds. One of them clears the filter by two seconds. Those videos arrive via UULF, which is exempt from the match filter, so in normal operation this is fine.

The risk is a transient UULF outage: rows then get discovery_source='uc_feed', the duration filter applies, and anything under two minutes would be dropped. This is why §4 re-queues a skipped_short row when a later UULF poll lists the video. With that repair in place the only residual loss case is a sub-120s video whose UULF feed never recovers inside the retention window, which would also be visible in the poll-failure counter.

A per-channel min_duration_seconds override was considered and declined; revisit if DolanDarkest starts posting sub-two-minute videos regularly.

Skip and defer semantics

Outcome	State	Retried?
Below min_duration_seconds	skipped_short	Only if a later UULF poll lists it (§4)
is_live or was_live	skipped_live	Never — livestreams are not wanted
is_upcoming	deferred	Yes — re-queued by later polls

is_upcoming must not be a permanent skip. A scheduled premiere becomes an ordinary downloadable video once it airs, so permanently rejecting it would silently lose content from any channel that uses premieres. Give up on a deferred row once it falls outside the retention window.

Concurrency and politeness

One download at a time. Single-digit channels over a 14-day window is a small workload; there is no reason to be aggressive and every reason not to be.

After a successful download

1. Read the .info.json from .work/
2. Compute season/episode/filename
3. Generate the episode .nfo (§7)
4. os.rename() all artefacts into the season directory
5. Update the DB row: state='downloaded', rel_path, downloaded_at, size_bytes
6. After the whole batch, trigger a Jellyfin library refresh (§7)

size_bytes is required — disk_cap_gb cannot work without it.

If any step 1–4 fails, clean up .work/ for that video ID and mark failed with attempts += 1. Give up after max_attempts (5) and surface it in the UI.

---

7. Metadata — NFO files, no Jellyfin plugin

Jellyfin reads Kodi-style NFO sidecars natively. The library is created as Shows with all internet metadata providers disabled, Prefer local metadata on, and Save artwork/metadata into media folders on. Do not write or install a metadata provider plugin.

The library is created programmatically via POST /Library/VirtualFolders using the stored admin API key, so the setup is reproducible rather than a hand-clicked state.

tvshow.nfo (per channel, written at subscribe time, refreshed on title change)

<?xml version="1.0" encoding="utf-8"?>
<tvshow>
  <title>Some Channel</title>
  <plot>Channel description from yt-dlp.</plot>
  <studio>YouTube</studio>
  <uniqueid type="youtube" default="true">UCabc123...</uniqueid>
</tvshow>

Episode .nfo (one per video, filename matches the media file)

<?xml version="1.0" encoding="utf-8"?>
<episodedetails>
  <title>Video Title</title>
  <showtitle>Some Channel</showtitle>
  <season>2026</season>
  <episode>8110</episode>
  <plot>Video description.</plot>
  <aired>2026-08-11</aired>
  <runtime>12</runtime>
  <studio>YouTube</studio>
  <uniqueid type="youtube" default="true">dQw4w9WgXcQ</uniqueid>
</episodedetails>

runtime is in minutes. Build these with xml.etree's serialiser, never string formatting — video descriptions are hostile input and contain everything.

Artwork

At subscribe time, yt-dlp --flat-playlist --playlist-items 0 -J <channel_url> returns a thumbnails array containing entries with id values like avatar_uncropped and banner_uncropped. Download the avatar to poster.jpg and the banner to fanart.jpg. Treat both as best-effort — if they're missing, carry on without them.

Per-episode thumbnails come from --write-thumbnail; rename to <basename>-thumb.jpg.

Library refresh

POST {jellyfin_url}/Library/Refresh with header X-Emby-Token: {api_key}, triggered after:

• a download batch that produced at least one new file
• a reap that deleted at least one file
• an unsubscribe

Refreshing after deletions matters — without it Jellyfin shows ghost episodes until its own scheduled scan.

---

8. Data model

SQLite at /var/lib/youtube-automate/subs.db. WAL mode — the cron job and the web server both write.

PRAGMA journal_mode = WAL;
PRAGMA foreign_keys = ON;

CREATE TABLE channel (
  id                INTEGER PRIMARY KEY,
  channel_id        TEXT NOT NULL UNIQUE,      -- UC...
  handle            TEXT,                      -- @handle, informational
  title             TEXT NOT NULL,
  description       TEXT,
  dir_name          TEXT NOT NULL UNIQUE,      -- sanitised, immutable after creation
  added_at          TEXT NOT NULL,
  backfilled        INTEGER NOT NULL DEFAULT 0,
  retention_days    INTEGER,                   -- NULL = use global setting
  last_polled_at    TEXT,
  last_poll_ok      INTEGER,
  consecutive_poll_failures INTEGER NOT NULL DEFAULT 0
);

CREATE TABLE video (
  id                INTEGER PRIMARY KEY,
  video_id          TEXT NOT NULL UNIQUE,
  channel_pk        INTEGER NOT NULL REFERENCES channel(id) ON DELETE CASCADE,
  title             TEXT,
  upload_date       TEXT,                      -- YYYY-MM-DD
  duration          INTEGER,
  season            INTEGER,
  episode           INTEGER,
  state             TEXT NOT NULL,
  discovery_source  TEXT NOT NULL,             -- 'uulf_feed' | 'uc_feed' | 'backfill'
  rel_path          TEXT,                      -- relative to media root, NULL unless downloaded
  size_bytes        INTEGER,
  attempts          INTEGER NOT NULL DEFAULT 0,
  last_error        TEXT,
  discovered_at     TEXT NOT NULL,
  downloaded_at     TEXT,
  deleted_at        TEXT
);

CREATE INDEX idx_video_state ON video(state);
CREATE INDEX idx_video_upload_date ON video(upload_date);
CREATE INDEX idx_video_channel ON video(channel_pk);

CREATE TABLE setting (
  key   TEXT PRIMARY KEY,
  value TEXT NOT NULL
);

Schema changes are applied by an idempotent migration function keyed on PRAGMA user_version.

video.state values

State	Meaning	Retried?
pending	discovered, queued	—
downloading	claimed by a worker	recovered on startup → pending
downloaded	on disk, rel_path set	—
deleted	aged out; tombstone prevents re-download	never
deferred	premiere/upcoming, not yet available	yes, on later polls
skipped_short	rejected by match filter	only if later seen in UULF (§4)
skipped_live	rejected by match filter	never
skipped_old	discovered but already outside the window	never
failed	download error	yes, up to max_attempts

The tombstone behaviour is the important part: a deleted row must never be re-discovered and re-downloaded, and the skipped_short repair in §4 must never resurrect one. This is why we keep our own state table rather than using yt-dlp's --download-archive.

Settings

Store as strings; provide typed accessors with defaults so a missing key never crashes.

Key	Default	Notes
retention_days	14	global default; per-channel override wins
backfill_days	7	window pulled when a channel is added
max_height	720
min_duration_seconds	120	fallback-path Shorts filter
sponsorblock_mark	true
write_subs	true
sub_langs	en.*
jellyfin_url	http://127.0.0.1:8096
jellyfin_api_key	(empty)	masked in the UI; set via CLI
pot_provider_url	http://127.0.0.1:4416
max_attempts	5
disk_cap_gb	0	0 = disabled; if set, evict oldest downloaded videos

Validate on save (integers parse, URLs well-formed) and re-render the form with an inline error rather than 500-ing.

Two further keys are not editable through the settings form and are never rendered: admin_password_hash (see §11) and session_secret (generated on first run).

No secret is committed to the repo. The Jellyfin API key and the admin password are set with CLI subcommands and live only in the DB.

---

9. Entry point and scheduling

/usr/local/bin/youtube-automate is a shim that execs the venv interpreter against the package. Subcommands:

Command	Purpose
run [--channel ID]	poll → download → reap. This is what cron calls.
poll [--channel ID]	discovery only
download	drain the pending queue
reap	retention pass
serve	admin HTTP server (systemd unit)
subscribe <url> / unsubscribe <id>	CLI equivalents, useful for debugging
set-password	prompt for and store the admin password (never on the command line)
set-jellyfin-key	store the Jellyfin API key
setup-jellyfin-library	create/verify the YouTube Shows library with the right options
doctor	yt-dlp version, POT provider reachable, DB writable, media root writable, Jellyfin reachable, permissions sane

run takes a non-blocking flock on /var/lib/youtube-automate/run.lock and exits 0 silently if already held. On startup it resets any downloading rows to pending and clears matching orphans out of .work/ (crash recovery).

run accepts --channel so the web subscribe handler can spawn an immediate backfill rather than waiting up to an hour.

Cron (susan's crontab, alongside the existing entries)

17 * * * * runitor -uuid 41a4d61a-7743-43d9-9b5d-d37d536e4726 -- sg mediaserver "/usr/local/bin/youtube-automate run"
40 4 * * 1 runitor -uuid 721e4cf0-d796-48e7-a184-79d21e1ba373 -- /var/lib/youtube-automate/venv/bin/python -m uv pip install --upgrade yt-dlp

Hourly is ample for single-digit channels; both checks are registered in Healthchecks with generous grace periods. The weekly updater is monitored separately so a stale yt-dlp alerts distinctly from a failed run — §12 calls that the top operational risk.

systemd unit for the admin server

Type=simple, Restart=always, User=susan, Group=mediaserver, UMask=0002. Binds 127.0.0.1:8085 only.

nginx

tube.jihakuz.xyz → 127.0.0.1:8085, public, TLS via certbot. The DNS record and njal.la update key already exist in update-dns.sh, so no DNS work is needed.

The hostname was already claimed. A leftover TubeArchivist server block inside sites-available/jihakuz.xyz served tube.jihakuz.xyz, proxying to the now-dead 127.0.0.1:8003. nginx uses the first server block matching a name and that file loads before any standalone tube.jihakuz.xyz file, so installing a second vhost changed nothing and every request returned 502. The old block also already owns the Let's Encrypt certificate for the hostname, so the fix is to repoint it rather than duplicate it — deploy/fix-nginx-tube.sh does that, and removes the redundant standalone vhost.

The forwarding headers are not cosmetic: the app throttles failed logins per client address and reads X-Forwarded-For. Without it every attempt appears to come from nginx itself, so a single attacker would lock out everyone.

Because the UI is public, the app carries real CSRF tokens and treats every input as hostile (§11).

---

10. Retention and deletion

Aging out (reap)

Effective retention for a video = its channel's retention_days if set, else the global retention_days. Candidates: state = 'downloaded' and upload_date < today - effective.

Per-channel overrides exist because the subscription mix spans wildly different cadences. EthosLab uploads roughly every two to three weeks (20 Jul, 1 Jul, 19 Jun, 1 Jun, 11 May), so at a flat 14 days his show would sit empty in Jellyfin more often than not.

Deletion removes the media file plus its .nfo, -thumb.jpg, .srt and .info.json siblings, then prunes the season directory if it's empty, and the channel directory if that leaves it bare of seasons. Set state='deleted', deleted_at, and rel_path=NULL. Keep the row — it's the tombstone. Trigger a Jellyfin refresh if anything was deleted.

Reap is a purely local operation and does not require Jellyfin to be reachable. Watch-state protection was considered and declined (§15), so a video can in principle vanish mid-watch; the mitigation available is raising that channel's retention_days.

Unsubscribe

Hard delete: shutil.rmtree() the channel directory, then DELETE FROM channel (cascades to video). Re-subscribing later starts from scratch and re-downloads the backfill window. The UI requires a confirmation step — this is destructive and irreversible.

Disk cap (optional, disk_cap_gb)

If enabled, after each reap sum size_bytes over downloaded rows and evict oldest-by- upload_date until under the cap.

---

11. Admin UI

Stdlib only — http.server.ThreadingHTTPServer + BaseHTTPRequestHandler. No Flask, no FastAPI, no npm. Server-rendered HTML with a single embedded <style> block. The only JavaScript is a confirm() on the unsubscribe button.

Authentication

The UI is publicly reachable, so this is real auth, and it must not nag — the requirement is "log in once per device, effectively never again".

• One shared password, stored as a scrypt hash (n=2**14, r=8, p=1, per-install salt) in setting.admin_password_hash. Set only via youtube-automate set-password, which prompts — never passed as an argument, never written to the repo.
• Login form posts to /login. On success, set a session cookie containing an HMAC-signed token (session_secret, generated on first run), with attributes Secure; HttpOnly; SameSite=Lax; Path=/; Max-Age=31536000 — a 365-day expiry.
• Every state-changing POST carries a CSRF token bound to the session and verified constant-time.
• Login throttling: after 5 consecutive failures from an IP, reject for 60 seconds. In-memory is fine.
• All comparisons of secrets use hmac.compare_digest.

Routes

Method	Path	Behaviour
GET	/login	login form
POST	/login	authenticate, set cookie, 303 → /
POST	/logout	clear cookie, 303 → /login
GET	/	channel list + add form + settings form
POST	/channels	resolve URL → insert → spawn backfill → 303 → /
POST	/channels/<id>/retention	set or clear the per-channel override
POST	/channels/<id>/delete	confirm-guarded hard delete → 303
POST	/settings	validate + persist → 303
POST	/channels/<id>/rescan	re-queue skipped_old inside the current window → 303
GET	/health	JSON: yt-dlp version, POT provider up, last run time, queue depth

All POSTs redirect (303) so refresh doesn't resubmit. Everything except /login requires a valid session — including /health, which the original spec left open. That was reasonable when the UI was going to be tailnet-only; on a public hostname it is gratuitous fingerprinting surface (yt-dlp version, queue depth, channel count) for no benefit, since the cron job is monitored by Healthchecks rather than by polling this endpoint.

Channel list should show, per channel

Title, @handle, video count on disk, disk usage, most recent upload date, last poll time, the effective retention (and whether it's an override), and a warning badge if consecutive_poll_failures > 2.

Channel URL resolution

Accept: https://www.youtube.com/@handle, /channel/UC..., /c/name, /user/name, a bare @handle, or a bare UC... ID.

yt-dlp --flat-playlist --playlist-items 0 -J <url>

--playlist-items 0 fetches channel metadata without enumerating the uploads, which is the cheap way to do this. Pull channel_id, channel, description, thumbnails from the result. If resolution fails or returns no channel_id, re-render the form with an error — don't insert a half-formed row.

Reject duplicates on channel_id, not on the submitted URL — the same channel has many URL forms.

---

12. Known gotchas

• yt-dlp churn is the top operational risk. Extractor args, client names, and PO token requirements change frequently. Weekly update under runitor. When something breaks, check the yt-dlp issue tracker before debugging our code.
• The UULF prefix is undocumented and could vanish. The channel_id fallback path is not optional. UUSH/UULV 404 means "none", not "broken".
• RSS feeds cap at ~15 entries. Fine for hourly polling, insufficient for backfill — hence the separate --flat-playlist path in §4.
• .work must share a filesystem with the media root, or every completed download becomes a full copy. Both are under /disks (/dev/sdb2).
• Don't transcode. If Jellyfin is transcoding these files, the format selection in §6 is wrong — fix it there, not by throwing hardware at it.
• Descriptions are hostile input for XML generation. Use xml.etree's serialiser.
• WAL mode is required — two writers.
• Group and umask matter (§2). Files that Jellyfin can't read are a silent failure.
• No passwordless sudo — root steps go in deploy/deploy.sh for the operator to run.

---

13. Acceptance criteria

Work through these in order; each is a real check, not a code-reading exercise.

1. doctor passes on a clean install.
2. Subscribing to a real, active channel creates the directory, tvshow.nfo, poster.jpg, and queues the backfill window.
3. At least one real video downloads end to end and lands in the right season directory with a correct filename, NFO, thumbnail, and subtitle sidecar.
4. ffprobe on the downloaded file shows h264 video and AAC audio.
5. Files and directories are readable by the jellyfin user (group mediaserver, 0664/0770).
6. Jellyfin, after a library scan, shows the channel as a show, the year as a season, and the video as an episode with the correct title, description, and air date — with all internet metadata providers disabled.
7. The video direct-plays on a client with no transcoding — the Jellyfin dashboard's active-streams panel must show neither video nor audio transcoding.
8. Re-running run immediately downloads nothing and errors on nothing (idempotency).
9. Manually backdating a video's upload_date past the retention window causes reap to delete it and its sidecars, prune the empty directory, and leave a deleted tombstone.
10. Re-running poll after that does not re-download the deleted video.
11. A per-channel retention_days override visibly changes which videos that channel's reap deletes, while other channels follow the global value.
12. Unsubscribing removes the directory and all rows.
13. Killing run mid-download leaves no orphan in .work, and the next run recovers the downloading row to pending.
14. The admin page requires login, stays logged in across a browser restart, and rejects a POST with a missing or wrong CSRF token.
15. The admin page renders, adds, removes, and persists settings; a bad settings value produces an inline error rather than a traceback.
16. Editing retention_days in the UI visibly changes reap behaviour on the next run.
17. pytest passes with no network access.

---

14. Build order

1. Skeleton: package layout, DB schema + migrations, settings accessors, doctor
2. venv + POT provider + a hardcoded single-video download proving §3 and §6 work — do this before writing anything else of substance
3. Channel resolution + subscribe/unsubscribe on the CLI
4. Discovery (poll), both feed paths, including the skipped_short repair
5. Download worker + NFO/artwork generation + move-into-place
6. Jellyfin library creation and verification (criteria 6 and 7) — a checkpoint, not a step
7. reap, including per-channel overrides
8. run orchestration, flock, crash recovery, cron + runitor
9. Admin server, including auth and CSRF
10. systemd unit, nginx vhost, certbot, deploy.sh

Steps 2 and 6 are the two places where this design could still turn out to be wrong. Hit them early and report back rather than building on top of an unverified assumption.

Tests

pytest, offline, no network and no real yt-dlp invocation. Fakes for yt-dlp and the Jellyfin API. Coverage targets:

• filename sanitisation, including hostile titles and the 120-char truncation
• season/episode numbering and the ordinal-within-day logic, including the clamp at 9
• RSS parsing from fixture XML, both UULF and fallback shapes, including a 404
• the skipped_short → pending repair, and proof it never resurrects a deleted tombstone
• NFO generation with descriptions full of &, <, emoji and newlines
• state machine transitions, including crash recovery and deferred
• reap candidate selection with and without per-channel overrides
• settings validation and typed accessors with missing keys
• auth: scrypt round-trip, cookie signing, CSRF verification, throttling

---

15. Changes from the original handover

Area	Change	Why
Naming	ytsubs → youtube-automate throughout	The original contradicted itself; the bare repo, checkout and state dir already existed under youtube-automate
Media root	/disks/Plex/YouTubeSubs → /disks/Plex/YouTube; .ingest → .work	TubeArchivist is not installed, the directory is empty and no Jellyfin library references it — verified
Admin UI	Tailnet-only + basic auth → public HTTPS at tube.jihakuz.xyz with app login, scrypt, 365-day signed cookie, CSRF	Requester's brother has no tailnet device; basic auth re-prompts on some mobile browsers
Watch state	Removed	Declined. There are five Jellyfin users, so a single jellyfin_user_id would protect one person's progress; per-channel retention is the mitigation instead
Retention	Added per-channel retention_days override	EthosLab's 2–3 week cadence would leave his show empty at a flat 14 days
Shorts filter	min_duration_seconds stays 120, but skipped_short is repaired on a later UULF sighting	DolanDarkest posts 122–167s videos; the repair removes the transient-outage data-loss path
Premieres	is_upcoming now deferred and retried, not permanently skipped	A premiere becomes downloadable once it airs
Subtitles	Added --write-subs alongside --write-auto-subs	Channels with real uploaded subs were being skipped
Match filter	!= → !=? on live_status	An unknown live_status shouldn't disqualify a video, same logic as duration>?
Download	Record size_bytes	disk_cap_gb was unimplementable without it
Refresh	Also refresh Jellyfin after reap and unsubscribe	Otherwise ghost episodes linger
Episode format	E08110 → E8110	Consistency with the <episode> value in the NFO
Permissions	Added Group=mediaserver, UMask=0002, setgid on media root	Original was silent on it; Jellyfin reads the tree only via that group
.work	Added an .ignore file as well as the dot prefix	Not worth relying on undocumented behaviour for the partial-download directory
Versions	Pinned bgutil to 1.3.1 / 1.3.1-deno	Floating latest on both halves invites version skew
yt-dlp	Install in the venv; leave /usr/local/bin/yt-dlp (2023.11.16) alone	Nothing on the box references it, so replacing it buys nothing and risks other jobs
run	Accepts --channel	§9 spawned run --channel but only defined the flag on poll
Monitoring	Two Healthchecks UUIDs, one per job	Matches the existing one-UUID-per-cron-entry convention on susan
Tests	Added §14 test plan	Requested

Found during build-order step 2 (the PO token proof)

These were discovered by actually running the pipeline, not by reading the doc.

Area	Change	Why
Format sort	-S "vcodec:h264,acodec:aac,res:720" → -S "vcodec:h264,res:720,acodec:aac"	The original selects 360p. Ranking acodec above res makes the combined 360p format (which has AAC) outrank the 720p video-only stream (acodec=none) under bv*. Verified: original gives 18 | 640x360, corrected gives 298+140 | 1280x720
JS runtime	Added Deno into the venv bin/, plus PATH requirement	New yt-dlp requirement. Without it, n challenge solving failed — 22 formats and zero mweb formats vs 29 with it. Deno's V8 runs fine without AVX
yt-dlp extras	yt-dlp → yt-dlp[default]	The [default] extra is what ships yt-dlp-ejs, the challenge solver scripts. Plain yt-dlp does not include it
Impersonation	Added curl-cffi<0.16	Unpinned installs get 0.16.0, which yt-dlp rejects as "unsupported"; the supported range is 0.5.10 or 0.10.x–0.15.x. Removes a per-download warning and helps against the bot-detection interstitial
Subtitles	De-duplicate en vs en-orig on move	--sub-langs "en.*" matches both, producing two identical English tracks in Jellyfin
doctor	Also compare /ping reported version against the installed plugin version	Version skew between container and plugin is silent otherwise

Found during the rest of the build

Area	Change	Why
Backfill	--flat-playlist has no dates; RSS is now the primary source	Verified: every flat-playlist entry returns timestamp: None, so the specced client-side date filter was impossible
skipped_old	Added explicit rescan	Otherwise raising a channel's retention appears to do nothing. EthosLab's 7-day backfill queued zero videos
/health	Now requires a session	The UI moved from tailnet-only to public
Reap	Channel directory is kept when its last season is pruned	Deleting it would make an active subscription vanish from Jellyfin and reappear later, and would bin the artwork
Premieres	is_upcoming → deferred, retried	A premiere becomes downloadable once it airs

Verified end to end on 2026-08-11

doctor passes clean. All three channels subscribe from three different URL forms, with tvshow.nfo and real JPEG artwork. Backfill queued the right windows. A real video downloaded to h264 Main 1280x720 / aac LC in mp4 with SponsorBlock chapters, landed as EthosLab - S2026E7010 - ... [j5jhOZvm5Vo].mp4 with all four sidecars and susan:mediaserver 0664. Jellyfin resolved it from the local NFO alone — correct title, season 2026, episode 7010, aired date, overview — and reports SupportsDirectPlay: true. Reap deleted all five artefacts, pruned the season directory, kept the channel artwork and left a tombstone that survives both poll and rescan. The lock refuses a second run instantly. 240 offline tests pass.