# `youtube-automate` — implementation spec **Target machine:** `susan` **Status:** design agreed and verified against the live machine on 2026-08-11. Ready to build. **Supersedes:** `specs.handover-original.md` (the original design handover). Section 15 lists every change and why. --- ## 1. What we're building A self-hosted pipeline that keeps a rolling retention window of recent YouTube uploads from a handful of subscribed channels, laid out on disk so Jellyfin presents each channel as a TV show and each video as an episode. A small web UI lets a non-operator (the requester's brother) subscribe, unsubscribe, and tweak settings. The mental model is **a DVR for YouTube subscriptions**, not an archiver. Videos are disposable. If a video ages out or a channel is unsubscribed, it goes away permanently. ### Explicitly in scope - Poll subscribed channels for new uploads - Download at ≤720p, h264/AAC, into a Jellyfin-friendly layout - Generate Kodi-style `.nfo` metadata + artwork (no Jellyfin plugin) - Delete videos older than the retention window - Web admin over public HTTPS: add/remove channels, edit settings - Cron-driven, idempotent, monitored via Healthchecks - Offline unit tests ### Explicitly out of scope — do not build these - **Cookie handling of any kind.** No browser extension, no cookie-receiving API, no `cookies.txt`. See §3. - Members-only, age-restricted, or private content (follows from the above) - Transcoding of any kind (see §6 — this matters on this hardware) - Multi-user support. One shared credential, one admin UI. - Jellyfin watch-state protection (declined; see §10 and §15) - CSV / Takeout importers --- ## 2. Environment facts All of the following was verified on the live machine on 2026-08-11. - **susan**: Debian 12, kernel 6.1.0-41. Dual **Xeon X5675** (Westmere), 2×6 cores = 12 threads, 47 GB RAM. **No AVX** — confirmed absent from `/proc/cpuinfo`. Software video transcoding is effectively off the table. - **Jellyfin 10.11.4**, running as a native systemd service (`User=jellyfin`, `Group=jellyfin`, `UMask=0022`) on `0.0.0.0:8096`, enabled at boot. Not containerised. - **Docker** running and `enabled` at boot, so `--restart unless-stopped` containers do come back after a reboot. No extra unit needed for the POT provider. - **Healthchecks** container on `127.0.0.1:8001`, published at `hc.jihakuz.xyz`. - **nginx** active, Let's Encrypt via certbot, DNS at njal.la driven by `~/.local/bin/update-dns.sh` (dynamic-DNS update keys, run every 15 min from cron). - **runitor** v1.4.1 at `/usr/local/bin/runitor`. Cron convention: entries live in **susan's** crontab with `HC_API_URL=https://hc.jihakuz.xyz/ping` set at the top and one UUID per job. - **Tailscale** 1.98.2, susan is `100.64.0.2`. Headscale runs on the Linode, not here. - susan is on a **residential IP**. This matters — YouTube flags datacenter IPs far more aggressively. Do not move any part of this to `victoria` (the Linode VPS). - **Toolchain**: uv 0.11.23, Python 3.11.2, ffmpeg/ffprobe 5.1.7, sqlite 3.40.1, flock 2.38.1. - **`/usr/local/bin/yt-dlp` is version 2023.11.16** — a standalone binary from Dec 2023. It is far too old to work against YouTube today. Nothing on the machine references it (checked `/usr/local/bin`, `/var/lib/radio`, `~/.local/bin`, and all crontabs), so it is left alone and we use our own venv copy instead. - **susan has no passwordless sudo.** Anything touching `/usr/local/bin`, `/etc/systemd/system` or `/etc/nginx` must be run by the operator via `deploy/deploy.sh`. ### TubeArchivist is gone The original handover said `/disks/Plex/YouTube` was in use by TubeArchivist and must not be touched. **This is no longer true.** Verified: no TubeArchivist container exists (running or stopped), the directory is empty, and no Jellyfin library references it — the libraries are Audiobooks, Anime, Recordings, TV Shows, Music and Films. `tube.jihakuz.xyz` still has a DNS record in `update-dns.sh` but nothing serves it, so we reuse that hostname for the admin UI. ### Paths | Purpose | Path | |---|---| | Source (bare repo) | `/disks/git-repos/youtube-automate.git` | | Checkout | `/opt/youtube-automate` | | Entry point | `/usr/local/bin/youtube-automate` | | Virtualenv | `/var/lib/youtube-automate/venv` | | Deno (JS runtime, §3) | `/var/lib/youtube-automate/venv/bin/deno` | | State DB | `/var/lib/youtube-automate/subs.db` | | Lock file | `/var/lib/youtube-automate/run.lock` | | Media root | `/disks/Plex/YouTube/` | | Scratch/work dir | `/disks/Plex/YouTube/.work/` | `/disks` is a single 5.5 TB ext4 volume (`/dev/sdb2`) with 1.3 TB free, so the work dir and the media root share a filesystem and finished downloads move into place with an atomic `rename()`. Estimated steady-state usage for the three test channels is 5–10 GB. Disk is a non-issue. ### Ownership and permissions — do not skip this The media tree convention on susan is `susan:mediaserver`, directories `0770`, files `0664`. `/disks/Plex` is `0770` with **no setgid bit**. Jellyfin reaches the tree only via its `mediaserver` supplementary group. If the service runs as susan with the default `umask 0022`, new directories land `susan:susan` and only work by the `o+rx` bit — which breaks the moment the unit is hardened. Therefore: - systemd unit runs `User=susan`, `Group=mediaserver`, `UMask=0002` - the media root gets the **setgid bit** so new directories inherit `mediaserver` - cron entries that call the CLI directly go through `sg mediaserver "..."`, matching the existing `radio` entries in susan's crontab --- ## 3. YouTube access: PO tokens, not cookies Verified against the yt-dlp wiki on 2026-08-11: the current recommendation is still the **`mweb` client plus a PO Token Provider plugin** supplying tokens for GVS (Google Video Server) requests. No account, no cookies, no browser. Cookies only unlock age-restricted / members-only / private content, which is out of scope, and carry a real risk of the Google account being banned. ### Setup 1. Run the provider as a Docker sidecar. Docker is enabled at boot and `unless-stopped` survives a reboot, so no systemd unit is required: ``` docker run --name bgutil-provider -d --restart unless-stopped --init \ -p 127.0.0.1:4416:4416 brainicism/bgutil-ytdlp-pot-provider:1.3.1-deno ``` **Pin the tag.** Floating `latest` on both halves is how you get a silent version skew. 2. Build the venv. **All four of these are required** — see "JS challenge solving" below for why the last two are not optional: ``` uv venv --python 3.11 /var/lib/youtube-automate/venv VIRTUAL_ENV=/var/lib/youtube-automate/venv uv pip install \ 'yt-dlp[default]' \ 'bgutil-ytdlp-pot-provider==1.3.1' \ 'curl-cffi<0.16' ``` - `yt-dlp[default]` — the `[default]` extra is what pulls in **`yt-dlp-ejs`**, the challenge solver scripts. Plain `pip install yt-dlp` does **not** include it. - `bgutil-ytdlp-pot-provider==1.3.1` — must match the container tag. - `curl-cffi<0.16` — enables request impersonation. yt-dlp 2026.7.4 accepts `0.5.10` or `0.10.x`–`0.15.x` only; the current release (0.16.0) is rejected as "unsupported", and an unpinned install silently gets you that. Verify with `yt-dlp --list-impersonate-targets`, which must list Chrome/Safari targets rather than only `(unavailable)` lines. 3. Install a JavaScript runtime. Deno is the recommended runtime and there is none on susan. Install it **inside the venv's `bin/`** so it needs no root and no PATH changes beyond the venv directory the CLI already uses: ``` curl -fsSL -o deno.zip \ https://github.com/denoland/deno/releases/latest/download/deno-x86_64-unknown-linux-gnu.zip unzip -q deno.zip && install -m 0755 deno /var/lib/youtube-automate/venv/bin/deno ``` Deno 2.9.5 (V8 15.0) runs fine on Westmere despite the absent AVX — verified. Minimum supported version is 2.3.0. yt-dlp locates it on `PATH`, so every yt-dlp invocation must run with `/var/lib/youtube-automate/venv/bin` prepended to `PATH`. ### JS challenge solving is mandatory — this is new since the original handover Without a JS runtime and `yt-dlp-ejs`, yt-dlp emits: ``` WARNING: n challenge solving failed: Some formats may be missing. Ensure you have a supported JavaScript runtime and challenge solver script distribution installed. ``` That's the `nsig` throttling signature. Unsolved, formats go missing and downloads can be throttled. Measured on susan: without the runtime, 22 formats and **no `mweb` formats at all** (everything came from the android client); with Deno + `yt-dlp-ejs`, **29 formats including the 12 `mweb` ones**, and a 44 MB download completed at 7.7 MiB/s with no throttling. Confirm the good state in `-v` output — all three lines must be present: ``` [debug] JS runtimes: deno-2.9.5 [debug] [youtube] [jsc] JS Challenge Providers: ... deno ... [youtube] [jsc:deno] Solving JS challenges using deno ``` A benign line you can ignore: `[pot:bgutil:script-deno] Script path doesn't exist: /home/susan/bgutil-ytdlp-pot-provider/server/...`. That is the *script* variant of the bgutil plugin probing for a local checkout we deliberately don't have; the *http* variant is the one in use and it reports as available. ### Verification Before writing any other code: ``` PATH=/var/lib/youtube-automate/venv/bin:$PATH yt-dlp -v '' -F ``` You must see: ``` [debug] [youtube] [pot] PO Token Providers: bgutil:http-1.3.1 (external), ... [youtube] [pot:bgutil:http] Generating a gvs PO Token for mweb client via bgutil HTTP server [debug] [youtube] : Retrieved a gvs PO Token for mweb client ``` If you don't, stop and fix this first — everything downstream depends on it. `doctor` and every `run` health-check the provider with a plain `GET` against `http://127.0.0.1:4416/ping` before a download batch, and fail loudly if it's down rather than silently accumulating 403s. The endpoint is confirmed to exist and returns `{"server_uptime": , "version": ""}` — `doctor` should also compare that `version` against the installed plugin version and warn on a mismatch. The exact extractor-arg key `youtubepot-bgutilhttp:base_url` was confirmed against the current plugin README. **Status: proven on susan on 2026-08-11.** Video `cBwBMyJ_bxU` downloaded end to end and `ffprobe` reports `h264 (Main) 1280x720 @60fps` + `aac (LC) stereo` in an mp4 container, with 3 SponsorBlock chapter marks embedded and no re-encoding. Files landed `susan:mediaserver` `0664` via the setgid bit and `umask 0002`. ### Caveats - The bgutil README warns that PO tokens no longer bypass the "Sign in to confirm you're not a bot" interstitial in most cases. That is a **different failure mode** (IP reputation) from the 403s on format URLs that this solves. Residential IP + low request volume + sleep intervals should keep us clear of it. If it starts firing, that's the point at which cookies get reconsidered — not before. - **yt-dlp's YouTube handling churns every few weeks.** Client recommendations, extractor-arg names, and which clients need tokens all move. Treat the flags in §6 as the shape of the answer, not as gospel, and re-read the wiki when something breaks. - Keep yt-dlp updated weekly under runitor (§9). A stale yt-dlp is the single most likely cause of "everything broke" — the 2023 binary already on this box is the cautionary example. --- ## 4. Discovery: RSS feeds YouTube publishes an unauthenticated Atom feed per channel. Use the **undocumented `UULF` playlist variant**, which returns long-form videos only — no Shorts, no livestreams: ``` https://www.youtube.com/feeds/videos.xml?playlist_id=UULF ``` So `UCabc123...` → `https://www.youtube.com/feeds/videos.xml?playlist_id=UULFabc123...` Related prefixes: `UU` all uploads, `UUSH` shorts, `UULV` livestreams, `UUMF`/`UUMO` members-only. ### The filtering is real — verified Tested on 2026-08-11 against the three subscription targets: | Channel | `UULF` | `UUSH` | `UULV` | Shorts leaking into `UULF` | |---|---|---|---|---| | clabretro | 200, 15 entries | 200, 2 entries | 200, 1 entry | **0** | | EthosLab | 200, 15 entries | 404 | 404 | n/a | | DolanDarkest | 200, 15 entries | 404 | 404 | n/a | clabretro is the useful case: it genuinely has 2 Shorts and 1 livestream, and **none of them appear in the `UULF` feed**. The primary discovery path does what it claims. Note also that **`UUSH`/`UULV` return HTTP 404 when a channel has none** — that means "empty", not "broken". Only a `UULF` 404 (or an empty `UULF` result) triggers the fallback. ### Robustness These prefixes are undocumented and there have been reports of intermittent failures and missing entries in YouTube's native feeds. So: - If the `UULF` feed 404s or returns zero entries, **fall back** to `?channel_id=UC...` and record `discovery_source = 'uc_feed'` on the resulting rows. - Rows discovered via the fallback path get the duration/live filter applied at download time (§6). Rows from `UULF` don't need it. - **A `skipped_short` row from the fallback path is re-queued if a later `UULF` poll lists that video.** `UULF` is authoritative about what is and isn't a Short, so this repairs the case where a transient feed outage would otherwise permanently drop a legitimate short-but-not-Short video. See §6 for why this matters concretely. - Track consecutive poll failures per channel and surface the count in the admin UI. Two channels quietly failing for a month is the realistic failure mode here. ### Parsing `xml.etree.ElementTree`, no dependencies. Namespaces: ```python NS = { "atom": "http://www.w3.org/2005/Atom", "yt": "http://www.youtube.com/xml/schemas/2015", "media": "http://search.yahoo.com/mrss/", } ``` Per `atom:entry`: `yt:videoId`, `atom:title`, `atom:published`, `media:group/media:description`. The feed does **not** carry duration. ### Backfill on subscribe New subscriptions pull the last `backfill_days` (default 7). **`--flat-playlist` carries no upload dates.** The original handover said to pull 50 entries with `--flat-playlist -J` and "date-filter the entries client-side". Verified against clabretro: every entry comes back with `timestamp: None`. There is no date to filter on. Entries *do* carry `id`, `title`, `duration` and `live_status`. So the backfill is RSS-first: 1. Read the `UULF` feed (falling back to `channel_id`), which is the only source that carries dates. It returns ~15 entries — enough to cover a 7-day window for any channel uploading less than twice a day, which is all three subscription targets. 2. Only if the feed's *oldest* entry is still inside the backfill window might there be more, in which case extend via `--flat-playlist --playlist-end 50` and resolve those extra dates one video at a time with `--print "%(upload_date)s"`. The playlist is reverse-chronological, so stop at the first video outside the window. In practice step 2 never runs for the current subscriptions. It exists so that raising `backfill_days` doesn't silently truncate at 15 videos. ### `skipped_old` and rescan A video discovered outside the window is `skipped_old`, which is terminal. That creates a trap: raising a channel's `retention_days` afterwards appears to do nothing, because every one of that channel's videos is already marked old. Observed with EthosLab — his most recent upload was 22 days before subscribe, so a 7-day backfill queued **nothing at all** for him. `rescan` re-queues `skipped_old` rows whose `upload_date` now falls inside the channel's current effective retention window. It is an explicit action (`poll --rescan`, or a button in the admin UI), deliberately *not* something poll does silently — doing it on every poll would make `backfill_days` meaningless, since it would immediately re-queue everything the initial backfill had just decided to leave behind. Tombstones are never touched: only `skipped_old` is eligible, never `deleted`. Verified: setting EthosLab to 60 days and rescanning queued exactly the 3 videos inside that window and left the other 12 alone. ### Getting the subscription list in Manual entry via the admin page, one channel URL at a time. With single-digit channels that is enough. No CSV importer. --- ## 5. On-disk layout Media root: `/disks/Plex/YouTube/` ``` /disks/Plex/YouTube/ ├── .work/ # scratch, ignored by Jellyfin │ └── .ignore # belt-and-braces, see below └── Some Channel/ ├── tvshow.nfo ├── poster.jpg ├── fanart.jpg └── Season 2026/ ├── Some Channel - S2026E8110 - Video Title [dQw4w9WgXcQ].mp4 ├── Some Channel - S2026E8110 - Video Title [dQw4w9WgXcQ].nfo ├── Some Channel - S2026E8110 - Video Title [dQw4w9WgXcQ]-thumb.jpg ├── Some Channel - S2026E8110 - Video Title [dQw4w9WgXcQ].en.srt └── Some Channel - S2026E8110 - Video Title [dQw4w9WgXcQ].info.json ``` The work dir is dot-prefixed (which Jellyfin skips) **and** contains an `.ignore` file, which Jellyfin also honours. Two independent mechanisms, because relying on undocumented hidden-folder behaviour for the directory holding partial downloads is not worth the risk. ### Season / episode numbering - **Season** = upload year (`2026`) - **Episode** = `MMDD * 10 + ordinal_within_day` So the first video uploaded on 11 August is `8110`, the second that day is `8111`, and the first on 12 August is `8120`. This sorts correctly across the whole year — 1 Jan is `1010` and 31 Dec is `12310` — and tolerates up to 10 uploads per channel per day. Clamp the ordinal at 9 and log a warning if exceeded. The ordinal must be computed against **what's already in the DB for that channel+date**, not against the current batch, so it's stable across runs. Write the episode number **unpadded** (`E8110`), matching the `` value in the NFO exactly. ### Filename sanitisation Strip `/ \ : * ? " < > |`, collapse runs of whitespace, strip leading/trailing dots and spaces, truncate the title component to 120 chars on a word boundary. The `[videoid]` suffix guarantees uniqueness regardless. Store the channel directory name in the DB (`channel.dir_name`) at subscribe time and never recompute it — channels rename themselves and we don't want orphaned directories. --- ## 6. Download ### Format selection — read this bit carefully susan has no AVX and cannot realistically transcode. YouTube serves 720p as VP9 or AV1 by default. If a client can't direct-play those, Jellyfin will try to transcode and it will be miserable. So we force h264 + AAC in mp4 at download time: ``` -f "bv*[height<=720]+ba/b[height<=720]" -S "vcodec:h264,res:720,acodec:aac" --merge-output-format mp4 ``` **The sort field order matters and the original handover had it wrong.** With `-S "vcodec:h264,acodec:aac,res:720"` — `acodec` ranked above `res` — this selects the legacy combined **360p** format 18 rather than 720p. `bv*` means "best video, possibly with audio", so ranking `acodec:aac` before `res` makes a combined 360p stream (which has AAC) outrank a 720p video-only stream (whose `acodec` is `none`). Verified empirically: the original ordering yields `18 | 640x360`, the corrected ordering yields `298+140 | 1280x720 | avc1 | mp4a`. Keep `vcodec` **first**, ahead of `res`. If a video has no h264 at 720p but does at 480p, this picks h264 at 480p — which direct-plays — rather than 720p VP9, which would force a transcode. On hardware that cannot transcode, codec fidelity beats resolution. Log any case that still falls back to VP9/AV1; they should be rare. Do **not** use `--sponsorblock-remove` — it requires cutting and re-encoding. Use `--sponsorblock-mark all` with `--embed-chapters`, which writes chapter markers only and is free. ### Full invocation ``` yt-dlp \ --extractor-args "youtube:player_client=default,mweb" \ --extractor-args "youtubepot-bgutilhttp:base_url=http://127.0.0.1:4416" \ -f "bv*[height<=720]+ba/b[height<=720]" \ -S "vcodec:h264,res:720,acodec:aac" \ --merge-output-format mp4 \ --no-playlist \ --write-info-json \ --write-thumbnail --convert-thumbnails jpg \ --write-subs --write-auto-subs --sub-langs "en.*" --convert-subs srt \ --sponsorblock-mark all --embed-chapters \ --retries 3 --fragment-retries 10 \ --sleep-requests 2 --sleep-interval 5 --max-sleep-interval 15 \ -P "/disks/Plex/YouTube/.work" \ -o "%(id)s.%(ext)s" \ "https://www.youtube.com/watch?v=" ``` `--write-subs` is included alongside `--write-auto-subs` so channels with real uploaded subtitles aren't silently skipped in favour of nothing. **Subtitle de-duplication.** `--sub-langs "en.*"` matches both `en` and `en-orig`, so a typical video yields *two* sidecars (`.en.srt` and `.en-orig.srt`) which Jellyfin would present as two identical "English" tracks. Observed on the test download. In the move-into-place step, keep `.en.srt` if present and discard `.en-orig.srt`; if only `en-orig` exists, rename it to `.en.srt`. ### Match filter (fallback-discovered videos only) For rows with `discovery_source = 'uc_feed'`, append: ``` --match-filter "duration>?{min_duration_seconds} & live_status!=?is_live & live_status!=?is_upcoming & !was_live" ``` The `>?` and `!=?` forms allow videos with unknown values through rather than rejecting them — an absent `live_status` should not disqualify a video any more than an absent duration does. `min_duration_seconds` defaults to **120**. ### Why 120 is a knowingly tight default DolanDarkest — one of the three subscription targets — uploads daily videos of **122–167 seconds**. One of them clears the filter by two seconds. Those videos arrive via `UULF`, which is exempt from the match filter, so in normal operation this is fine. The risk is a transient `UULF` outage: rows then get `discovery_source='uc_feed'`, the duration filter applies, and anything under two minutes would be dropped. This is why §4 re-queues a `skipped_short` row when a later `UULF` poll lists the video. With that repair in place the only residual loss case is a sub-120s video whose `UULF` feed never recovers inside the retention window, which would also be visible in the poll-failure counter. A per-channel `min_duration_seconds` override was considered and declined; revisit if DolanDarkest starts posting sub-two-minute videos regularly. ### Skip and defer semantics | Outcome | State | Retried? | |---|---|---| | Below `min_duration_seconds` | `skipped_short` | Only if a later `UULF` poll lists it (§4) | | `is_live` or `was_live` | `skipped_live` | Never — livestreams are not wanted | | `is_upcoming` | `deferred` | **Yes** — re-queued by later polls | `is_upcoming` must not be a permanent skip. A scheduled premiere becomes an ordinary downloadable video once it airs, so permanently rejecting it would silently lose content from any channel that uses premieres. Give up on a `deferred` row once it falls outside the retention window. ### Concurrency and politeness One download at a time. Single-digit channels over a 14-day window is a small workload; there is no reason to be aggressive and every reason not to be. ### After a successful download 1. Read the `.info.json` from `.work/` 2. Compute season/episode/filename 3. Generate the episode `.nfo` (§7) 4. `os.rename()` all artefacts into the season directory 5. Update the DB row: `state='downloaded'`, `rel_path`, `downloaded_at`, **`size_bytes`** 6. After the whole batch, trigger a Jellyfin library refresh (§7) `size_bytes` is required — `disk_cap_gb` cannot work without it. If any step 1–4 fails, clean up `.work/` for that video ID and mark `failed` with `attempts += 1`. Give up after `max_attempts` (5) and surface it in the UI. --- ## 7. Metadata — NFO files, no Jellyfin plugin Jellyfin reads Kodi-style NFO sidecars natively. The library is created as **Shows** with all internet metadata providers disabled, *Prefer local metadata* on, and *Save artwork/metadata into media folders* on. Do not write or install a metadata provider plugin. The library is created programmatically via `POST /Library/VirtualFolders` using the stored admin API key, so the setup is reproducible rather than a hand-clicked state. ### `tvshow.nfo` (per channel, written at subscribe time, refreshed on title change) ```xml Some Channel Channel description from yt-dlp. YouTube UCabc123... ``` ### Episode `.nfo` (one per video, filename matches the media file) ```xml Video Title Some Channel 2026 8110 Video description. 2026-08-11 12 YouTube dQw4w9WgXcQ ``` `runtime` is in **minutes**. Build these with `xml.etree`'s serialiser, never string formatting — video descriptions are hostile input and contain everything. ### Artwork At subscribe time, `yt-dlp --flat-playlist --playlist-items 0 -J ` returns a `thumbnails` array containing entries with `id` values like `avatar_uncropped` and `banner_uncropped`. Download the avatar to `poster.jpg` and the banner to `fanart.jpg`. Treat both as best-effort — if they're missing, carry on without them. Per-episode thumbnails come from `--write-thumbnail`; rename to `-thumb.jpg`. ### Library refresh `POST {jellyfin_url}/Library/Refresh` with header `X-Emby-Token: {api_key}`, triggered after: - a download batch that produced at least one new file - a reap that deleted at least one file - an unsubscribe Refreshing after deletions matters — without it Jellyfin shows ghost episodes until its own scheduled scan. --- ## 8. Data model SQLite at `/var/lib/youtube-automate/subs.db`. **WAL mode** — the cron job and the web server both write. ```sql PRAGMA journal_mode = WAL; PRAGMA foreign_keys = ON; CREATE TABLE channel ( id INTEGER PRIMARY KEY, channel_id TEXT NOT NULL UNIQUE, -- UC... handle TEXT, -- @handle, informational title TEXT NOT NULL, description TEXT, dir_name TEXT NOT NULL UNIQUE, -- sanitised, immutable after creation added_at TEXT NOT NULL, backfilled INTEGER NOT NULL DEFAULT 0, retention_days INTEGER, -- NULL = use global setting last_polled_at TEXT, last_poll_ok INTEGER, consecutive_poll_failures INTEGER NOT NULL DEFAULT 0 ); CREATE TABLE video ( id INTEGER PRIMARY KEY, video_id TEXT NOT NULL UNIQUE, channel_pk INTEGER NOT NULL REFERENCES channel(id) ON DELETE CASCADE, title TEXT, upload_date TEXT, -- YYYY-MM-DD duration INTEGER, season INTEGER, episode INTEGER, state TEXT NOT NULL, discovery_source TEXT NOT NULL, -- 'uulf_feed' | 'uc_feed' | 'backfill' rel_path TEXT, -- relative to media root, NULL unless downloaded size_bytes INTEGER, attempts INTEGER NOT NULL DEFAULT 0, last_error TEXT, discovered_at TEXT NOT NULL, downloaded_at TEXT, deleted_at TEXT ); CREATE INDEX idx_video_state ON video(state); CREATE INDEX idx_video_upload_date ON video(upload_date); CREATE INDEX idx_video_channel ON video(channel_pk); CREATE TABLE setting ( key TEXT PRIMARY KEY, value TEXT NOT NULL ); ``` Schema changes are applied by an idempotent migration function keyed on `PRAGMA user_version`. ### `video.state` values | State | Meaning | Retried? | |---|---|---| | `pending` | discovered, queued | — | | `downloading` | claimed by a worker | recovered on startup → `pending` | | `downloaded` | on disk, `rel_path` set | — | | `deleted` | aged out; tombstone prevents re-download | **never** | | `deferred` | premiere/upcoming, not yet available | yes, on later polls | | `skipped_short` | rejected by match filter | only if later seen in `UULF` (§4) | | `skipped_live` | rejected by match filter | never | | `skipped_old` | discovered but already outside the window | never | | `failed` | download error | yes, up to `max_attempts` | The tombstone behaviour is the important part: a `deleted` row must never be re-discovered and re-downloaded, and the `skipped_short` repair in §4 must never resurrect one. This is why we keep our own state table rather than using yt-dlp's `--download-archive`. ### Settings Store as strings; provide typed accessors with defaults so a missing key never crashes. | Key | Default | Notes | |---|---|---| | `retention_days` | `14` | global default; per-channel override wins | | `backfill_days` | `7` | window pulled when a channel is added | | `max_height` | `720` | | | `min_duration_seconds` | `120` | fallback-path Shorts filter | | `sponsorblock_mark` | `true` | | | `write_subs` | `true` | | | `sub_langs` | `en.*` | | | `jellyfin_url` | `http://127.0.0.1:8096` | | | `jellyfin_api_key` | *(empty)* | masked in the UI; set via CLI | | `pot_provider_url` | `http://127.0.0.1:4416` | | | `max_attempts` | `5` | | | `disk_cap_gb` | `0` | `0` = disabled; if set, evict oldest downloaded videos | Validate on save (integers parse, URLs well-formed) and re-render the form with an inline error rather than 500-ing. Two further keys are **not** editable through the settings form and are never rendered: `admin_password_hash` (see §11) and `session_secret` (generated on first run). **No secret is committed to the repo.** The Jellyfin API key and the admin password are set with CLI subcommands and live only in the DB. --- ## 9. Entry point and scheduling `/usr/local/bin/youtube-automate` is a shim that execs the venv interpreter against the package. Subcommands: | Command | Purpose | |---|---| | `run [--channel ID]` | poll → download → reap. This is what cron calls. | | `poll [--channel ID]` | discovery only | | `download` | drain the pending queue | | `reap` | retention pass | | `serve` | admin HTTP server (systemd unit) | | `subscribe ` / `unsubscribe ` | CLI equivalents, useful for debugging | | `set-password` | prompt for and store the admin password (never on the command line) | | `set-jellyfin-key` | store the Jellyfin API key | | `setup-jellyfin-library` | create/verify the YouTube Shows library with the right options | | `doctor` | yt-dlp version, POT provider reachable, DB writable, media root writable, Jellyfin reachable, permissions sane | `run` takes a **non-blocking `flock`** on `/var/lib/youtube-automate/run.lock` and exits 0 silently if already held. On startup it resets any `downloading` rows to `pending` and clears matching orphans out of `.work/` (crash recovery). `run` accepts `--channel` so the web subscribe handler can spawn an immediate backfill rather than waiting up to an hour. ### Cron (susan's crontab, alongside the existing entries) ``` 17 * * * * runitor -uuid 41a4d61a-7743-43d9-9b5d-d37d536e4726 -- sg mediaserver "/usr/local/bin/youtube-automate run" 40 4 * * 1 runitor -uuid 721e4cf0-d796-48e7-a184-79d21e1ba373 -- /var/lib/youtube-automate/venv/bin/python -m uv pip install --upgrade yt-dlp ``` Hourly is ample for single-digit channels; both checks are registered in Healthchecks with generous grace periods. The weekly updater is monitored separately so a stale yt-dlp alerts distinctly from a failed run — §12 calls that the top operational risk. ### systemd unit for the admin server `Type=simple`, `Restart=always`, `User=susan`, `Group=mediaserver`, `UMask=0002`. Binds `127.0.0.1:8085` only. ### nginx `tube.jihakuz.xyz` → `127.0.0.1:8085`, public, TLS via certbot. The DNS record and njal.la update key already exist in `update-dns.sh`, so no DNS work is needed. **The hostname was already claimed.** A leftover TubeArchivist server block inside `sites-available/jihakuz.xyz` served `tube.jihakuz.xyz`, proxying to the now-dead `127.0.0.1:8003`. nginx uses the *first* server block matching a name and that file loads before any standalone `tube.jihakuz.xyz` file, so installing a second vhost changed nothing and every request returned **502**. The old block also already owns the Let's Encrypt certificate for the hostname, so the fix is to repoint it rather than duplicate it — `deploy/fix-nginx-tube.sh` does that, and removes the redundant standalone vhost. The forwarding headers are not cosmetic: the app throttles failed logins per client address and reads `X-Forwarded-For`. Without it every attempt appears to come from nginx itself, so a single attacker would lock out everyone. Because the UI is public, the app carries real CSRF tokens and treats every input as hostile (§11). --- ## 10. Retention and deletion ### Aging out (`reap`) Effective retention for a video = its channel's `retention_days` if set, else the global `retention_days`. Candidates: `state = 'downloaded'` and `upload_date < today - effective`. Per-channel overrides exist because the subscription mix spans wildly different cadences. EthosLab uploads roughly every two to three weeks (20 Jul, 1 Jul, 19 Jun, 1 Jun, 11 May), so at a flat 14 days his show would sit empty in Jellyfin more often than not. Deletion removes the media file plus its `.nfo`, `-thumb.jpg`, `.srt` and `.info.json` siblings, then prunes the season directory if it's empty, and the channel directory if that leaves it bare of seasons. Set `state='deleted'`, `deleted_at`, and `rel_path=NULL`. **Keep the row** — it's the tombstone. Trigger a Jellyfin refresh if anything was deleted. Reap is a purely local operation and does not require Jellyfin to be reachable. Watch-state protection was considered and declined (§15), so a video can in principle vanish mid-watch; the mitigation available is raising that channel's `retention_days`. ### Unsubscribe Hard delete: `shutil.rmtree()` the channel directory, then `DELETE FROM channel` (cascades to `video`). Re-subscribing later starts from scratch and re-downloads the backfill window. The UI requires a confirmation step — this is destructive and irreversible. ### Disk cap (optional, `disk_cap_gb`) If enabled, after each reap sum `size_bytes` over `downloaded` rows and evict oldest-by- `upload_date` until under the cap. --- ## 11. Admin UI Stdlib only — `http.server.ThreadingHTTPServer` + `BaseHTTPRequestHandler`. No Flask, no FastAPI, no npm. Server-rendered HTML with a single embedded `