# `ytsubs` — implementation handover **Target machine:** `susan` **Audience:** a Claude Code agent implementing this from scratch **Status:** design agreed, not started --- ## 1. What we're building A self-hosted pipeline that keeps a rolling 14-day window of recent YouTube uploads from a handful of subscribed channels, laid out on disk so Jellyfin presents each channel as a TV show and each video as an episode. A small web UI lets a non-operator (the requester's brother) subscribe, unsubscribe, and tweak settings. The mental model is **a DVR for YouTube subscriptions**, not an archiver. Videos are disposable. If a video ages out or a channel is unsubscribed, it goes away permanently. ### Explicitly in scope - Poll subscribed channels for new uploads - Download at ≤720p, h264/AAC, into a Jellyfin-friendly layout - Generate Kodi-style `.nfo` metadata + artwork (no Jellyfin plugin) - Delete videos older than the retention window - Web admin: add/remove channels, edit settings - Cron-driven, idempotent, monitored via Healthchecks ### Explicitly out of scope — do not build these - **Cookie handling of any kind.** No browser extension, no cookie-receiving API, no `cookies.txt`. See §3. - Members-only, age-restricted, or private content (follows from the above) - Migration from the existing TubeArchivist install - Multi-user support, authentication beyond a single shared credential - Transcoding of any kind (see §6 — this matters on this hardware) --- ## 2. Environment facts you need - **susan**: Linux rack server, dual Xeon X58, 12 threads, **no AVX**. Old and slow. Software video transcoding is effectively off the table. - Already running on susan: Jellyfin, Docker, nginx (Let's Encrypt), Netdata, Healthchecks (`hc.jihakuz.xyz`), `runitor` for cron wrapping. - susan is on a **residential IP**. This matters — YouTube flags datacenter IPs far more aggressively. Do not move any part of this to `victoria` (the Linode VPS). - Tailnet: Headscale at `headscale.jihakuz.xyz`. The admin UI should bind to the tailnet interface rather than being published publicly (see §9). - `/disks/Plex/YouTube` is **already in use by TubeArchivist**. Do not touch it. Use a fresh root (§5). - Existing conventions to match: state under `/var/lib//`, deployed executables under `/usr/local/bin/`, bare git repos under `/disks/git-repos/`. ### Proposed paths | Purpose | Path | |---|---| | Source (bare repo) | `/disks/git-repos/youtube-automate.git` | | Checkout | `/opt/youtube-automate` | | Entry point | `/usr/local/bin/youtube-automate` | | State DB | `/var/lib/youtube-automate/subs.db` | | Lock file | `/var/lib/youtube-automate/run.lock` | | Media root | `/disks/Plex/YouTube/` | | Scratch/work dir | `/disks/Plex/YouTube/.ingest/` | The work dir **must** be on the same filesystem as the media root so finished downloads move into place with an atomic `rename()` rather than a copy. Jellyfin should be configured to ignore `.ingest` (it's dot-prefixed, which Jellyfin skips by default — verify). --- ## 3. YouTube access: PO tokens, not cookies This was the main open question in design and the answer is that it collapses. yt-dlp's current recommended setup for reliable downloads is the **mweb client plus a PO Token Provider plugin** supplying tokens for GVS (Google Video Server) requests. No account, no cookies, no browser. Cookies only unlock age-restricted / members-only / private content, which is out of scope, and carry a real risk of the Google account being banned. ### Setup 1. Run the provider as a Docker sidecar: !! tom: needs a way of starting up when system reboots ``` docker run --name bgutil-provider -d --restart unless-stopped --init \ -p 127.0.0.1:4416:4416 brainicism/bgutil-ytdlp-pot-provider ``` 2. Install the matching plugin into the same Python environment as yt-dlp: !! tom: do this in uv venv ``` pip install bgutil-ytdlp-pot-provider ``` The plugin and server versions should match. 3. Verify before writing any other code: ``` yt-dlp -v 'https://www.youtube.com/watch?v=' -F ``` You must see a line like `[debug] [youtube] [pot] PO Token Providers: bgutil:http-1.x.x (external), ...` If you don't, stop and fix this first — everything downstream depends on it. 4. `ytsubs` should health-check the provider before each download batch (a plain HTTP GET against `http://127.0.0.1:4416/ping`) and fail loudly if it's down, rather than silently accumulating 403s. ### Caveats to be aware of - The bgutil README carries a warning that PO tokens no longer bypass the "Sign in to confirm you're not a bot" interstitial in most cases. That is a **different failure mode** (IP reputation) from the 403s on format URLs that this solves. Residential IP + low request volume + sleep intervals should keep us clear of it. If it does start firing, that's the point at which cookies get reconsidered — not before. - **yt-dlp's YouTube handling churns every few weeks.** Client recommendations, extractor-arg names, and which clients need tokens all move. Before implementing, read the current `PO Token Guide` and `Extractors` pages on the yt-dlp wiki and adjust the flags in §6 accordingly. Do not treat the flags in this document as authoritative — treat them as the shape of the answer. - Keep yt-dlp itself updated (weekly `pip install -U yt-dlp` under the same runitor wrapper). A stale yt-dlp is the single most likely cause of "everything broke". --- ## 4. Discovery: RSS feeds YouTube publishes an unauthenticated Atom feed per channel. Use the **undocumented `UULF` playlist variant**, which returns long-form videos only — no Shorts, no livestreams: ``` https://www.youtube.com/feeds/videos.xml?playlist_id=UULF ``` So `UCabc123...` → `https://www.youtube.com/feeds/videos.xml?playlist_id=UULFabc123...` This does most of the Shorts/livestream filtering for free, at the cheapest possible point in the pipeline. Related prefixes, for reference: `UU` all uploads, `UUSH` shorts, `UULV` livestreams, `UUMF`/`UUMO` members-only. ### Robustness These prefixes are undocumented and there have been reports through 2026 of intermittent failures and missing entries in YouTube's native feeds. So: - If the `UULF` feed 404s or returns zero entries, **fall back** to `?channel_id=UC...` and record `discovery_source = 'uc_feed'` on the resulting rows. - Rows discovered via the fallback path get the duration/live filter applied at download time (§6). Rows from `UULF` don't need it. - Track consecutive poll failures per channel and surface the count in the admin UI. Two channels quietly failing for a month is the realistic failure mode here. ### Parsing `xml.etree.ElementTree`, no dependencies. Namespaces: ```python NS = { "atom": "http://www.w3.org/2005/Atom", "yt": "http://www.youtube.com/xml/schemas/2015", "media": "http://search.yahoo.com/mrss/", } ``` Per `atom:entry`: `yt:videoId`, `atom:title`, `atom:published`, `media:group/media:description`. The feed does **not** carry duration. ### Backfill on subscribe New subscriptions pull the last **7 days**. The RSS feed only returns ~15 items, which can be under 7 days for a prolific channel. For the initial backfill only, use: ``` yt-dlp --flat-playlist --playlist-end 50 -J \ 'https://www.youtube.com/playlist?list=UULF<...>' ``` and date-filter the entries client-side. Steady-state polling uses the RSS feed. ### Getting the subscription list in There is no need for a Google account or a subscriptions RSS feed. Either the brother pastes channel URLs into the admin page one at a time, or he does a one-off Google Takeout export (`subscriptions.csv`) and we bulk-insert. With single-digit channels, manual entry is fine — do not build a CSV importer unless asked. --- ## 5. On-disk layout Media root: `/disks/Plex/YouTubeSubs/` ``` /disks/Plex/YouTubeSubs/ ├── .work/ # scratch, ignored by Jellyfin └── Some Channel/ ├── tvshow.nfo ├── poster.jpg ├── fanart.jpg └── Season 2026/ ├── Some Channel - S2026E08110 - Video Title [dQw4w9WgXcQ].mp4 ├── Some Channel - S2026E08110 - Video Title [dQw4w9WgXcQ].nfo ├── Some Channel - S2026E08110 - Video Title [dQw4w9WgXcQ]-thumb.jpg ├── Some Channel - S2026E08110 - Video Title [dQw4w9WgXcQ].en.srt └── Some Channel - S2026E08110 - Video Title [dQw4w9WgXcQ].info.json ``` ### Season / episode numbering - **Season** = upload year (`2026`) - **Episode** = `MMDD * 10 + ordinal_within_day` So the first video uploaded on 11 August is `8110`, the second that day is `8111`, and the first on 12 August is `8120`. This scheme sorts correctly across the whole year (unlike appending a suffix to a bare `MMDD`) and tolerates up to 10 uploads per channel per day. Clamp the ordinal at 9 and log a warning if exceeded. The ordinal must be computed against **what's already in the DB for that channel+date**, not against the current batch, so it's stable across runs. ### Filename sanitisation Strip `/ \ : * ? " < > |`, collapse runs of whitespace, strip leading/trailing dots and spaces, truncate the title component to 120 chars on a word boundary. The `[videoid]` suffix guarantees uniqueness regardless. Store the channel directory name in the DB (`channel.dir_name`) at subscribe time and never recompute it — channels rename themselves and we don't want orphaned directories. --- ## 6. Download ### Format selection — read this bit carefully susan has no AVX and cannot realistically transcode. YouTube serves 720p as VP9 or AV1 by default. If a client can't direct-play those, Jellyfin will try to transcode and it will be miserable. So we force h264 + AAC in mp4 at download time: ``` -f "bv*[height<=720]+ba/b[height<=720]" -S "vcodec:h264,acodec:aac,res:720" --merge-output-format mp4 ``` If a video genuinely has no h264 rendition at ≤720p, `-S` will fall back to VP9 rather than fail. Log those cases; they should be rare. Do **not** use `--sponsorblock-remove` — it requires cutting and re-encoding. Use `--sponsorblock-mark all` with `--embed-chapters`, which writes chapter markers only and is free. ### Full invocation ``` yt-dlp \ --extractor-args "youtube:player_client=default,mweb" \ --extractor-args "youtubepot-bgutilhttp:base_url=http://127.0.0.1:4416" \ -f "bv*[height<=720]+ba/b[height<=720]" \ -S "vcodec:h264,acodec:aac,res:720" \ --merge-output-format mp4 \ --no-playlist \ --write-info-json \ --write-thumbnail --convert-thumbnails jpg \ --write-auto-subs --sub-langs "en.*" --convert-subs srt \ --sponsorblock-mark all --embed-chapters \ --retries 3 --fragment-retries 10 \ --sleep-requests 2 --sleep-interval 5 --max-sleep-interval 15 \ -P "/disks/Plex/YouTubeSubs/.work" \ -o "%(id)s.%(ext)s" \ "https://www.youtube.com/watch?v=" ``` Verify the `youtubepot-bgutilhttp:base_url` arg name against the current plugin README — this is the kind of thing that gets renamed. ### Match filter (fallback-discovered videos only) For rows with `discovery_source = 'uc_feed'`, append: ``` --match-filter "duration>?{min_duration_seconds} & live_status!=is_live & live_status!=is_upcoming & !was_live" ``` The `>?` form allows videos with unknown duration through rather than rejecting them. `min_duration_seconds` defaults to 120 (Shorts can now run to 3 minutes, but so can legitimate short videos — this is the tradeoff, and it's why `UULF` is the primary path). A video rejected by the match filter should be recorded as `skipped_short` / `skipped_live` and **never retried**. ### Concurrency and politeness One download at a time. Single-digit channels over a 14-day window is a small workload; there is no reason to be aggressive and every reason not to be. ### After a successful download 1. Read the `.info.json` from `.work/` 2. Compute season/episode/filename 3. Generate the episode `.nfo` (§7) 4. `os.rename()` all artefacts into the season directory 5. Update the DB row: `state='downloaded'`, `rel_path`, `downloaded_at` 6. After the whole batch, trigger a Jellyfin library refresh: `POST {jellyfin_url}/Library/Refresh` with header `X-Emby-Token: {api_key}` If any step 1–4 fails, clean up `.work/` for that video ID and mark `failed` with `attempts += 1`. Give up after 5 attempts and surface it in the UI. --- ## 7. Metadata — NFO files, no Jellyfin plugin Jellyfin reads Kodi-style NFO sidecars natively. Configure the library as **Shows**, disable all internet metadata providers, enable *Prefer local metadata* and *Save artwork/metadata into media folders*. Do not write or install a metadata provider plugin. ### `tvshow.nfo` (per channel, written at subscribe time, refreshed on title change) ```xml Some Channel Channel description from yt-dlp. YouTube UCabc123... ``` ### Episode `.nfo` (one per video, filename matches the media file) ```xml Video Title Some Channel 2026 8110 Video description. 2026-08-11 12 YouTube dQw4w9WgXcQ ``` `runtime` is in **minutes**. Escape all text content properly — video descriptions contain everything. ### Artwork At subscribe time, `yt-dlp --flat-playlist --playlist-items 0 -J ` returns a `thumbnails` array containing entries with `id` values like `avatar_uncropped` and `banner_uncropped`. Download the avatar to `poster.jpg` and the banner to `fanart.jpg`. Treat both as best-effort — if they're missing, carry on without them. Per-episode thumbnails come from `--write-thumbnail`; rename to `-thumb.jpg`. --- ## 8. Data model SQLite at `/var/lib/ytsubs/subs.db`. **Enable WAL mode** — the cron job and the web server both write. ```sql PRAGMA journal_mode = WAL; PRAGMA foreign_keys = ON; CREATE TABLE channel ( id INTEGER PRIMARY KEY, channel_id TEXT NOT NULL UNIQUE, -- UC... handle TEXT, -- @handle, informational title TEXT NOT NULL, description TEXT, dir_name TEXT NOT NULL UNIQUE, -- sanitised, immutable after creation added_at TEXT NOT NULL, backfilled INTEGER NOT NULL DEFAULT 0, last_polled_at TEXT, last_poll_ok INTEGER, consecutive_poll_failures INTEGER NOT NULL DEFAULT 0 ); CREATE TABLE video ( id INTEGER PRIMARY KEY, video_id TEXT NOT NULL UNIQUE, channel_pk INTEGER NOT NULL REFERENCES channel(id) ON DELETE CASCADE, title TEXT, upload_date TEXT, -- YYYY-MM-DD duration INTEGER, season INTEGER, episode INTEGER, state TEXT NOT NULL, discovery_source TEXT NOT NULL, -- 'uulf_feed' | 'uc_feed' | 'backfill' rel_path TEXT, -- relative to media root, NULL unless downloaded size_bytes INTEGER, attempts INTEGER NOT NULL DEFAULT 0, last_error TEXT, discovered_at TEXT NOT NULL, downloaded_at TEXT, deleted_at TEXT ); CREATE INDEX idx_video_state ON video(state); CREATE INDEX idx_video_upload_date ON video(upload_date); CREATE TABLE setting ( key TEXT PRIMARY KEY, value TEXT NOT NULL ); ``` ### `video.state` values | State | Meaning | Retried? | |---|---|---| | `pending` | discovered, queued | — | | `downloading` | claimed by a worker | recovered on startup → `pending` | | `downloaded` | on disk, `rel_path` set | — | | `deleted` | aged out; tombstone prevents re-download | never | | `skipped_short` | rejected by match filter | never | | `skipped_live` | rejected by match filter | never | | `skipped_old` | discovered but already outside the window | never | | `failed` | download error | yes, up to 5 attempts | The tombstone behaviour is the important part: a `deleted` row must never be re-discovered and re-downloaded. This is why we keep our own state table rather than using yt-dlp's `--download-archive`. ### Settings (all editable from the admin page) Store as strings; provide typed accessors with defaults so a missing key never crashes. | Key | Default | Notes | |---|---|---| | `retention_days` | `14` | delete videos with `upload_date` older than this | | `backfill_days` | `7` | window pulled when a channel is added | | `max_height` | `720` | | | `min_duration_seconds` | `120` | fallback-path Shorts filter | | `sponsorblock_mark` | `true` | | | `write_subs` | `true` | | | `sub_langs` | `en.*` | | | `respect_jellyfin_watch_state` | `true` | §10 | | `jellyfin_url` | `http://127.0.0.1:8096` | | | `jellyfin_api_key` | *(empty)* | masked in the UI | | `jellyfin_user_id` | *(empty)* | | | `pot_provider_url` | `http://127.0.0.1:4416` | | | `max_attempts` | `5` | | | `disk_cap_gb` | `0` | `0` = disabled; if set, evict oldest downloaded videos | Validate on save (integers parse, URLs well-formed) and re-render the form with an inline error rather than 500-ing. --- ## 9. Entry point and scheduling Single executable `ytsubs` with subcommands: | Command | Purpose | |---|---| | `ytsubs run` | poll → download → reap. This is what cron calls. | | `ytsubs poll [--channel ID]` | discovery only | | `ytsubs download` | drain the pending queue | | `ytsubs reap` | retention pass | | `ytsubs serve` | admin HTTP server (systemd unit) | | `ytsubs subscribe ` / `unsubscribe ` | CLI equivalents, useful for debugging | | `ytsubs doctor` | check yt-dlp version, POT provider reachable, DB writable, media root writable, Jellyfin reachable | `run` takes a **non-blocking `flock`** on `/var/lib/ytsubs/run.lock` and exits 0 silently if already held. On startup, reset any `downloading` rows to `pending` (crash recovery). ### Cron ``` 17 * * * * runitor -uuid -- /usr/local/bin/ytsubs run ``` Hourly is ample for single-digit channels. Register the check in Healthchecks with a generous grace period. ### systemd unit for the admin server Standard `simple` unit, `Restart=always`, running as the same unprivileged user that owns the DB and media root. When the user subscribes to a channel via the web UI, the handler should spawn `ytsubs run --channel ` detached so the backfill starts immediately rather than waiting up to an hour. ### nginx Bind the admin server to `127.0.0.1:8085`. Front it with nginx on the **tailnet interface only** — `yt.jihakuz.xyz` resolving inside Headscale, HTTP basic auth on top. Given the brother will be on the tailnet anyway, there's no reason to expose this publicly, and it removes the need for a Let's Encrypt cert and any real CSRF story. Confirm this before implementing; if it does need to be public, add proper CSRF tokens. --- ## 10. Retention and deletion ### Aging out (`ytsubs reap`) Candidates: `state = 'downloaded'` and `upload_date < today - retention_days`. Before deleting, if `respect_jellyfin_watch_state` is on: 1. `GET {jellyfin_url}/Items?userId={user_id}&recursive=true&includeItemTypes=Episode&fields=Path,UserData` with header `X-Emby-Token: {api_key}` — one call per reap, build a `path → UserData` map. 2. **Skip** any candidate where `UserData.PlaybackPositionTicks > 0 and not UserData.Played` (part-watched) or `UserData.IsFavorite` is true. 3. Apply a hard backstop: delete anyway once `upload_date` is older than `retention_days * 2`, so a half-watched video doesn't live forever. Nothing wrecks "set and forget" like a video disappearing at the 20-minute mark. This check is worth the complexity. Deletion removes the media file plus its `.nfo`, `-thumb.jpg`, `.srt`, and `.info.json` siblings, then prunes the season directory if empty. Set `state='deleted'`, `deleted_at`, and `rel_path=NULL`. **Keep the row** — it's the tombstone. If the Jellyfin API is unreachable, skip the whole reap for that run and log it. Do not delete blindly. ### Unsubscribe Hard delete, as agreed: `shutil.rmtree()` the channel directory, then `DELETE FROM channel` (cascades to `video`). Re-subscribing later starts from scratch and re-downloads the last 7 days. Put a confirmation step in the UI — this is destructive and irreversible. ### Disk cap (optional, `disk_cap_gb`) If enabled, after each reap, sum `size_bytes` over `downloaded` rows and evict oldest-by- `upload_date` until under the cap, honouring the same watch-state protection. --- ## 11. Admin UI Stdlib only — `http.server.ThreadingHTTPServer` + `BaseHTTPRequestHandler`. No Flask, no FastAPI, no npm. Server-rendered HTML with a single embedded `