A DVR for YouTube subscriptions, laid out so Jellyfin presents each channel as a show and each video as an episode. Cron-driven, idempotent, with a public admin UI for a non-operator. Verified end to end on susan against three real channels: PO tokens, h264 downloads, Jellyfin resolution from local NFOs with all providers disabled, retention and tombstones. Corrections to the original design handover (specs.md documents each with the evidence, and specs.handover-original.md preserves the original): - The format sort selected 360p. Ranking acodec above res makes `bv*` prefer the combined 360p stream, which carries AAC, over the 720p video-only stream whose acodec is none. vcodec now leads, so a video without h264 at 720p yields h264 lower down rather than VP9 this hardware cannot transcode. - yt-dlp now requires a JS runtime and the yt-dlp-ejs solver scripts, which only ship with the [default] extra. Without them the n challenge fails and the mweb formats disappear entirely. - --flat-playlist carries no upload dates, so the specced client-side date filter for backfill was impossible. Backfill is RSS-first. - skipped_old was terminal, so raising a channel's retention appeared to do nothing. Added an explicit rescan. - is_upcoming premieres now defer and retry instead of being skipped forever. - TubeArchivist is gone, so the media root and the tube.jihakuz.xyz vhost were both reclaimed; the latter still pointed at its dead port. 240 offline tests, no network and no real yt-dlp invocation. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
657 lines
26 KiB
Markdown
657 lines
26 KiB
Markdown
# `ytsubs` — implementation handover
|
||
|
||
**Target machine:** `susan`
|
||
**Audience:** a Claude Code agent implementing this from scratch
|
||
**Status:** design agreed, not started
|
||
|
||
---
|
||
|
||
## 1. What we're building
|
||
|
||
A self-hosted pipeline that keeps a rolling 14-day window of recent YouTube uploads from a
|
||
handful of subscribed channels, laid out on disk so Jellyfin presents each channel as a TV
|
||
show and each video as an episode. A small web UI lets a non-operator (the requester's
|
||
brother) subscribe, unsubscribe, and tweak settings.
|
||
|
||
The mental model is **a DVR for YouTube subscriptions**, not an archiver. Videos are
|
||
disposable. If a video ages out or a channel is unsubscribed, it goes away permanently.
|
||
|
||
### Explicitly in scope
|
||
|
||
- Poll subscribed channels for new uploads
|
||
- Download at ≤720p, h264/AAC, into a Jellyfin-friendly layout
|
||
- Generate Kodi-style `.nfo` metadata + artwork (no Jellyfin plugin)
|
||
- Delete videos older than the retention window
|
||
- Web admin: add/remove channels, edit settings
|
||
- Cron-driven, idempotent, monitored via Healthchecks
|
||
|
||
### Explicitly out of scope — do not build these
|
||
|
||
- **Cookie handling of any kind.** No browser extension, no cookie-receiving API, no
|
||
`cookies.txt`. See §3.
|
||
- Members-only, age-restricted, or private content (follows from the above)
|
||
- Migration from the existing TubeArchivist install
|
||
- Multi-user support, authentication beyond a single shared credential
|
||
- Transcoding of any kind (see §6 — this matters on this hardware)
|
||
|
||
---
|
||
|
||
## 2. Environment facts you need
|
||
|
||
- **susan**: Linux rack server, dual Xeon X58, 12 threads, **no AVX**. Old and slow.
|
||
Software video transcoding is effectively off the table.
|
||
- Already running on susan: Jellyfin, Docker, nginx (Let's Encrypt), Netdata,
|
||
Healthchecks (`hc.jihakuz.xyz`), `runitor` for cron wrapping.
|
||
- susan is on a **residential IP**. This matters — YouTube flags datacenter IPs far more
|
||
aggressively. Do not move any part of this to `victoria` (the Linode VPS).
|
||
- Tailnet: Headscale at `headscale.jihakuz.xyz`. The admin UI should bind to the tailnet
|
||
interface rather than being published publicly (see §9).
|
||
- `/disks/Plex/YouTube` is **already in use by TubeArchivist**. Do not touch it. Use a
|
||
fresh root (§5).
|
||
- Existing conventions to match: state under `/var/lib/<service>/`, deployed executables
|
||
under `/usr/local/bin/`, bare git repos under `/disks/git-repos/`.
|
||
|
||
### Proposed paths
|
||
|
||
| Purpose | Path |
|
||
|---|---|
|
||
| Source (bare repo) | `/disks/git-repos/youtube-automate.git` |
|
||
| Checkout | `/opt/youtube-automate` |
|
||
| Entry point | `/usr/local/bin/youtube-automate` |
|
||
| State DB | `/var/lib/youtube-automate/subs.db` |
|
||
| Lock file | `/var/lib/youtube-automate/run.lock` |
|
||
| Media root | `/disks/Plex/YouTube/` |
|
||
| Scratch/work dir | `/disks/Plex/YouTube/.ingest/` |
|
||
|
||
The work dir **must** be on the same filesystem as the media root so finished downloads
|
||
move into place with an atomic `rename()` rather than a copy. Jellyfin should be
|
||
configured to ignore `.ingest` (it's dot-prefixed, which Jellyfin skips by default —
|
||
verify).
|
||
|
||
---
|
||
|
||
## 3. YouTube access: PO tokens, not cookies
|
||
|
||
This was the main open question in design and the answer is that it collapses.
|
||
|
||
yt-dlp's current recommended setup for reliable downloads is the **mweb client plus a PO
|
||
Token Provider plugin** supplying tokens for GVS (Google Video Server) requests. No
|
||
account, no cookies, no browser. Cookies only unlock age-restricted / members-only /
|
||
private content, which is out of scope, and carry a real risk of the Google account being
|
||
banned.
|
||
|
||
### Setup
|
||
|
||
1. Run the provider as a Docker sidecar:
|
||
!! tom: needs a way of starting up when system reboots
|
||
|
||
```
|
||
docker run --name bgutil-provider -d --restart unless-stopped --init \
|
||
-p 127.0.0.1:4416:4416 brainicism/bgutil-ytdlp-pot-provider
|
||
```
|
||
|
||
2. Install the matching plugin into the same Python environment as yt-dlp:
|
||
!! tom: do this in uv venv
|
||
|
||
```
|
||
pip install bgutil-ytdlp-pot-provider
|
||
```
|
||
|
||
The plugin and server versions should match.
|
||
|
||
3. Verify before writing any other code:
|
||
|
||
```
|
||
yt-dlp -v 'https://www.youtube.com/watch?v=<some_id>' -F
|
||
```
|
||
|
||
You must see a line like
|
||
`[debug] [youtube] [pot] PO Token Providers: bgutil:http-1.x.x (external), ...`
|
||
If you don't, stop and fix this first — everything downstream depends on it.
|
||
|
||
4. `ytsubs` should health-check the provider before each download batch (a plain HTTP
|
||
GET against `http://127.0.0.1:4416/ping`) and fail loudly if it's down, rather than
|
||
silently accumulating 403s.
|
||
|
||
### Caveats to be aware of
|
||
|
||
- The bgutil README carries a warning that PO tokens no longer bypass the "Sign in to
|
||
confirm you're not a bot" interstitial in most cases. That is a **different failure
|
||
mode** (IP reputation) from the 403s on format URLs that this solves. Residential IP +
|
||
low request volume + sleep intervals should keep us clear of it. If it does start
|
||
firing, that's the point at which cookies get reconsidered — not before.
|
||
- **yt-dlp's YouTube handling churns every few weeks.** Client recommendations,
|
||
extractor-arg names, and which clients need tokens all move. Before implementing, read
|
||
the current `PO Token Guide` and `Extractors` pages on the yt-dlp wiki and adjust the
|
||
flags in §6 accordingly. Do not treat the flags in this document as authoritative —
|
||
treat them as the shape of the answer.
|
||
- Keep yt-dlp itself updated (weekly `pip install -U yt-dlp` under the same runitor
|
||
wrapper). A stale yt-dlp is the single most likely cause of "everything broke".
|
||
|
||
---
|
||
|
||
## 4. Discovery: RSS feeds
|
||
|
||
YouTube publishes an unauthenticated Atom feed per channel. Use the **undocumented
|
||
`UULF` playlist variant**, which returns long-form videos only — no Shorts, no
|
||
livestreams:
|
||
|
||
```
|
||
https://www.youtube.com/feeds/videos.xml?playlist_id=UULF<channel_id without the UC prefix>
|
||
```
|
||
|
||
So `UCabc123...` → `https://www.youtube.com/feeds/videos.xml?playlist_id=UULFabc123...`
|
||
|
||
This does most of the Shorts/livestream filtering for free, at the cheapest possible
|
||
point in the pipeline.
|
||
|
||
Related prefixes, for reference: `UU` all uploads, `UUSH` shorts, `UULV` livestreams,
|
||
`UUMF`/`UUMO` members-only.
|
||
|
||
### Robustness
|
||
|
||
These prefixes are undocumented and there have been reports through 2026 of intermittent
|
||
failures and missing entries in YouTube's native feeds. So:
|
||
|
||
- If the `UULF` feed 404s or returns zero entries, **fall back** to
|
||
`?channel_id=UC...` and record `discovery_source = 'uc_feed'` on the resulting rows.
|
||
- Rows discovered via the fallback path get the duration/live filter applied at download
|
||
time (§6). Rows from `UULF` don't need it.
|
||
- Track consecutive poll failures per channel and surface the count in the admin UI. Two
|
||
channels quietly failing for a month is the realistic failure mode here.
|
||
|
||
### Parsing
|
||
|
||
`xml.etree.ElementTree`, no dependencies. Namespaces:
|
||
|
||
```python
|
||
NS = {
|
||
"atom": "http://www.w3.org/2005/Atom",
|
||
"yt": "http://www.youtube.com/xml/schemas/2015",
|
||
"media": "http://search.yahoo.com/mrss/",
|
||
}
|
||
```
|
||
|
||
Per `atom:entry`: `yt:videoId`, `atom:title`, `atom:published`,
|
||
`media:group/media:description`. The feed does **not** carry duration.
|
||
|
||
### Backfill on subscribe
|
||
|
||
New subscriptions pull the last **7 days**. The RSS feed only returns ~15 items, which
|
||
can be under 7 days for a prolific channel. For the initial backfill only, use:
|
||
|
||
```
|
||
yt-dlp --flat-playlist --playlist-end 50 -J \
|
||
'https://www.youtube.com/playlist?list=UULF<...>'
|
||
```
|
||
|
||
and date-filter the entries client-side. Steady-state polling uses the RSS feed.
|
||
|
||
### Getting the subscription list in
|
||
|
||
There is no need for a Google account or a subscriptions RSS feed. Either the brother
|
||
pastes channel URLs into the admin page one at a time, or he does a one-off Google
|
||
Takeout export (`subscriptions.csv`) and we bulk-insert. With single-digit channels,
|
||
manual entry is fine — do not build a CSV importer unless asked.
|
||
|
||
---
|
||
|
||
## 5. On-disk layout
|
||
|
||
Media root: `/disks/Plex/YouTubeSubs/`
|
||
|
||
```
|
||
/disks/Plex/YouTubeSubs/
|
||
├── .work/ # scratch, ignored by Jellyfin
|
||
└── Some Channel/
|
||
├── tvshow.nfo
|
||
├── poster.jpg
|
||
├── fanart.jpg
|
||
└── Season 2026/
|
||
├── Some Channel - S2026E08110 - Video Title [dQw4w9WgXcQ].mp4
|
||
├── Some Channel - S2026E08110 - Video Title [dQw4w9WgXcQ].nfo
|
||
├── Some Channel - S2026E08110 - Video Title [dQw4w9WgXcQ]-thumb.jpg
|
||
├── Some Channel - S2026E08110 - Video Title [dQw4w9WgXcQ].en.srt
|
||
└── Some Channel - S2026E08110 - Video Title [dQw4w9WgXcQ].info.json
|
||
```
|
||
|
||
### Season / episode numbering
|
||
|
||
- **Season** = upload year (`2026`)
|
||
- **Episode** = `MMDD * 10 + ordinal_within_day`
|
||
|
||
So the first video uploaded on 11 August is `8110`, the second that day is `8111`, and
|
||
the first on 12 August is `8120`. This scheme sorts correctly across the whole year
|
||
(unlike appending a suffix to a bare `MMDD`) and tolerates up to 10 uploads per channel
|
||
per day. Clamp the ordinal at 9 and log a warning if exceeded.
|
||
|
||
The ordinal must be computed against **what's already in the DB for that channel+date**,
|
||
not against the current batch, so it's stable across runs.
|
||
|
||
### Filename sanitisation
|
||
|
||
Strip `/ \ : * ? " < > |`, collapse runs of whitespace, strip leading/trailing dots and
|
||
spaces, truncate the title component to 120 chars on a word boundary. The `[videoid]`
|
||
suffix guarantees uniqueness regardless.
|
||
|
||
Store the channel directory name in the DB (`channel.dir_name`) at subscribe time and
|
||
never recompute it — channels rename themselves and we don't want orphaned directories.
|
||
|
||
---
|
||
|
||
## 6. Download
|
||
|
||
### Format selection — read this bit carefully
|
||
|
||
susan has no AVX and cannot realistically transcode. YouTube serves 720p as VP9 or AV1 by
|
||
default. If a client can't direct-play those, Jellyfin will try to transcode and it will
|
||
be miserable. So we force h264 + AAC in mp4 at download time:
|
||
|
||
```
|
||
-f "bv*[height<=720]+ba/b[height<=720]"
|
||
-S "vcodec:h264,acodec:aac,res:720"
|
||
--merge-output-format mp4
|
||
```
|
||
|
||
If a video genuinely has no h264 rendition at ≤720p, `-S` will fall back to VP9 rather
|
||
than fail. Log those cases; they should be rare.
|
||
|
||
Do **not** use `--sponsorblock-remove` — it requires cutting and re-encoding. Use
|
||
`--sponsorblock-mark all` with `--embed-chapters`, which writes chapter markers only and
|
||
is free.
|
||
|
||
### Full invocation
|
||
|
||
```
|
||
yt-dlp \
|
||
--extractor-args "youtube:player_client=default,mweb" \
|
||
--extractor-args "youtubepot-bgutilhttp:base_url=http://127.0.0.1:4416" \
|
||
-f "bv*[height<=720]+ba/b[height<=720]" \
|
||
-S "vcodec:h264,acodec:aac,res:720" \
|
||
--merge-output-format mp4 \
|
||
--no-playlist \
|
||
--write-info-json \
|
||
--write-thumbnail --convert-thumbnails jpg \
|
||
--write-auto-subs --sub-langs "en.*" --convert-subs srt \
|
||
--sponsorblock-mark all --embed-chapters \
|
||
--retries 3 --fragment-retries 10 \
|
||
--sleep-requests 2 --sleep-interval 5 --max-sleep-interval 15 \
|
||
-P "/disks/Plex/YouTubeSubs/.work" \
|
||
-o "%(id)s.%(ext)s" \
|
||
"https://www.youtube.com/watch?v=<VIDEO_ID>"
|
||
```
|
||
|
||
Verify the `youtubepot-bgutilhttp:base_url` arg name against the current plugin README —
|
||
this is the kind of thing that gets renamed.
|
||
|
||
### Match filter (fallback-discovered videos only)
|
||
|
||
For rows with `discovery_source = 'uc_feed'`, append:
|
||
|
||
```
|
||
--match-filter "duration>?{min_duration_seconds} & live_status!=is_live & live_status!=is_upcoming & !was_live"
|
||
```
|
||
|
||
The `>?` form allows videos with unknown duration through rather than rejecting them.
|
||
`min_duration_seconds` defaults to 120 (Shorts can now run to 3 minutes, but so can
|
||
legitimate short videos — this is the tradeoff, and it's why `UULF` is the primary path).
|
||
|
||
A video rejected by the match filter should be recorded as `skipped_short` /
|
||
`skipped_live` and **never retried**.
|
||
|
||
### Concurrency and politeness
|
||
|
||
One download at a time. Single-digit channels over a 14-day window is a small workload;
|
||
there is no reason to be aggressive and every reason not to be.
|
||
|
||
### After a successful download
|
||
|
||
1. Read the `.info.json` from `.work/`
|
||
2. Compute season/episode/filename
|
||
3. Generate the episode `.nfo` (§7)
|
||
4. `os.rename()` all artefacts into the season directory
|
||
5. Update the DB row: `state='downloaded'`, `rel_path`, `downloaded_at`
|
||
6. After the whole batch, trigger a Jellyfin library refresh:
|
||
`POST {jellyfin_url}/Library/Refresh` with header `X-Emby-Token: {api_key}`
|
||
|
||
If any step 1–4 fails, clean up `.work/` for that video ID and mark `failed` with
|
||
`attempts += 1`. Give up after 5 attempts and surface it in the UI.
|
||
|
||
---
|
||
|
||
## 7. Metadata — NFO files, no Jellyfin plugin
|
||
|
||
Jellyfin reads Kodi-style NFO sidecars natively. Configure the library as **Shows**,
|
||
disable all internet metadata providers, enable *Prefer local metadata* and *Save
|
||
artwork/metadata into media folders*. Do not write or install a metadata provider plugin.
|
||
|
||
### `tvshow.nfo` (per channel, written at subscribe time, refreshed on title change)
|
||
|
||
```xml
|
||
<?xml version="1.0" encoding="utf-8"?>
|
||
<tvshow>
|
||
<title>Some Channel</title>
|
||
<plot>Channel description from yt-dlp.</plot>
|
||
<studio>YouTube</studio>
|
||
<uniqueid type="youtube" default="true">UCabc123...</uniqueid>
|
||
</tvshow>
|
||
```
|
||
|
||
### Episode `.nfo` (one per video, filename matches the media file)
|
||
|
||
```xml
|
||
<?xml version="1.0" encoding="utf-8"?>
|
||
<episodedetails>
|
||
<title>Video Title</title>
|
||
<showtitle>Some Channel</showtitle>
|
||
<season>2026</season>
|
||
<episode>8110</episode>
|
||
<plot>Video description.</plot>
|
||
<aired>2026-08-11</aired>
|
||
<runtime>12</runtime>
|
||
<studio>YouTube</studio>
|
||
<uniqueid type="youtube" default="true">dQw4w9WgXcQ</uniqueid>
|
||
</episodedetails>
|
||
```
|
||
|
||
`runtime` is in **minutes**. Escape all text content properly — video descriptions
|
||
contain everything.
|
||
|
||
### Artwork
|
||
|
||
At subscribe time, `yt-dlp --flat-playlist --playlist-items 0 -J <channel_url>` returns a
|
||
`thumbnails` array containing entries with `id` values like `avatar_uncropped` and
|
||
`banner_uncropped`. Download the avatar to `poster.jpg` and the banner to `fanart.jpg`.
|
||
Treat both as best-effort — if they're missing, carry on without them.
|
||
|
||
Per-episode thumbnails come from `--write-thumbnail`; rename to `<basename>-thumb.jpg`.
|
||
|
||
---
|
||
|
||
## 8. Data model
|
||
|
||
SQLite at `/var/lib/ytsubs/subs.db`. **Enable WAL mode** — the cron job and the web
|
||
server both write.
|
||
|
||
```sql
|
||
PRAGMA journal_mode = WAL;
|
||
PRAGMA foreign_keys = ON;
|
||
|
||
CREATE TABLE channel (
|
||
id INTEGER PRIMARY KEY,
|
||
channel_id TEXT NOT NULL UNIQUE, -- UC...
|
||
handle TEXT, -- @handle, informational
|
||
title TEXT NOT NULL,
|
||
description TEXT,
|
||
dir_name TEXT NOT NULL UNIQUE, -- sanitised, immutable after creation
|
||
added_at TEXT NOT NULL,
|
||
backfilled INTEGER NOT NULL DEFAULT 0,
|
||
last_polled_at TEXT,
|
||
last_poll_ok INTEGER,
|
||
consecutive_poll_failures INTEGER NOT NULL DEFAULT 0
|
||
);
|
||
|
||
CREATE TABLE video (
|
||
id INTEGER PRIMARY KEY,
|
||
video_id TEXT NOT NULL UNIQUE,
|
||
channel_pk INTEGER NOT NULL REFERENCES channel(id) ON DELETE CASCADE,
|
||
title TEXT,
|
||
upload_date TEXT, -- YYYY-MM-DD
|
||
duration INTEGER,
|
||
season INTEGER,
|
||
episode INTEGER,
|
||
state TEXT NOT NULL,
|
||
discovery_source TEXT NOT NULL, -- 'uulf_feed' | 'uc_feed' | 'backfill'
|
||
rel_path TEXT, -- relative to media root, NULL unless downloaded
|
||
size_bytes INTEGER,
|
||
attempts INTEGER NOT NULL DEFAULT 0,
|
||
last_error TEXT,
|
||
discovered_at TEXT NOT NULL,
|
||
downloaded_at TEXT,
|
||
deleted_at TEXT
|
||
);
|
||
|
||
CREATE INDEX idx_video_state ON video(state);
|
||
CREATE INDEX idx_video_upload_date ON video(upload_date);
|
||
|
||
CREATE TABLE setting (
|
||
key TEXT PRIMARY KEY,
|
||
value TEXT NOT NULL
|
||
);
|
||
```
|
||
|
||
### `video.state` values
|
||
|
||
| State | Meaning | Retried? |
|
||
|---|---|---|
|
||
| `pending` | discovered, queued | — |
|
||
| `downloading` | claimed by a worker | recovered on startup → `pending` |
|
||
| `downloaded` | on disk, `rel_path` set | — |
|
||
| `deleted` | aged out; tombstone prevents re-download | never |
|
||
| `skipped_short` | rejected by match filter | never |
|
||
| `skipped_live` | rejected by match filter | never |
|
||
| `skipped_old` | discovered but already outside the window | never |
|
||
| `failed` | download error | yes, up to 5 attempts |
|
||
|
||
The tombstone behaviour is the important part: a `deleted` row must never be
|
||
re-discovered and re-downloaded. This is why we keep our own state table rather than
|
||
using yt-dlp's `--download-archive`.
|
||
|
||
### Settings (all editable from the admin page)
|
||
|
||
Store as strings; provide typed accessors with defaults so a missing key never crashes.
|
||
|
||
| Key | Default | Notes |
|
||
|---|---|---|
|
||
| `retention_days` | `14` | delete videos with `upload_date` older than this |
|
||
| `backfill_days` | `7` | window pulled when a channel is added |
|
||
| `max_height` | `720` | |
|
||
| `min_duration_seconds` | `120` | fallback-path Shorts filter |
|
||
| `sponsorblock_mark` | `true` | |
|
||
| `write_subs` | `true` | |
|
||
| `sub_langs` | `en.*` | |
|
||
| `respect_jellyfin_watch_state` | `true` | §10 |
|
||
| `jellyfin_url` | `http://127.0.0.1:8096` | |
|
||
| `jellyfin_api_key` | *(empty)* | masked in the UI |
|
||
| `jellyfin_user_id` | *(empty)* | |
|
||
| `pot_provider_url` | `http://127.0.0.1:4416` | |
|
||
| `max_attempts` | `5` | |
|
||
| `disk_cap_gb` | `0` | `0` = disabled; if set, evict oldest downloaded videos |
|
||
|
||
Validate on save (integers parse, URLs well-formed) and re-render the form with an inline
|
||
error rather than 500-ing.
|
||
|
||
---
|
||
|
||
## 9. Entry point and scheduling
|
||
|
||
Single executable `ytsubs` with subcommands:
|
||
|
||
| Command | Purpose |
|
||
|---|---|
|
||
| `ytsubs run` | poll → download → reap. This is what cron calls. |
|
||
| `ytsubs poll [--channel ID]` | discovery only |
|
||
| `ytsubs download` | drain the pending queue |
|
||
| `ytsubs reap` | retention pass |
|
||
| `ytsubs serve` | admin HTTP server (systemd unit) |
|
||
| `ytsubs subscribe <url>` / `unsubscribe <id>` | CLI equivalents, useful for debugging |
|
||
| `ytsubs doctor` | check yt-dlp version, POT provider reachable, DB writable, media root writable, Jellyfin reachable |
|
||
|
||
`run` takes a **non-blocking `flock`** on `/var/lib/ytsubs/run.lock` and exits 0 silently
|
||
if already held. On startup, reset any `downloading` rows to `pending` (crash recovery).
|
||
|
||
### Cron
|
||
|
||
```
|
||
17 * * * * runitor -uuid <hc-uuid> -- /usr/local/bin/ytsubs run
|
||
```
|
||
|
||
Hourly is ample for single-digit channels. Register the check in Healthchecks with a
|
||
generous grace period.
|
||
|
||
### systemd unit for the admin server
|
||
|
||
Standard `simple` unit, `Restart=always`, running as the same unprivileged user that owns
|
||
the DB and media root. When the user subscribes to a channel via the web UI, the handler
|
||
should spawn `ytsubs run --channel <id>` detached so the backfill starts immediately
|
||
rather than waiting up to an hour.
|
||
|
||
### nginx
|
||
|
||
Bind the admin server to `127.0.0.1:8085`. Front it with nginx on the **tailnet
|
||
interface only** — `yt.jihakuz.xyz` resolving inside Headscale, HTTP basic auth on top.
|
||
Given the brother will be on the tailnet anyway, there's no reason to expose this
|
||
publicly, and it removes the need for a Let's Encrypt cert and any real CSRF story.
|
||
Confirm this before implementing; if it does need to be public, add proper CSRF tokens.
|
||
|
||
---
|
||
|
||
## 10. Retention and deletion
|
||
|
||
### Aging out (`ytsubs reap`)
|
||
|
||
Candidates: `state = 'downloaded'` and `upload_date < today - retention_days`.
|
||
|
||
Before deleting, if `respect_jellyfin_watch_state` is on:
|
||
|
||
1. `GET {jellyfin_url}/Items?userId={user_id}&recursive=true&includeItemTypes=Episode&fields=Path,UserData`
|
||
with header `X-Emby-Token: {api_key}` — one call per reap, build a `path → UserData` map.
|
||
2. **Skip** any candidate where `UserData.PlaybackPositionTicks > 0 and not UserData.Played`
|
||
(part-watched) or `UserData.IsFavorite` is true.
|
||
3. Apply a hard backstop: delete anyway once `upload_date` is older than
|
||
`retention_days * 2`, so a half-watched video doesn't live forever.
|
||
|
||
Nothing wrecks "set and forget" like a video disappearing at the 20-minute mark. This
|
||
check is worth the complexity.
|
||
|
||
Deletion removes the media file plus its `.nfo`, `-thumb.jpg`, `.srt`, and `.info.json`
|
||
siblings, then prunes the season directory if empty. Set `state='deleted'`,
|
||
`deleted_at`, and `rel_path=NULL`. **Keep the row** — it's the tombstone.
|
||
|
||
If the Jellyfin API is unreachable, skip the whole reap for that run and log it. Do not
|
||
delete blindly.
|
||
|
||
### Unsubscribe
|
||
|
||
Hard delete, as agreed: `shutil.rmtree()` the channel directory, then `DELETE FROM
|
||
channel` (cascades to `video`). Re-subscribing later starts from scratch and re-downloads
|
||
the last 7 days. Put a confirmation step in the UI — this is destructive and irreversible.
|
||
|
||
### Disk cap (optional, `disk_cap_gb`)
|
||
|
||
If enabled, after each reap, sum `size_bytes` over `downloaded` rows and evict oldest-by-
|
||
`upload_date` until under the cap, honouring the same watch-state protection.
|
||
|
||
---
|
||
|
||
## 11. Admin UI
|
||
|
||
Stdlib only — `http.server.ThreadingHTTPServer` + `BaseHTTPRequestHandler`. No Flask, no
|
||
FastAPI, no npm. Server-rendered HTML with a single embedded `<style>` block. The only
|
||
JavaScript should be a `confirm()` on the unsubscribe button.
|
||
|
||
### Routes
|
||
|
||
| Method | Path | Behaviour |
|
||
|---|---|---|
|
||
| `GET` | `/` | Channel list + add form + settings form |
|
||
| `POST` | `/channels` | Resolve URL → insert → spawn backfill → redirect to `/` |
|
||
| `POST` | `/channels/<id>/delete` | Confirm-guarded hard delete → redirect |
|
||
| `POST` | `/settings` | Validate + persist → redirect |
|
||
| `GET` | `/health` | JSON: yt-dlp version, POT provider up, last run time, queue depth |
|
||
|
||
All POSTs redirect (303) so refresh doesn't resubmit.
|
||
|
||
### Channel list should show, per channel
|
||
|
||
Title, `@handle`, video count on disk, disk usage, most recent upload date, last poll
|
||
time, and a warning badge if `consecutive_poll_failures > 2`.
|
||
|
||
### Channel URL resolution
|
||
|
||
Accept: `https://www.youtube.com/@handle`, `/channel/UC...`, `/c/name`, `/user/name`,
|
||
a bare `@handle`, or a bare `UC...` ID.
|
||
|
||
```
|
||
yt-dlp --flat-playlist --playlist-items 0 -J <url>
|
||
```
|
||
|
||
`--playlist-items 0` fetches channel metadata **without enumerating the uploads**, which
|
||
is the cheap way to do this. Pull `channel_id`, `channel`, `description`, `thumbnails`
|
||
from the result. If resolution fails or returns no `channel_id`, re-render the form with
|
||
an error — don't insert a half-formed row.
|
||
|
||
Reject duplicates on `channel_id` (not on the submitted URL — the same channel has many
|
||
URL forms).
|
||
|
||
---
|
||
|
||
## 12. Known gotchas
|
||
|
||
- **yt-dlp churn is the top operational risk.** Extractor args, client names, and PO
|
||
token requirements change frequently. Weekly `pip install -U yt-dlp` under runitor.
|
||
When something breaks, check the yt-dlp issue tracker before debugging our code.
|
||
- **The `UULF` prefix is undocumented** and could vanish. The `channel_id` fallback path
|
||
is not optional.
|
||
- **RSS feeds cap at ~15 entries.** Fine for hourly polling, insufficient for backfill —
|
||
hence the separate `--flat-playlist` path in §4.
|
||
- **`.work` must share a filesystem with the media root**, or every completed download
|
||
becomes a full copy.
|
||
- **Don't transcode.** If Jellyfin is transcoding these files, the format selection in §6
|
||
is wrong — fix it there, not by throwing hardware at it.
|
||
- **Descriptions are hostile input** for XML generation. Use `xml.etree`'s serialiser
|
||
rather than string-formatting the NFO by hand.
|
||
- **WAL mode is required** — two writers.
|
||
- Don't reuse `/disks/Plex/YouTube` (TubeArchivist's tree).
|
||
|
||
---
|
||
|
||
## 13. Acceptance criteria
|
||
|
||
Work through these in order; each is a real check, not a code-reading exercise.
|
||
|
||
1. `ytsubs doctor` passes on a clean install.
|
||
2. Subscribing to a real, active channel creates the directory, `tvshow.nfo`,
|
||
`poster.jpg`, and queues the last 7 days of videos.
|
||
3. At least one real video downloads end to end and lands in the right season directory
|
||
with a correct filename, NFO, thumbnail, and subtitle sidecar.
|
||
4. `ffprobe` on the downloaded file shows **h264 video and AAC audio**.
|
||
5. Jellyfin, after a library scan, shows the channel as a show, the year as a season, and
|
||
the video as an episode with the correct title, description, and air date — with all
|
||
internet metadata providers disabled.
|
||
6. The video **direct-plays** on a client with no transcoding (check the Jellyfin
|
||
dashboard's active-streams panel).
|
||
7. Re-running `ytsubs run` immediately downloads nothing and errors on nothing
|
||
(idempotency).
|
||
8. Manually backdating a video's `upload_date` past the retention window causes `reap` to
|
||
delete it and its sidecars, prune the empty directory, and leave a `deleted` tombstone.
|
||
9. Re-running `poll` after that does **not** re-download the deleted video.
|
||
10. Marking a video part-watched in Jellyfin protects it from reap (with
|
||
`respect_jellyfin_watch_state` on).
|
||
11. Unsubscribing removes the directory and all rows.
|
||
12. Killing `ytsubs run` mid-download leaves no orphan in `.work`, and the next run
|
||
recovers the `downloading` row to `pending`.
|
||
13. The admin page renders, adds, removes, and persists settings; a bad settings value
|
||
produces an inline error rather than a traceback.
|
||
14. Editing `retention_days` in the UI visibly changes reap behaviour on the next run.
|
||
|
||
---
|
||
|
||
## 14. Suggested build order
|
||
|
||
!! tom: we also wants tests through this
|
||
1. Skeleton: DB schema, migrations, settings accessors, `doctor`
|
||
2. POT provider + a hardcoded single-video download proving §3 and §6 work — **do this
|
||
before writing anything else of substance**
|
||
3. Channel resolution + `subscribe`/`unsubscribe` on the CLI
|
||
4. Discovery (`poll`), both feed paths
|
||
5. Download worker + NFO/artwork generation + move-into-place
|
||
6. Jellyfin library verification (acceptance criteria 5 and 6) — a checkpoint, not a step
|
||
7. `reap`, including the watch-state check
|
||
8. `run` orchestration, flock, crash recovery, cron + runitor + Healthchecks
|
||
9. Admin server
|
||
10. systemd unit, nginx on the tailnet, deploy scripts
|
||
|
||
Steps 2 and 6 are the two places where this design could turn out to be wrong. Hit them
|
||
early and report back rather than building on top of an unverified assumption.
|