Desmos Guy

What it does

Feed it a Family Guy clip. You get back a vertical video with a Desmos graph on top drawing the scene as line art, expression list and all, and the original clip playing underneath.

It all runs off one command. autopilot.py --count 10 goes and finds ten clips, works out which episode each one is from, throws out the ones it can't caption, renders them, uploads them to YouTube, and drops the files and captions in a folder for me.

About 4,100 lines of Python across 10 modules. No framework, mostly because I didn't realise I'd want one until it was too annoying to add. It leans on yt-dlp, faster-whisper, playwright, opencv, numpy and ffmpeg to do the heavy parts.

The next four sections are the render path, in order. Each has an interactive demo, because most of this was easier to understand once I could see it move.

Tracing a frame

Before Desmos can draw anything, a frame has to stop being pixels and start being lines.

Each frame gets decoded to grayscale, run through a Sobel filter to find edges, thresholded, and then I walk the edge pixels one at a time to get outlines that are actually in order. Basically a list of points you could draw with a pen without lifting it. Even a simple frame comes back with thousands of points, which is more than Desmos will happily take. So the last step throws most of them away.

That's running the real algorithms in your browser, on a cat I drew in code so there's nothing copyrighted sitting in this repo. Same Sobel operator, same border-following, same Ramer–Douglas–Peucker simplification the renderer uses.

Push the tolerance slider around and you'll see the tradeoff I spent a while stuck on. Too tight and the expression lists get enormous and Desmos crawls. Too loose and the cat stops looking like a cat. I ended up around 1.5 px, which drops roughly 90% of the points and still keeps the shape.

Packing it into Desmos

Now I've got a pile of polylines per frame, and I need Desmos to show exactly one frame at a time.

The whole thing is list-valued parametric expressions with a single slider n choosing the frame. Except Desmos caps how long a list can be, so I can't dump every frame into one list. Frames get packed into chunks of about ten.

Which creates a new problem. When the slider is sitting on frame 14, chunks 1 and 3 still get evaluated. If they index past the end of their list the whole graph throws an error and nothing renders.

What fixed it was gating. Each chunk gets an expression that equals 1 inside its own frame range and is undefined everywhere else. Undefined doesn't error in Desmos. It just makes that curve vanish. And the index gets clamped so it's always a legal position in the list, even when that chunk isn't the one being drawn. No boundary special-casing anywhere.

Exactly one chunk is defined at a time. The other two evaluate to nothing and disappear, and the clamped index means they're still doing perfectly legal arithmetic while they do it.

The screenshot problem

Once the graph exists, I screenshot it frame by frame in a headless Chromium and stack it over the source clip with ffmpeg. This is where all the time goes, and where the worst bug was hiding.

I assumed the slow part was tracing. My tracer runs 8 worker processes on a 15-core machine, so the obvious move was more workers. I profiled it to work out how many to add.

Screenshotting was 94% of it, one frame at a time, 486 ms each, in a single browser page. So I parallelised the capture instead of the tracing and the same clip went from 171.6 s to 110.4 s.

Then the frames came out different.

The frames had been wrong the whole time

My first thought was that the parallel version was broken. Serial had been running for months. Obviously serial was the correct one. So I tested serial against itself. Rendered the same clip twice on an idle machine and diffed every frame.

Zero of 40 frames differed. Perfectly deterministic. I took that as proof it was correct, which is why the bug lasted as long as it did. A deterministic process repeats the same mistake every run, so consistency said nothing about whether the output was right.

The bug was in the settle logic. After moving the slider I wait for the canvas to stop changing before I screenshot it, and "stopped changing" meant 2 unchanged animation frames, about 33 ms. Desmos pauses for longer than that in the middle of an update. So I was catching the background redrawn and the character not.

To find the real answer I raised the threshold until the output stopped changing. stable=12 and stable=24 come out byte-identical, so 12 is converged and anything at or above 4 is fine.

Every video I'd made before this had incomplete linework on basically every frame. Nobody noticed, including me, because it was wrong the same way every time. The fix corrected the linework and made the render faster, since the parallel capture that exposed the bug was also the speed-up.

What a render costs

Render time scales with clip length at roughly 11–12×, but the spread is wide: 7.3× to 15.5× in the first batch, depending on how much the tracer can reuse between frames. My first five measurements all landed between 10× and 13×, which made me think the constant was way tighter than it is.

Version by version

None of this was designed up front. Each version came out of a measurement that disagreed with what I expected.

Both of the largest improvements came from measuring something I assumed I already understood. I was sure tracing was the bottleneck and I was sure serial capture was correct. Neither held up once there was a number attached to it.

Finding clips

Rendering is only half of it. Something has to go find things worth rendering, and that half had its own set of problems.

Shorts were invisible

yt-dlp --flat-playlist reports duration: None for every single YouTube Short. My length filter read:

if not vid or not dur: continue

which threw away 100% of them. Shorts are under 60 seconds by definition. I was searching for 8 to 45 second clips. So the best source I had was being dropped entirely, and the log printed 40 results, 0 in band, which looks exactly like a source that just doesn't have anything good in it.

Fixed by letting unknown-duration entries through to the full metadata fetch. Candidate pool went from 81 to 120 per run, and the first run after the fix pulled back 46 Shorts.

Two channels that were never there

I'd seeded the harvest with a couple of channel handles I typed from memory instead of checking. One had no /videos tab. The other 404'd. A dead channel prints 0 results, same as a channel I've already used up, so the curated half of my harvest did nothing for its entire existence and never said so. Replaced them with URLs I actually opened; the best one gives 16 usable clips out of 40. Dead sources now warn loudly instead of shrugging.

Nothing joined to anything

Performance data was keyed by Instagram's re-encoded copy (ig_<media_id>) and source data by render tag (auto_s03e09). There was no shared key. So I couldn't answer the question I'd built the whole thing for: did the original's view count predict anything? However many posts piled up, there was no way to check.

Duration looks like the obvious join key. It isn't: 26 matches out of 68, and tightening the tolerance makes it worse (12 matches at ±0.05 s), because Instagram's re-encode shifts the duration by more than a frame.

What worked was dialogue. Transcribe both sides, match on n-gram overlap, and require a margin over the runner-up so near-ties get recorded as ambiguous instead of guessed. 56 of 68 linked, median overlap 0.95, median margin 0.94, zero ambiguous.

Uploads went to the wrong channel

A Google account can own several YouTube channels, and the OAuth consent screen has a channel picker that is very easy to click straight past. My first uploads landed on my personal channel. The youtube.upload scope is write-only, so the tool couldn't even read back where it had put them.

Fixed by adding youtube.readonly, asserting the channel handle before every upload, and verifying each video is live afterwards with a retry backoff, because videos.list lags videos.insert. That once gave me five false "not visible" reports on videos that were completely fine.

YouTube also has two separate daily limits: an API project quota and a per-channel upload cap around 26 a day. The second one comes back as HTTP 400 and doesn't contain the word "quota" anywhere. My handler checked for "quota". It missed, and burned 30 doomed uploads in a row.

How long a clip can safely be

2 of 140 uploads got Content-ID blocked. Both were over 89 seconds. Nothing under 80 was ever touched. So I capped clips at 45 seconds. That costs nothing in practice, because the entire top ten by plays is under 32 seconds anyway, and it removes most of the exposure.

The picker that didn't work

The original plan was a model that picks which clips to render, ranked by predicted views. I built it, and it does not predict well enough to be worth using. The measurements below are why.

A retention model is real and holds up out of sample. Cross-validated R² of 0.49. The problem is it's basically duration wearing a hat, and duration doesn't separate winners from losers. The top ten posts by plays run 8 to 32 seconds. The bottom ten run 6 to 40.

Predicting plays directly is worse than useless: cross-validated R² of −0.89, meaning I'd have done better guessing the average every single time. And the part of retention that does predict plays only becomes measurable once the post is already live, which is too late to help me decide what to render.

What it did on Instagram

8 days, 85 posts on Instagram and 140 uploads on YouTube. 12,837 hours of watch time. That's 534.9 days, or about 1 year 5 months of continuous viewing, off 85 clips.

Watch time and plays are different numbers and easy to confuse. 12,837 is hours watched. Plays are 4.3 million.

People are watching them twice

Average watch time is 19.6 seconds, and plenty of these clips are only 9 to 16 seconds long. Viewers are reaching the end and starting again without leaving, which is the behaviour Instagram's ranking rewards.

Shares came out higher than I expected. 97,274 shares is 2.25% of all plays, which is a high rate for the format.

Keeping these numbers current

Every figure in this section is pulled from the Instagram Graph API by tools/refresh_data.py, which writes data/ and regenerates the page's data file. A GitHub Action runs it weekly, commits only if something actually moved, and the site rebuilds itself. The access token lives in repo secrets and gets rotated by the same workflow, since Instagram's long-lived tokens expire after 60 days.

None of these numbers are typed into the sentences. They're <span data-dg="total_plays"> placeholders filled in from the data file at load, so refreshing the data rewrites the prose too. Without that the stat tiles would update while the paragraph beside them kept last month's figure.

Still broken

• About a week of data, one account, one show. None of this generalises anywhere, and the correlations above come from a small sample of a single genre.
• The source-popularity signal is untested. It's a hypothesis. The test is written and currently reports n = 0. I have not shown it works.
• Instagram posting is still manual. YouTube is fully automatic. The platform that produced all 12,837 hours is the one I still upload by hand.
• yt-dlp is a single point of failure. Whenever YouTube changes something, harvest breaks. It fails loudly and skips the run rather than posting bad output.
• Content-ID risk scales with automation. Two of 140 got blocked; a 45-second cap is a mitigation, not a guarantee.