The Missing Layer in AI Video Generation: Why Programming, AI, and Real Footage Need to Be Separated
The Current State
Two dominant approaches exist for programmatic video generation today:
- HTML/CSS to video (Hyperframes) — render web layouts, output frames as video.
- Canvas with HTML injection (Remotion) — render into canvas, composite HTML on top.
Both work because they give agents a vector-programming layer to write code against, and agents are good at code. That's the whole appeal.
The limitation: neither approach can touch real footage. No background removal, no masking, no object tracking, no replacing a masked silhouette with new footage while preserving the surrounding noise (grain, lighting, compression artifacts). These are exactly the things AI video editing is good at, and neither Hyperframes nor Remotion gives you access to them.
The Core Thesis
AI is not yet trusted for quality, and that's fair. AI is exceptional at imagination, not at reliability. Deterministic code is exceptional at reliability, not at imagination. The fix isn't picking one, it's routing each part of a video to whichever system is actually good at it.
| Video Element | Handled By |
|---|---|
| Zoom in / out | Programming layer |
| Lighting | Programming layer |
| Camera motion + transitions requiring new information | AI footage |
| Person's face and eyes | Real footage |
| Clothes / low-attention background | AI footage |
| Mask capture | SAM 3.1 (segmentation) |
| Gap-filling after mask merge | AI footage |
| Extending a real clip | AI extension of real clip |
| New angle of same person talking, same expression | Reference footage re-rendered as a new clip |
The unresolved case is motion typography — text-in/out effects. SAM 3.1 with templates handles simple layering fine, but advanced text motion graphics still need a generative model like Minimax H3 Max. This is a genuine seam in the architecture, not a solved problem yet.
One idea worth pursuing: store the periphery (background, camera motion, lighting) as structured metadata in a markdown file, separate from the person's expression data. This keeps the "director's intent" portable and reusable across renders. Nobody has built this properly yet. It's a real gap, not just an unexplored corner.
Layering: Three Methods, Tested
This is the fundamental problem to solve. Three approaches were tried.
Method 1: Direct Face Swap
Reference video + reference image → face-swap the person, sync audio with sync.so.
Result: Failed. Face consistency broke down across frames. This is expected, not surprising — face-swap models optimize per-frame identity transfer, not temporal coherence. Without a mechanism enforcing frame-to-frame consistency (like a shared latent or tracked mesh), drift is inevitable at anything beyond a few frames. This method is a dead end as-is, not something to keep tuning.
Method 2: Storyboard-to-Video
Extract camera motion and background from a reference video. Generate 3–4 time-mapped stills of yourself superimposed on that motion at the right frames. Feed the stills into a video model as a storyboard with a timestamp-level prompt. Test across models to reconstruct the 7-second clip with matching audio.
Result: Untested to completion. This is the most promising of the three and it's the one that got the least follow-through. The approach is sound: it turns "generate a video of me" into "interpolate between these known-correct anchor frames," which plays to what video models are actually good at (in-betweening) instead of what they're bad at (long-horizon identity consistency).
Method 3: Overlay Recording
Record yourself speaking the same content in equal time, overlay it with ffmpeg on top of the reference video, then use editing software to preserve the real expression while AI fills in the lag/gap information. Audio from the overlay is unused, handled separately.
Result: Not fully evaluated. Structurally this is closest to the proposed thesis (real footage for face, AI for periphery) but it's currently manual, not a harness. It needs a defined mask boundary and a defined blend strategy to be reproducible.
The Honest Assessment
Method 1 should be dropped, not iterated on. It fails for a structural reason, not a tuning reason.
Method 2 is the strongest candidate and it's also the least finished. That's the wrong order of priorities. Right now this reads as a promising direction with only one method (the weakest one) taken to a real result.
The layering table is good but it's a design spec, not a system. Nothing in these notes describes how the programming layer and the AI layer hand off to each other, what the interface between them looks like, or how the mask boundary is defined and maintained frame to frame. That handoff is the actual hard engineering problem. Everything else here is composition of known building blocks (SAM, video gen models, ffmpeg).
What to Build Next
- Finish Method 2 first. Take the storyboard approach through a full 7-second clip, across at least 2-3 video models (Seedance, Kling, Minimax), and score identity consistency and audio-sync quality objectively, not by eye.
- Define the markdown periphery-metadata schema. Camera motion, lighting, background, timing, all as structured, versioned data separate from the expression layer. This is what makes the harness reusable instead of one-off.
- Build the mask handoff spec. Define exactly how a SAM 3.1 mask boundary is tracked frame-to-frame and how the AI-filled region is blended with real footage without a visible seam (noise-matching, not just alpha compositing).
- Treat motion typography as a separate subsystem. Don't try to force it into the same mask/programming split as everything else. It has its own pipeline (template + generative text model) and should be spec'd that way from the start.
- Kill Method 1 in the writeup. Keep it only as a documented negative result, not as an open thread.