Adobe Research · University of Rochester · Technical Report
A Unified Autoregressive Framework for Interactive Video Generation and Editing
Generation, transformation and editing on a single backbone
Denoising steps at inference, guidance-free
720p streaming throughput on a single GPU
Faster than the multi-step teacher it is distilled from
Interactive video generation and editing are becoming increasingly important for creative design. EditStream unifies multiple video creation and manipulation tasks within a single DiT-based model through flexible task-specific conditioning, and further transforms it into a fast, few-step autoregressive model for efficient streaming. It supports Text-to-Video, Image-to-Video, Video-to-Video, Editing Propagation, Reference-guided Video Editing, and Camera Pose Change — enabling flexible control over video generation, transformation, and editing within one system.
To make the unified model practical for interactive use, we develop a two-stage distillation approach that combines Velocity Moment Matching (VMM) with autoregressive unrolling. VMM matches conditional velocity moments at student-reached intermediate states to preserve generation quality and motion, while unrolling exposes the student to its own autoregressive predictions to improve temporal stability. Together, they alleviate common challenges in few-step autoregressive video generation — over-saturation, degraded motion, temporal instability, and complex training — and bridge high-quality diffusion-based video models with interactive creative workflows.
Two ideas carry the work: a conditioning interface that admits every task through the same two ports, and a distillation recipe that reaches a causal few-step student without ever causalizing the teacher.
Controls that share the output's pixel grid ride the channel axis — stacking them costs no attention length. Controls that do not are appended as tokens, where self-attention performs in-context transfer.
| Task | Conditioning | Port |
|---|---|---|
| Text-to-Video | prompt only | — |
| Image-to-Video | first frame locked at timestep 0 | channel |
| Video-to-Video | source video | channel |
| Editing propagation | source video + edited first frame | channel + token |
| Reference-guided editing | source video + reference image | channel + token |
| Camera pose change | warped view + validity mask | channel |
The teacher is first adapted toward the student's own generation pattern; the student is then distilled from it in two stages that optimise the same VMM objective, differing only in where the corrected state comes from.
Teacher Adaptation
Full multi-step bidirectional DiT trained across all six tasks through the conditioning interface above.
25–50 steps · full attention
A mixture of full-sequence, first-block cold-start and prefix–suffix objectives: one 3-frame block is noised while its neighbours stay clean at timestep 0.
still bidirectional
Student Distillation
Single-block Velocity Moment Matching against the teacher-forced teacher reaches a usable causal student — no precomputed ODE-trajectory dataset.
200 iters · vs ≈5,000 for ODE init
The student generates every block end-to-end through its own KV cache, exactly as at inference; the correction is queried along that self-generated trajectory.
800 iters · 4-step causal student
An auxiliary model is trained online to track what the student currently produces. Subtracting it from the teacher leaves a residual that points from the student's distribution toward the teacher's — and it is read off at the student's own intermediate state, not at a state drawn from data.
All clips are recompressed for the web from native resolution. Comparisons below are pulled directly from the release's own evaluation sweeps.
Teacher (bidirectional, 25–50 steps) vs. a self-forcing baseline distilled from an ODE-pair init vs. our 4-step VMM-Forcing student — same prompt, same seed family, seven VBench-derived samples.
A 30-second interview clip, edited end-to-end in a single causal streaming pass — one instruction rewrites the entire scene while identity, pose and delivery stay untouched. Drag the divider to compare source and edit frame-for-frame; use the arrows to step through styles.
Teacher vs. KiwiEdit (showlab/Kiwi-Edit, stage3_refvid_only) vs. our VMM student, on RefVIE / RefEdit Bench — a reference image supplies the new subject or background through the token port, the source video supplies everything else through the channel port.
Replace the background with a Chinese ink painting, featuring a large golden mountain peak rising above swirling clouds, ensuring it appears in the same position and pose within the video scene.
Replace the background with a serene anime-style landscape, featuring a massive, fluffy cumulonimbus cloud towering over a windswept grassy hill, ensuring it appears in the same position and pose within the video scene.
Replace the background with a Van Gogh style oil painting, featuring a swirling blue starry night sky and a dark cypress tree, ensuring it appears in the same position and pose within the video scene.
Replace the sofa with a mid-century modern lounge chair with an ottoman. All other parts of the video must remain unchanged.
Replace the man with a low-poly 3D model of a medieval knight. The character is composed entirely of sharp, faceted geometric triangles. Flat lighting, vibrant colors (blue tunic, silver armor). All other parts of the video must remain unchanged.
Replace the girl's cloth with a matte black techwear tactical jacket. All other parts of the video must remain unchanged.
Add a small handmade amigurumi crochet cactus sitting in a beige yarn pot next to the window. All other parts of the video must remain unchanged.
Replace the man's outfit suit with a festive 'ugly' Christmas sweater. All other parts of the video must remain unchanged.
ReShoot: any video, reconstructed in 3D from the single input on the left. Both results on the right are EditStream — the same reconstruction replayed along two different camera trajectories: a wide S-curve orbit that sways and pushes in, and a straight orbit around the subject.
GenPropBench, prompt-guided: an edit is made to the first frame, then propagated through the entire clip in one streaming pass.
Change aspect ratio from vertical to widescreen, revealing more landscape on both sides.
Remove the person’s legs and feet, preserving original background and motion.
Change video aspect ratio to widescreen, revealing more background on both sides.
Change aspect ratio from vertical to horizontal, revealing more tunnel sides.
Add fire and flames to the car and overlay falling snow, preserving original background and motion.
Remove the conical hat from the dog's head, preserving original background and motion.
Remove the boy in red shirt, preserving original background and motion.
Recolor the left person blue and the right person red, preserving original background and motion.
@misc{zhou2026editstream,
title = {EditStream: A Unified Autoregressive Framework
for Interactive Video Generation and Editing},
author = {Zhou, Yuqian and Zhou, Zhenghong and Wu, Zongze and
Smith, Cameron and Zhang, Richard and Luo, Jiebo and
Shechtman, Eli and Lin, Zhe},
year = {2026},
eprint = {2608.21424},
archivePrefix = {arXiv},
primaryClass = {cs.CV}
}