The Joy Future Academy team published JoyAI-Video-Edit, an open autoregressive diffusion model with 16 billion parameters that performs instruction-guided video editing on a live stream, with no access to future frames and no predefined clip duration. In the authors’ own measurements, the full end-to-end pipeline delivers 30.19 FPS at 720×1280. The model combines an MLLM-based condition encoder, a causal video VAE, and a multimodal diffusion transformer.
Deployment code and scripts are available on GitHub, and the DiT and causal VAE weights are hosted on Hugging Face under the Apache 2.0 license. Training corpora and the team’s own LongV2VBench benchmark have not been released, though the data and weights of the related image-editing model JoyAI-Image-Edit are public along with its code, and the editing supervision was transferred from exactly that source. Worth noting: only inference code is open. There are no training scripts in the repository, so reproducing the training run from scratch is not possible with the published materials.
Why current approaches fall short
High-quality diffusion video editors run offline: they take the whole clip, apply bidirectional attention across time, and return a result only after every denoising step is done. That works for short clips but rules out live broadcasting or video calls. On top of that, the number of temporal tokens and the activation memory grow with input length, while cutting a stream into pieces and processing them independently produces visible artifacts at the seams.
Making the model causal is not enough on its own. A causal model looks only backwards when generating each frame, at frames already produced, and never peeks ahead; technically this just requires a different attention mask. But another problem appears. During training the model receives real previous frames as history, clean and correct, while in operation it has no real frames at all, only what it generated one step earlier, together with every small flaw: a slightly shifted hue, a texture that drifted a little. The model treats that frame as ground truth, builds the next one on top of it, and that one comes out slightly worse, so within a minute of streaming the picture noticeably departs from the original. This gap between training conditions and actual operation is called train–inference mismatch, and the accumulated distortion is known as temporal drift. For editing the problem is twofold: every new chunk must both continue the previous ones and match the current source frame, preserving everything the instruction did not ask to change.
JoyAI-Video-Edit architecture
JoyAI-Video-Edit consists of three blocks:
- a multimodal large language model (MLLM) acting as the condition encoder;
- a causal video VAE;
- a multimodal diffusion transformer (MM-DiT).
The MLLM receives the first frame and the instruction and outputs condition tokens that encode both the content of the scene and the intent of the edit. The VAE compresses video at a ratio of 8 × 24 × 24, meaning one latent frame corresponds to eight video frames. Two modes are supported, V2V and IV2V, the latter transferring appearance from a reference image.
Architecture overview: MLLM, causal VAE, MM-DiT blocks with condition and vision streams. On the right is the attention mask showing which token groups the active chunk can see.
Chunks and the fight against drift
The video is split into aligned chunks of one latent frame each, that is, eight frames at a time. Attention is bidirectional inside a chunk and causal across chunks. Cross-chunk attention is restricted to a sliding window, otherwise computation would grow with stream length: the active chunk attends to a fixed number of recent chunks plus the very first one, which stays in context permanently and acts as an anchor frame (global sink) for the entire sequence.
Training starts on real previous frames (teacher forcing), and those real frames are later replaced with what the model generates itself during training (the resampling forcing scheme): each history chunk is reassembled with a single denoising step, and gradients do not flow through it. Training therefore runs on a history distribution close to what the model encounters at inference.
SA-DMD and the long horizon
Ordinary diffusion takes dozens of denoising steps; for real-time operation these are compressed down to two using Distribution Matching Distillation. Three models participate in training on a shared backbone with LoRA adapters: the causal student generator, a trainable model that estimates the student’s own distribution, and a frozen teacher that knows what the correct distribution looks like. The student is pulled toward the teacher through the divergence between those two estimates.
The researchers developed and applied Source-Anchored DMD, which anchors the teacher to the part of the source video that is temporally aligned with the current fragment. To do this, they applied classifier-free guidance to the teacher’s prediction along two independent axes: the influence of the text instruction is amplified with weight w_txt, and fidelity to the original with weight w_src. The second branch works like this: the source segment is removed from the teacher’s input and the shift in its prediction is measured.
Long-Horizon Autoregressive Distillation splits a long rollout into short clips, computes the backward pass for each, and clears the graph before the next one, accumulating gradients for a single optimizer step. The model thus sees late states carrying accumulated error without running out of GPU memory.
Where the 30 FPS comes from
At inference the stream is cut into eight-frame chunks, with FP8 quantization, operator fusion, and computation-graph compilation applied throughout. VAE encoding takes 22 ms, DiT denoising 185 ms, and decoding 19 ms, which yields a request-to-response latency of 226 ms. Clean KV-cache construction and pseudo encoding add another 31 and 9 ms, bringing the full cycle to 266 ms, exactly 30.1 FPS.
Stage-wise breakdown of processing a single chunk on one Nvidia B200.
Full pipeline latency on 81 frames came to 2.68 s, against 4.48 s for StreamDiffusionV2 and 5.58 s for SANA-Streaming. The VAE alone reaches 200 FPS, which removes the autoencoding bottleneck. Full results in the table:
Benchmark results
On the open OpenVE-Bench the model scores 3.60 out of 5 as judged by a Gemini multimodal evaluator. That is 0.98 above SANA-Streaming, 1.60 above LiveEdit, 1.73 above XMax-X2.0, and 2.37 above StreamDiffusionV2. The score is comparable to offline editors: Kling-3.0 Omni gets 3.64 and Bernini-R 3.72. On local object removal, 4.06 points is the best result among all compared methods, offline ones included.
For minute-long clips the authors assembled LongV2VBench, 229 tasks across five categories: background change, global style transfer, plus local addition, local modification, and local removal of objects. Here the model ranks first in all five categories with an overall score of 3.30 against 1.71 for its closest rival XMax-X2.0, at a throughput of 30.19 FPS at 720×1280 versus 20.90 FPS for XMax-X2.0 at a lower resolution.
In pairwise human evaluation on clips under 10 seconds, JoyAI-Video-Edit wins 90% of comparisons against LiveEdit, 87% against SANA-Streaming and StreamDiffusionV2, and 81% against XMax-X2.0. Against offline models it is roughly even: versus Bernini-R the split is 48% to 44% in favour of JoyAI-Video-Edit, while Kling-3.0 Omni and Seedance 2.0 come out ahead with 56% of the votes each:
What each component contributes
The ablation shows the contribution of both ideas. The baseline configuration scores 2.81, SA-DMD raises it to 3.23 and delivers the single largest gain: global style goes from 3.61 to 4.24, local change from 3.43 to 4.00. LHAD alone yields 3.06 and helps most where error accumulates longest, namely background change and local removal. Together the two strategies reach 3.30 and win all three local editing tasks, so they complement each other rather than overlap.
Takeaway
Video editing used to force a choice between two kinds of models. Streaming ones processed frames on the fly and suited live use, but their editing quality was noticeably worse. Offline ones produced good results, yet demanded the entire clip and returned it only after a long render. JoyAI-Video-Edit shows that this trade-off is no longer mandatory: the model processes frames as they arrive and takes just two denoising steps instead of dozens, while holding its own against offline editors on quality. In practice that means the result is visible during playback, the instruction can be rewritten on the spot, and there is no waiting for the whole file to render.
The weak spots are visible too. The claimed 30 frames per second were measured on a single Nvidia B200, hardly a consumer card. Resolution is fixed at 720p. And background change remains the model’s weakest category: 2.49 out of 5 on minute-long clips, well below the rest.
