
Researchers from Apple and Cornell University have introduced FastFrames, a family of autoregressive diffusion models for real-time video generation. Each new frame takes less than 50 ms. Human-computer interaction studies show that people already notice delays of about 50-60 ms. That is why games and interactive world simulators need this speed. FastFrames-S is built on the fastest previous model, Causal Forcing++, and shares its transformer. Yet FastFrames-S needs only 36 ms per frame instead of 272 ms on a single B200 GPU. On VBench, it matches the original. FastFrames-L, with 14 billion parameters, fits into 45 ms on eight B200 GPUs. According to the authors, no other current model generates a frame faster than 50 ms.
Apple has released the inference and training code under the Apple Sample Code License. However, the weights are available only for non-commercial research, and Apple has not published its internal dataset of 10,000 videos. The code is on GitHub, the FastFrames-S and FastFrames-L weights are on Hugging Face, and video examples are on the project page.
Why FPS is not enough for real-time video generation
An autoregressive model generates video from text frame by frame and relies on the frames it has already made. Therefore, it can take a new command between steps, for example a camera turn. Papers often back claims of real-time video generation with frames per second (FPS). However, this number says nothing about reaction speed. A model that generates a batch of 30 frames every second runs at 30 FPS, yet the user still waits a full second for a response to a key press. For example, Helios claims 19.5 FPS on a single H100. Still, in the FastFrames authors’ measurements on a B200, its response takes 2.5 seconds.
That is why the authors measure end-to-end latency: the time from a new command to the first frame that reacts to it. From it, they compute responses per second: RPS = 1000 / latency in ms. The 50 ms threshold equals 20 RPS. FastFrames-S reaches 27.8 RPS, while Causal Forcing++ reaches only 3.68.

Where the time goes
First, the authors broke down Causal Forcing++, which builds on Wan 2.1 with 1.3 billion parameters, step by step. In each step, the model generates one latent frame, and the decoder expands it into four regular frames. On a single B200, one step takes 271.8 ms:
- KV cache update, 41.6 ms. The diffusion transformer (DiT) remembers past frames through the KV cache, that is, the stored keys and values of its attention layers. To write a clean version of the previous frame there, the model runs an extra DiT pass;
- Diffusion, 83.2 ms. Two denoising steps take 41.6 ms each. With one step, quality drops noticeably, while latency falls by only 15%;
- VAE decoder, 147 ms. It converts the latent into pixels.
So denoising itself takes only about 30% of the time, and the VAE decoder alone exceeds the 50 ms budget three times over. Therefore, to reach real-time video generation, FastFrames speeds up the decoder and the cache first.

A pixel decoder instead of the VAE
The Wan VAE decoder has only 73.3 million parameters. However, it upsamples many times and runs dense 3D convolutions. So the authors checked whether they needed it at all. As it turned out, a single linear layer trained to reproduce the VAE output already gave blurry but recognizable video.
The final decoder consists of three causal 3D blocks in the ConvNeXt style and works directly on the DiT token grid. Then a final linear layer turns each token into a patch of four frames of 16×16 pixels. With 63.9 million parameters, the decoder takes 0.9 ms instead of 147 ms. First, the authors train it to reproduce the VAE output while the DiT stays frozen. Then they fine-tune it against a DINOv3-based discriminator on 10,000 real videos. This adversarial training in pixel space became practical because the new decoder needs about 5 GB of GPU memory, compared with 68 GB for the VAE. The downside is that one VAE serves all Wan models, while each model needs its own pixel decoder.

Updating the KV cache in 2 ms
At the last denoising step, the DiT still sees a noisy frame. As a result, the keys and values it writes to the cache come from noise. To fix this, previous models ran the DiT once more on the clean frame just for the cache and threw away the output of that pass. However, you cannot simply drop this pass. Without it, objects in the scene change within a couple of seconds, and training the model on a noisy cache produces artifacts.
Instead, FastFrames replaces the pass with a small network. Its four transformer blocks read the clean frame, and a separate head for each DiT layer predicts corrections to the keys and values. The cache update takes 2 ms instead of 41.6 ms for the 1.3-billion-parameter model and 6 ms for the 14-billion-parameter one, with no loss in quality.

GPU optimizations
Now the DiT takes most of the time. The authors sped it up with engineering techniques such as CUDA graphs, fusing the Q, K and V projections and computing attention over separate cache chunks in parallel. As a result, a DiT pass dropped from 41.6 to 16.4 ms. On eight B200 GPUs, the cards also split the tokens between them (context parallelism). Thus a pass takes 5.8 ms for the smaller model and 20.3 ms for the larger one.
Real-time video generation without quality loss
The authors ran all models with the same prompts and seeds at about 480p. For autoregressive models, they measured latency with a full KV cache. FastFrames-S scores 84.23 on VBench versus 84.11 for Causal Forcing++ with the same DiT weights. The authors attribute the gain to the adversarial training of the decoder. Meanwhile, FastFrames-L scores 84.44 versus 84.42 for Krea-Realtime and runs 10.6 times faster on a single GPU: 158 ms versus 1,675 ms. On the semantic part of VBench, however, both FastFrames models trail their baselines slightly: 79.92 versus 80.20 and 80.29 versus 81.81.


Real-time video generation with camera and physics control
The FastFrames techniques for real-time video generation transfer to existing models without a large dataset, because the authors get the target frames and the clean cache by running the model itself. In this way, they sped up the LingBot world models with camera control at 1.3 and 14 billion parameters. They also sped up RealWonder, a 1.3-billion-parameter model that accounts for scene physics. LingBot 1.3B latency fell from 842 ms in the best official configuration to 52 ms on eight B200 GPUs, a 16-fold drop. Similarly, RealWonder went from 1141 to 51 ms (22 times less), and LingBot 14B went from 1373 to 233 ms. Still, their design keeps them from catching up with FastFrames: LingBot runs four diffusion steps and generates four latent frames at once, while RealWonder runs three of each. Finally, the repository includes a LingBot 1.3B demo that you control with the WASD keys.

Limitations
The main measurements use 480p on expensive B200 server GPUs. At 720p, FastFrames-S produces a frame in 127 ms on one B200, and FastFrames-L needs 116 ms on eight. So real-time video generation at 720p is still out of reach. On an H100, the same operations run 25-50% slower.

Controllable models are still slower than plain text-to-video generation. Therefore, the authors advise training base models with low latency in mind, for example by feeding the camera pose through channels rather than as extra tokens in the sequence. Finally, long-term scene consistency and physical plausibility still need work.









