Researchers from Tsinghua University, NVIDIA, CMU, UC Berkeley, Princeton, NUS and Reactor released StreamEval, a benchmark that measures how well streaming video generation handles real-time interaction: can a model execute user requests in the middle of a stream without breaking what is already in the frame? The benchmark contains 300 scripted sessions lasting from 15 to 300 seconds and covers five task types, such as talking to a digital avatar, asking it to raise a hand, moving the camera in a generated scene or recoloring an object in a live video. The authors ran 37 models through StreamEval, and the results are far from perfect.
The best instruction-following score for avatars is 36.90 out of 100, for text-conditioned world generation it is 25.93, and the best model completes a fully correct reference-based edit in only 15% of cases. At the same time, these models keep the picture in good shape. Vidu S2 Avatar, the leader in avatar performance, scores 97.25 out of 100 for video stability and 86.12 for similarity to the source portrait, so it reliably preserves the character’s appearance and image quality. Yet it executes the requested action correctly in only about a third of attempts. The project is open: the evaluation code is on GitHub, test scenarios with timelines and media files are on Hugging Face, and model results are collected on the leaderboard.
How Streaming Video Generation Differs from Regular Video Generation
A standard text-to-video model receives a prompt once and returns a finished clip. Streaming models generate video frame by frame (autoregressively), so the user can step in mid-session: ask an avatar to raise its hand, turn the camera in a generated world or recolor a vase in a video. Each such request carries three conditions: when the change should happen, what exactly should change and what must stay untouched.
Existing benchmarks cover separate parts of this problem. VBench and EvalCrafter assess visual quality, WBench looks at multi-turn control of generated worlds, and LongV2VBench at editing long videos. StreamEval brings avatars, world generation and editing together in a unified protocol. Its 300-second editing sessions are five times longer than those in LongV2VBench, which tops out at 60 seconds.
How the StreamEval Benchmark Works
Each of the five tasks has 60 sessions:
- Digital-human dialogue: the avatar responds to user speech in context, remembers the conversation history and handles interruptions correctly.
- Avatar performance: the avatar speaks a given script and at the same time executes commands like “look left” or “raise your right hand”.
- World generation: the model extends the scene in response to commands for objects, actions and the camera.
- Instruction editing: the model changes an object in a live video based on a text command.
- Reference editing: the same, but the target appearance is given as an image.
Sessions are balanced by duration: 12 each at 15, 30, 60, 120 and 300 seconds. Together they contain 1,347 planned online events, and one full pass takes 525 minutes of active generation. A scheduler releases requests gradually on the session clock, so the model learns about a command only when it arrives. Each model runs every session three times.
Scenarios and Data
Qwen3.8-Max drafted the scenarios: session goals, timed commands, attributes that must stay intact and expected outcomes. Avatar portraits were generated with GPT-Image-2 and Qwen-Image (only synthetic adults or fictional characters), and speech was synthesized by Qwen3.5-Omni, including recordings with background noise and reverberation. For editing, the authors used licensed videos from Wikimedia Commons, NASA and Blender Studio with no cuts, looping or speed changes.
How Streaming Video Generation Models Are Scored
Ten metrics on a 0 to 100 scale fall into four groups: task execution (Dialogue, Following, Edit Success), state consistency (Stability, Continuity, Preservation), visual fidelity (Quality, Identity, Reference) and audiovisual synchronization (A-V Sync). For example, Following measures the share of commands the model executed on the correct object. Edit Success is stricter: an edit counts only if it appeared on time, affected only the intended region, persisted for the required duration and did not damage the rest of the frame. A-V Sync checks whether lip movement matches speech by comparing how wide the mouth is open at each moment with how loud the voice is at that moment.
Scoring is done by VLM judges (the LLM-as-a-judge approach). They were calibrated in advance on positive and negative probes: no more than 5% false positives and no more than 10% false negatives and unresolved decisions. For Identity, the lower score of two judges (Qwen3.8-Max-0902 and Qwen3-VL-Plus) is used, so a model cannot win by swapping the face for a similar one in the same style. Missing output or an unsupported operation scores 0. The final Overall score is the mean of the applicable metrics, and models that let users upload their own avatar portrait get a +5 bonus.
Results: Good Visuals, Weak Instruction Following
Avatars. Vidu S2 Avatar leads dialogue with 93.98 Overall and the best dialogue score itself (75.29). MuseTalk 1.5 is a telling case: Quality 95.34 and Stability 97.11, but lip sync of only 38.09.
In avatar performance the gap is even wider. Following ranges from 6.37 to 36.90 across all seven models, even though the leaders score above 80 on Identity and above 96 on Stability.
World generation. MemFlow has the best Overall (33.36), followed by Rolling Forcing (33.01). HappyOyster 1.0 Adventure produces the highest-quality picture (Quality 58.34) but executes only 5.56% of commands, and its Continuity is 2.78. Krea Realtime has the best Following among 17 configurations: 25.93.
Editing. Vidu S2 Editing leads instruction editing with 45.43 Overall and an Edit Success of 57.20, while JoyAI-Video-Edit preserves the source video best (Preservation 83.33). Even so, Quality does not exceed 22.99 for any model.
Reference editing turned out to be the hardest task: the top Edit Success is 15.00.
A recorded session shows what goes wrong. The model first has to dress a rider in a red velvet cape, switch it to a gold satin one at 10 seconds and remove the edit at 20 seconds. The model preserves the scene, but keeps the original grey cape instead of the gold satin one.
All scores come from automated judges, and the authors name comparison with human pairwise judgments as the next step. The editing tables show only configurations with Overall of at least 5, and this threshold was chosen after evaluation. The dialogue task does not test whether the model sees the user through a camera. One more detail: Jintao Zhang is part of the StreamEval team and is also the first author of the Vidu S2 paper, the model that leads four of the five tasks.
The authors’ main conclusion: streaming video generation is limited not by image quality but by reliable request execution under timing and preservation constraints. Models can already keep visuals and character appearance stable over long sessions, but still struggle to understand what, when and where the user asked to change.
