Researchers from Tsinghua University and Shengshu Technology have introduced Vidu S2, two video models that let you build a digital avatar and hold a video call with it or run a livestream, while an incoming video stream can be edited in real time.
Vidu S2-Avatar turns a single photograph into a talking digital character: it speaks the audio track you feed it with accurate lip sync, changes facial expression, shifts its gaze, gestures, and follows text or voice instructions, up to and including dance. The stream has no length limit. A new reference image can be handed to the model at any moment mid-stream: show it a mug and the character picks it up, show it a jacket and the character puts it on, show it a beach and the character walks out of the room and ends up on the beach.
Vidu S2-Editing works on a video stream in real time and reshapes it on the fly from a text instruction and a reference image: it repaints the clip in a different style (watercolor, anime, cyberpunk and more than 50 others), changes the clothes on the person in frame, replaces that person with another character, or swaps the background. The motion and timing of the source are preserved frame for frame. Editing also applies to what a person sees through a headset camera, and both models can deliver their output as a stereo pair for VR.
Resolution has gone from 540p in the previous version, S1, to 720p at 25–42 FPS. On StreamAV-Bench the model took first place across all nine metrics at once; on the ViViD virtual try-on test its VFID dropped to 9.95 against 19.51 for the previous leader; and in blind pairwise comparisons evaluators preferred Vidu S2-Avatar over competitors in 85.7–100% of cases. The project is closed: no code, no weights and no datasets have been published. The models are available in the web app and through the API.
Real-time generation with Vidu S2-Avatar costs 1.5 credits per second, and a credit costs $0.005. That works out to $0.0075 per second, or roughly 45 cents per minute and $27 for an hour of continuous streaming. Voice cloning: the first 10 voices are free, after that about $4.5 per voice. New users receive trial credits when they register.
Why this is a hard problem
Sora, Veo and similar generators work offline: the user sends a prompt, waits several minutes and receives a finished clip in one piece. The diffusion model denoises the entire video synchronously over many steps, so an intermediate result does not exist even in principle. For live conversation, streaming and games this is unusable, because the picture has to respond to a person immediately.
Vidu S1 already generated an endless stream, but it hit a ceiling: 540p, a reference image fixed before the stream starts, poor reproduction of large body motion such as dance, and no editing of an incoming stream at all.
Self-Replay Forcing: gradients across the whole rollout
The stream is generated in segments, and each new segment builds on what the model generated itself. Small errors carry forward and gradually turn into facial drift or a complete collapse of the picture. Self-Forcing solves half the problem: the model is trained on its own segments rather than on clean frames from the dataset. Two gaps remain. The history is fed in clean, even though at inference time it is noisy. And it is detached from the computation graph, so no gradient flows through it.
Self-Replay Forcing closes both. The model performs a long autoregressive rollout, then the whole trajectory is independently re-noised and replayed in a single gradient-enabled pass. Within that replay the segments stay inside one graph, so an error in a late segment reaches the early ones through the gradient. Supervision comes from DMD, plus a perceptual loss that guards against collapsing into sameness. At the streaming stage they add Streaming NFT on top, reinforcement learning from human preferences on the model’s own trajectories.
How they reached 720p
The backbone keeps computing at low resolution, and a separate one-step Refiner raises it, working directly in latent space. The trick is in the asymmetric noise levels: the backbone holds a high-noise cache and handles long-range motion, while the Refiner holds a low-noise cache and handles fine detail and identity. Temporal consistency and spatial detail stop competing for the same capacity.
Data: clarity matters more than resolution
Training clips were selected not by nominal resolution but by measured clarity, because heavily compressed video looks mushy even at 1080p. The score is assembled from resolution, frame rate, codec, bitrate, edge sharpness and the severity of compression artifacts. Cut points are found by a VLM, with the false rejection rate held below 2%. A background stabilization operator geometrically compensates for camera movement so that dynamic dance clips do not have to be thrown out wholesale. Captions were redesigned to be chronological: events are described in order with boundaries in time, instead of a structural description of the scene.
For the editing model they collected 800,000 filtered videos and carved out four disjoint subsets of 200,000 clips each: style transfer, subject replacement, background replacement and virtual try-on.
How stream editing works
The key detail is called frame-aligned attention. Each target frame looks only at the source frame with the same temporal index, which is why the motion and timing of the source are preserved. The reference image, meanwhile, is visible to every frame, so the new appearance carries through the whole clip. Source frames are not kept in the cache; they are consumed together with the frame they produced.
Inference on low-cost GPUs
To hold real time on inexpensive GPUs, the authors assembled their own stack. For each layer they separately pick SageAttention, SpargeAttention or Sparse-Linear Attention. Linear layers are computed in per-block W8A8 GEMM. Adjacent operations are fused into custom Triton/CUDA kernels, and stable launch sequences are wrapped in CUDA Graphs. Ulysses-style context parallelism ships quantized tensors between cards. The VAE encoder, backbone, Refiner and VAE decoder share the same GPUs on a common schedule instead of each sitting idle on its own.
Spatial video for VR
A separate part of the work is spatial video. The finished stream is turned into a stereo pair, two slightly different images of the same scene taken as if from the positions of the left and right eye. That is how VR headsets work: each eye is shown its own frame, the brain fuses them into volume, and the viewer reads off the distance to objects. This sense of presence is the whole point, since on a flat screen a digital character stays a picture.
The conversion runs on top of an already generated stream. Depth is estimated for each frame and translated into a horizontal shift, and the frame is pushed apart into two views. At object boundaries the shift opens up areas that were not in the source frame; light post-processing fills them in, and depth estimates are smoothed over time so the picture does not jitter.
Results
On StreamAV-Bench, Vidu S2-Avatar posts the best value in every column: visual aesthetics 0.687 against 0.661 for Live Avatar, audio–video synchronization error 0.617 against 0.648, and subject consistency 0.998.
The editing model scores 3.74 on Sparkle-Bench and takes first place on all six criteria. In the joint OpenVE and RefVIE evaluation the total is 4.26 against 3.92 for the strongest offline model, Bernini-R 14B, meaning the streaming model beats the offline ones rather than merely catching up. On an internal benchmark of 150 paired cases, the editing model wins 72.7% of comparisons against Decart-Lucy2.5 and 86.7% against XMax-X2.0.
The degradation-over-time chart deserves separate attention. On clips from 10 to 90 seconds, competitors’ scores fall: Vidu S1 slides from 4.5 to 3.2 on overall quality, and PixVerse breaks off as early as 40 seconds. Vidu S2-Avatar holds an almost horizontal line at 4.7–4.9 across all five criteria.
In blind pairwise comparisons, evaluators picked Vidu S2-Avatar unanimously on motion quality against Runway and HeyGen, and on expression quality against PixVerse and HeyGen. The margin is thinnest on audio–video synchronization, where most pairs were judged equivalent.
Qualitative examples show the competitors’ typical failures: with PixVerse, facial features and body proportions drift; Runway distorts hair, fingers and eyebrows.
Limitations
The weak spots are visible in the paper itself. The internal benchmarks and pairwise comparisons were run by the authors with twenty evaluators, and the public benchmarks do not cover every claimed scenario. The team names spatial video as the main unsolved problem: it occupies most of the field of view and demands both higher resolution and lower latency at the same time, or the picture lags behind head movement. As their next goal the authors name panoramic spatial video, where a user could look around freely inside a scene generated on the fly. But the main limitation is practical: none of these numbers can be checked independently, because there are no weights and no code, only the demo and a paid API.
