Researchers from Zhejiang University, Robbyant, Ant Group and HKUST introduced 4DAnyone, a framework that turns a video of a person shot on a single camera into a 4D model of that person, viewable from any angle. The model generates 16 mutually consistent viewpoints, and a volumetric scene in 4D Gaussian Splatting format is then reconstructed from them. Until now this required a studio with dozens of synchronized cameras. In reconstruction quality 4DAnyone beats every method it was compared against: 24.15 PSNR versus 20.55 for the closest competitor on DNA-Rendering, and 23.28 versus 19.86 on DyMVHumans. The framework is built on Wan2.2-TI2V-5B, a video diffusion transformer fine-tuned to generate new viewpoints of a person from a 3D skeleton while preserving the appearance from the source video. The project is open: the code is published on GitHub, the weights on Hugging Face.
Limitations of existing methods
A 3D model is static geometry: an object you can walk around, but frozen in a single pose. A 4D model adds a fourth axis, time. The scene changes from frame to frame, the person moves, and at every moment in time a complete volumetric representation of them exists. In other words, you can scrub through time and change the viewpoint independently of each other.
Building a 4DGS model of a person normally takes a calibrated studio with dozens of synchronized cameras, such as the 48-camera rig used in DNA-Rendering. The obvious alternative is to first generate the missing viewpoints with video diffusion and reconstruct afterwards. Existing camera-controlled models do produce such videos, but consistency falls apart once you scale to dozens of viewpoints.
The authors trace this to an architectural constraint: the attention context of a single DiT forward pass is bounded by memory and compute. When there are more target views than fit at once, they have to be split into groups, and two bottlenecks appear. The first is on the reference side: giving each group every previously generated view overflows the context length, while trimming it weakens the appearance guidance. The second is on the target side: groups are denoised independently and exchange no information, so the global structure of the person drifts from group to group.
A skeleton instead of a depth map
Methods like Gen3C or TrajectoryCrafter rely on dense geometry: depth maps and camera parameters. For casual videos these estimates are unreliable, and errors in them impose contradictory constraints that make generation diverge. The authors formulate an accuracy-over-density principle and take a sparse but dependable signal instead: a 3D skeleton, which modern pose recovery methods (here GVHMR) extract from monocular video with confidence.
The model receives the skeleton not as a list of coordinates but as an image, rendered for each target viewpoint. Rendered flat, it loses depth information: an arm in front of the chest and an arm behind the back produce the same line across the torso, and the model cannot tell which pose to reproduce. So the rendering accounts for distance to the camera, and nearer body parts occlude farther ones, as in ordinary 3D rendering.
The skeleton keeps 40 keypoints: 17 on the body, 6 on the feet, 10 at palm level and 7 auxiliary ones. The detailed face and finger-joint annotations were dropped, because those points are detected imprecisely and the model, following a wrong position, produces artifacts. Facial expressions and hand shape it takes from the source video instead.
Two mechanisms against drift
Reference Context Packing (RCP) solves the growing-context problem. Neighboring viewpoints are highly redundant, so storing them at full resolution makes no sense. RCP applies patchify layers with different strides: instead of the standard (1, 2, 2) kernel it uses variants compressing by 2× and 4×, yielding 4 and 16 times fewer tokens. References are packed into a fixed set of slots, and reference-context complexity drops from O(N) to O(1) no matter how many views have already been generated.
Target Context Routing (TCR) solves the isolated-groups problem. The observation is straightforward: high-noise steps establish global structure, low-noise steps refine details. So at high noise the view indices are cyclically shifted, groups are reassembled at every step, and information spreads across all viewpoints. At low noise the groups are fixed on adjacent views to refine details stably. The switching point was found by sweeping: quality improves up to t_s/T = 0.2 (16 of 20 steps with shifting) and then plateaus.
Data and training
A separate contribution of the work is MVGameHuman, a synthetic dataset rendered in the authors’ own game engine. Instead of a real studio, a character is captured by 24 virtual cameras at once, which gives perfectly synchronized viewpoints: 38k videos with 318 characters at 2560×1440 resolution. To it they added the synthetic SynCamVideo (34k videos), recordings from a real studio in DNA-Rendering (51k videos from 48 cameras), and the ordinary monocular TedTalk and Pexels sets so the model would also handle everyday footage.
Training proceeds in three stages:
- first, DNA-Rendering alone with the background removed: the model learns the core skill of turning a person to follow the skeleton;
- then all multi-view datasets are brought in with backgrounds, so the model sees lighting and shadows rather than just a silhouette on a green screen;
- in the final stage monocular footage is added and the finger keypoints are dropped from the skeleton.
Results
4DAnyone was compared against MV-Performer, TrajectoryCrafter and ReCamMaster, with the last one fine-tuned on the same data and equipped with the same RCP and TCR, to fairly contrast implicit conditioning on camera parameters against explicit conditioning on a skeleton. 4DAnyone wins on all three evaluation dimensions: consistency of the generated videos (24.33 PSNR versus 21.47), quality of the resulting 4DGS, and accuracy of the videos themselves against ground truth.
Qualitatively the picture looks like this: MV-Performer renders the front acceptably but distorts side and back views, TrajectoryCrafter accumulates depth errors and breaks down on a 180-degree turn, and ReCamMaster produces plausible videos with imprecise camera placement, which leaves the 4DGS noisy.
The ablation confirms that both mechanisms are needed together. Without TCR and RCP consistency drops to 21.09 PSNR, each component on its own gives roughly 22.0–22.2, and the full version with sliding regrouping reaches 22.63. Curiously, randomly permuting the viewpoints yields no gain, and strided permutation actually degrades the metrics: preserving local adjacency between views matters.
Where the model breaks
The skeleton says nothing about clothing that moves independently of the body. A long flowing dress is generated differently in different viewpoints, and the reconstruction degrades. The second failure mode is inherited: if the pose estimate is wrong, generation faithfully follows the wrong skeleton. In the paper’s example a ballerina stands en pointe while the pose estimator predicts a flat foot, and this error repeats across all 16 viewpoints.
The authors address the risks separately: the model synthesizes realistic videos of people, so it can be used for deepfakes and privacy violations. Their position is that results should be labeled as synthetic and created only with the consent of the person depicted.
