Alaya Lab and the University of Tokyo have released WorldSculpt, a model that turns images of a room or a street into a 3D copy in which every item stays a separate object. Instead of fusing the scene into one continuous surface, the model outputs a set of meshes: you can click a chair, drag it aside, or drop it in a physics engine. Existing methods either fuse everything into a single blob of geometry or leave holes wherever one object is hidden behind another.
The numbers show how well this works. The authors built their own set of test scenes, UE-MeshyScene, where the densest one holds 701 objects. Quality is measured with CD-ℓ2, which tells you how far the generated surface drifts from the real one, so lower is better. The closest competitor, ShapeR, averages 7.42; WorldSculpt scores 2.48. The median error per object drops even further, from 2.47 to 0.25. Because the median improved more than the mean, the gain comes from the typical object in a scene rather than a handful of lucky cases. The share of correctly recovered surface (F-Score) rose from 0.813 to 0.951. In a separate Toys4k test where each view hid 75% of the object, sixteen views yield an error of 2.25 against 63.95 for single-view Pixal3D, roughly 28 times lower.
The project is open: project page, code on Github, weights on Hugging Face. The UE-MeshyScene benchmark, with 2,299 objects, 5,964 views, masks, 3D boxes and depth maps, is published in full.
Why compositional 3D generation is hard
Generative world models such as Marble and HY-World 2.0 already hallucinate plausible geometry from a single picture. The catch is the output format: the scene arrives as one fused mesh or a cloud of Gaussians, where a chair and a vase are not separate entities. For games, AR/VR, simulation and robotics that result is useless, because these applications need individual assets they can move independently.
Geometry-first methods (DUSt3R, VGGT and other pointmap models) recover visible parts of a scene accurately, but occluded regions stay empty. Compositional approaches such as MIDI, SceneGen and SceneMaker do split a scene into objects, yet they were only tested on simple layouts along the lines of a few items on a tabletop.
WorldSculpt architecture
The core idea is simple: the model is never trained on scenes at all. WorldSculpt is a wrapper around Pixal3D. The authors took an off-the-shelf single-object 3D generator, froze its weights, and trained only a small set of new layers on top, which let the model take several viewpoints into account at once.
Pixal3D itself belongs to the native-3D family, meaning it operates directly in a three-dimensional representation rather than assembling shape from rendered pictures. It is built on the TRELLIS.2 backbone and generates an object through several consecutive passes (stages) using flow-matching. Only the first two geometry stages are used here. The first roughly marks which cells of a 64×64×64 grid the object occupies and which stay empty. The second refines shape inside the occupied cells and produces a high-resolution representation that is later decoded into a mesh. The texture and material stage is left untouched.
Two things separate WorldSculpt from Pixal3D. Pixal3D works from a single picture and fills in hidden parts based on general priors about how similar objects look, while WorldSculpt gathers evidence from every frame where the object appears, so what is hidden in one view is taken from another. And Pixal3D outputs an object on its own, unanchored to any scene, whereas WorldSculpt generates it directly in scene coordinates, so the generated items land in their proper places.
For each object the pipeline builds a virtual canonical cube whose orientation comes from an anchor view, the frame where the object is seen most fully. Views are projected into that cube, DINOv3 features from every view are lifted into a voxel volume, and then merged by an IBRNet-style aggregator. The aggregator accepts any number of views and ignores their order: it computes cross-view mean and variance, and two lightweight MLPs predict per-feature corrections and softmax weights. The resulting 3D condition is injected into the frozen prior through zero-initialized projections, while the network adapts via LoRA (rank 32, scaling factor 32). Only the aggregator, the conditioning injection layers and LoRA are trained.
One more trick makes the transfer to real scenes work. The model learns from clean renders of isolated objects, while in an actual scene an object is barely visible: neighbours block it, its mask is cut sloppily, the camera pose is imprecise, and the item takes up a tiny patch of the frame. To close that gap, the authors degrade the input images during training in four ways: they occlude the object with random 2D masks or with 3D-consistent occluders that move coherently across views, add noise to non-anchor camera poses, corrupt mask boundaries, and downsample resolution. The degradation ramps up: each distortion grows from zero to its maximum over the first 3k iterations, so the model masters clean examples before facing hard ones. The ground-truth geometry is never touched, only the input.
A phone video alone is not enough. The model has to know where each frame was shot from and at what angle, and it also needs object masks and coarse 3D boxes. Camera trajectory comes from COLMAP or VGGT, masks from SAM3, so several separate pipeline steps run before WorldSculpt starts.
UE-MeshyScene: a benchmark built for real clutter
Existing datasets either offer few objects or a fused, incomplete mesh instead of honest per-object geometry. The authors assembled six environments in Unreal Engine 5.8, an aircraft hangar, an abandoned city, an old cathedral, an office, a Japanese school and a desert town, and rendered them at 2560×1440. Each scene holds between 93 and 701 objects and between 265 and 1,758 views. The dataset totals 2,299 objects and 5,964 views. The scenes were composed by hand from known 3D models, so the authors know the exact shape of every item and its position in advance. The ground truth is exact, and metrics can be computed in the world coordinate frame, meaning the axes shared by the whole scene, where every item has coordinates relative to a single origin rather than only a shape of its own.
One protocol detail matters here. For single-object evaluation the predicted mesh is aligned to the ground truth with ICP (Iterative Closest Point), which searches for the rotation and translation that make the two surfaces overlap best. That makes the metric independent of the orientation a method happened to output and measures shape quality alone. On scenes ICP is not applied: the object is compared to the ground truth exactly where the model placed it. The resulting number therefore reflects both the geometry of the item and the accuracy of its placement.
What the metrics show
Quality is measured with Chamfer distance (the average distance between the predicted and the ground-truth surfaces), Earth Mover’s Distance, and F-Score at thresholds 0.02 and 0.05.
On clean inputs with a single view the model roughly matches the original Pixal3D (CD-ℓ2 2.59 versus 2.17), which indicates that the multi-view adaptation barely hurt the underlying generative ability. Then the gap opens. With sixteen views and no occlusion CD-ℓ2 reaches 1.19, and quality holds up all the way to 50% per-view occlusion: 1.44 against 22.81 for single-view Pixal3D in the same regime. TRELLIS in multi-view mode gains little from extra frames, its CD-ℓ2 at 16 views and 50% occlusion stays around 18.51.
Scenes tell the same story. On HouseCat6D, real captures of tabletop scenes, CD-ℓ2 is 0.28 against 1.26 for ShapeR. On the synthetic Toys4k-Scene it is 0.61 against 8.38, with F-Score 0.981 versus 0.746.
The view-fusion ablation confirms the logic of the method. The learned aggregator is nearly indistinguishable from plain feature averaging when an object is well observed (HouseCat6D), but on UE-MeshyScene it cuts CD-ℓ2 by 12% and lifts F-Score from 0.944 to 0.951. Weighting views pays off precisely when observations differ in reliability.
Converting generated 3D worlds into editable meshes
The authors tested one more scenario. Marble is a World Labs model that generates a whole 3D world from a single image or a text prompt. It delivers that world as 3D Gaussian Splatting, a cloud of translucent blobs that renders beautifully from any viewpoint but contains no separate items and no meshes you could edit.
WorldSculpt turns such a world into a set of objects. First a virtual camera flies through the generated world and captures ordinary frames along with depth maps. Camera poses are known exactly, since the frames are rendered programmatically. SAM3 then detects and tracks objects across the sequence. A single item occasionally breaks into several tracks, and those fragments are merged back using mask overlap and spatial proximity. Finally the object’s pixels are back-projected into 3D space, outliers are cleaned with DBSCAN, and the remaining cloud is wrapped in a bounding box.
That completes the preparation. Standard WorldSculpt inference then runs without any retraining on the new data, and the Gaussian cloud becomes a set of editable meshes.
Limitations
The method leans on external estimates of camera poses, masks and coarse boxes. The augmentations help it survive moderate errors, but a badly misplaced box still ruins the geometry. The model generates geometry only; textures and materials are left for future work, even though both Pixal3D and TRELLIS.2 support them. Finally, the scene is assumed static: all views are treated as observations of the same fixed shape, so moving or deforming objects are not supported yet.
