Researchers from the University of Washington, USC, the National University of Singapore, UNC Chapel Hill, and the Allen Institute for AI taught the Unitree G1 humanoid robot to catch thrown boxes. Their method, Reflex, handles boxes of up to 2 kg thrown from up to 7 meters away. The robot sees the box only through the RGB-D camera on its torso. The box stays in the air for about a second. In that time, the robot steps into its path and receives the box on its forearms. Then it presses the box against its chest and absorbs the impact without losing balance.
Humanoid robots have caught basketball passes and blocked balls as goalkeepers before. However, they got the ball’s coordinates from motion capture or retroreflective markers. With Reflex, the real G1 humanoid robot caught 65% of throws from 2.5 m and 40% from 5 m. That is 76% and 59% of the human success rate at the same distances. So far, the project is only partially open. The paper, videos, and an interactive 3D demo are already available. Training and evaluation code, checkpoints, and robot deployment code will come later. There is no license yet, and there are no weights or datasets on Hugging Face. The code will go to the authors’ GitHub repository, and videos and the demo are on the project page.
What a humanoid robot must do to catch a box
The task combines three problems. First, whole-body control. Hands alone can catch a ball, but a box needs a different grip. The robot takes it on its forearms and presses it against its torso. At the same time, it absorbs the box’s momentum while stepping and keeping its balance. Second, latency. The box arrives in about a second. Meanwhile, the median latency from capturing a frame to the controller’s command is 69 ms. So reacting to the latest frame is too late. Instead, the humanoid robot has to predict where the box is going and pick the catch point in advance. Third, egocentric vision. The camera moves with the robot, so the box’s motion in the image mixes with the camera’s own motion. Also, right before contact, the robot’s arms block the box from view.
How Reflex works
Training a robot with reinforcement learning directly from pixels is expensive. Every simulation step would need RGB-D rendering, and the catch reward arrives only at the end of a throw. On a single RTX 3090, the 59 million steps of the second stage take about 35 minutes without rendering. With rendering, they would take about 11 hours. So the authors split the policy into two parts. Vision turns the history of frames into an eight-dimensional latent vector that describes the box’s dynamics. Then the controller turns this vector and the robot’s state into target angles for 29 joints, 50 times per second. Training runs in the Isaac Lab simulator in three stages:
- Whole-body control. A teacher policy receives the box’s position and velocity and learns to catch it with PPO. The box data are noisy, drop out, and arrive late, just as on the real robot. The reward encourages moving toward the box, holding it against the chest with the forearms, and keeping the catch. The humanoid robot also imitates human motions retargeted to its skeleton. Meanwhile, an adversarial motion prior (AMP) rewards the robot when a discriminator cannot tell its motion from a human’s;
- Box dynamics. A privileged encoder receives box data directly from the simulator. However, the data come with a 20-60 ms delay and gaps, just like camera data. The encoder compresses them into a vector z. First, the controller copies the teacher’s actions (behavior cloning). Then it acts on its own while the teacher suggests the right actions (DAgger). Finally, PPO fine-tunes it;
- Vision. A visual encoder learns to predict the same z. Its inputs are the last 25 RGB-D frames at 192×144 and the camera’s own motion (ego-motion). A convolutional neural network marks the box on a heatmap, and a GRU recurrent network tracks it over time. The controller stays frozen, so this stage uses ordinary supervised regression instead of RL. As a result, only about 1.6 million steps need rendering.
Results: humanoid robot vs. humans
On the real Unitree G1, the authors ran only 20 throws per condition. Because of this, the margin of error reaches ±20 points. From 2.5 meters, the humanoid robot caught 65% of throws, from 5 meters 40%, and from 7 meters 15%. Three people at the same distances caught 85%, 68%, and 28%, so from 7 meters everyone struggles. Small boxes, large boxes, and boxes weighted up to 2 kg were harder for the robot. It caught 20% to 60% of such throws. Sometimes the throwers aimed not straight at the robot but to its left or right. Even then, it caught 8 throws out of 20.
After the catch, control passes to the ready-made SONIC motion controller. The humanoid robot then carries the box 7.8 m, hands it to a person, and walks 7.3 m back. Out of 20 attempts, the catch succeeded in 65%, and the carry in 35%. The handover worked in 30%, and the full cycle including the return in 25%.
What matters under the hood
In simulation, the encoder cannot see motion without frame history. With a single frame, success drops from 85.2% to 49.6% on centered throws. At a 0.6 m offset, it falls from 66.6% to 26.8%. Without ego-motion, the results are 74.8% and 44.0%, and without heatmap supervision 77.8% and 55.7%. Training vision to predict the full box state instead of the latent also hurts: success drops to 66.5% and 47.1%.
Without the first stage, a controller trained from scratch catches only 6.7% of boxes. Behavior cloning in the second stage gives 5.2 points more on average than simply mapping the teacher’s weights. Latency randomization adds 5-11 points. Color augmentation, in turn, raises success on blue and black boxes from 60.2% and 61.2% to 88.0% and 88.8%.
When the landing point moves sideways, the visual controller outperforms the first-stage teacher. This happens even though the teacher sees the exact box state. The scores are 66.6% vs. 54.1% at 0.6 m and 35.6% vs. 23.4% at 0.9 m. The gain comes from PPO in the second stage. The authors also built a classic pipeline. It combines the same convolutional network as a detector, a Kalman filter, and the first-stage policy. Within the training distances, Reflex beat it in all 13 conditions, by 5.0-15.6 points. However, feeding the filtered state into the Reflex decoder shrinks the gap to 1.5 points on average. So the advantage comes from the second-stage latent interface. The learned temporal filter itself performs on par with a tuned Kalman filter.
Limitations
The main weak spot is long throws. The authors trained the third stage on throws from 3.0-3.5 m. As a result, at 5 m in simulation the visual controller catches 43.0% versus 79.7% with privileged data. Adding throws from 2.5-7 m to the training data raises success at 5 m to 75.8%. However, the team has not yet tested this on the real robot.
The controller also does not know the box size. In simulation, it intercepts small and large boxes in 58.1% and 86.1% of cases. Yet it holds on to them in only 25.5% and 39.7%. Moreover, the real latency, measured for the YOLO detector variant, exceeds the 20-60 ms range used in training. The rendering does not model motion blur or auto-exposure either. The authors see box catching as one of many skills where the body must get ready before contact. These include interception, avoidance, handing over objects, and physical assistance to people.
