Robotizing Human Videos Watch video

Human → Robot Video Editing for Manipulation Learning

Robotizing human videos with physically consistent interactions.

We preserve what matters in a manipulation demonstration: where the robot grasps the object, and how that interaction should appear in the scene.

Watch overview Explore the method

Ching-Lam Cheng, Shengfeng He, Bin Zhu

Singapore Management University

PROJECT OVERVIEW / 01 INTERACTION + VISIBILITY
From human demonstration to interaction-faithful robot supervision.
+22.25points OOD
DP-ViT vs. Masquerade
59.25%average OOD
success with DP-Swin
4 + 2simulation +
real-world tasks
01 / ABSTRACT

Beyond visual replacement.

Human videos offer scalable manipulation data, but simply rendering a robot over a human hand can create missed contacts and impossible occlusions.

Our pipeline addresses both failures. It reconstructs dense 3D hand–object contacts and converts them into stable parallel-jaw grasps, then uses scene and robot depth to enforce physically correct visibility. The resulting robotized videos are co-trained with robot demonstrations and improve policy performance—especially when the visual scene changes.

02 / MOTIVATION

Same demonstration.
More faithful interaction.

Existing video-editing methods may produce robot grippers that miss demonstrated contacts or appear in front of objects that should occlude them. Our method preserves both grasp geometry and scene visibility.

Comparison between a human demonstration, Masquerade editing, and the proposed interaction-aware result
Fig. 1 Human demonstration, prior editing, and our physically consistent result.

03 / METHOD

Two constraints.
One coherent interaction.

The pipeline treats physical consistency as both a geometry problem and a visibility problem.

A

Interaction-aware contact reconstruction

Hand–object segmentation and mesh-level prediction recover dense 3D contacts. We reduce these to a temporally stable thumb–finger pair and retarget the grasp to a parallel-jaw gripper.

  • Dense mesh contacts
  • Temporally stabilized grasp
  • Surface-normal constraints
B

Depth-consistent compositing

Scene depth is compared with rendered robot depth at every gripper pixel. Pixels behind the scene are removed, restoring the correct object–robot visibility relationship.

  • Inpainted-scene depth
  • Rendered robot depth
  • Occlusion-aware overlay
Overview of contact reconstruction, depth-consistent compositing, and policy co-training
Fig. 2 The full editing and policy co-training framework.

04 / RESULTS

The largest gains appear
when the scene changes.

Across four RoboTwin tasks, our edited demonstrations deliver the best average success rates with both visual encoders. Improvements over Masquerade are especially strong in out-of-distribution scenes.

15.5–22.25percentage-point OOD gain over Masquerade across encoders
87.25%average in-distribution success with DP-Swin
+3 / 10real-world OOD sponge grasps over robot-only training

05 / EVALUATION

From controlled simulation to a physical robot.

Policies are trained only in clean scenes, then evaluated both in-distribution and with new visual distractors. The same human videos and robot demonstrations are used for every co-training comparison.

Adjust BottleGrab RollerPlace Burger FriesShake BottlePick Up BoxPick Up Sponge
In-distribution and out-of-distribution RoboTwin scenes across four tasks
SIMULATION / ROBOTWIN
Real-world in-distribution and out-of-distribution scenes for sponge and box pickup
REAL WORLD / FRANKA RESEARCH 3

06 / TAKEAWAY

Human-video supervision is only as useful as the interaction it preserves.

Contact-aware retargeting improves grasp geometry. Depth-aware compositing preserves the visual evidence of interaction. Together, they turn abundant human videos into higher-quality robot-learning supervision.