Real2Sim2Real for Non-Prehensile Manipulation

Non-prehensile motion allows for the manipulation of objects too large to fit in a parallel-jaw gripper. We're broadly interested in coming up with a scalable method to learn these manipulation strategies, inspired by the prior works in Real2Sim and Sim2Real!

Capture real objects and a human demonstration
Build a digital copy of the cell and objects, plus contact targets from the tracked hand and object
Train with reinforcement learning in simulation, guided by those contact targets
Render successful runs from many views and lighting conditions
Train a vision policy on the renders, then test it on the robot

Real2Sim

Parallel-jaw graps can tolerate small shape errors, but in non-prehensile manipulation (such as pushing or tipping), a small shape error changes where contact happens and how the object moves. With this method, I observed errors under 1 mm (< 1%) in key dimensions, which in combination with domain randomization of the mesh during PPO, helps policies transfer to real.

An agent does the reconstruction, inspired by Agentic Real2Sim and RoboSimStudio. The goal is for this pipeline to work robustly for arbitrary objects.

Mesh construction

The agent separates the object from its background, then writes code that fits a mesh to the depth and outlines across views. It uses the simplest shape the evidence supports and labels unseen surfaces.

Photograph of a red food can on a foam mat — Photo

Grayscale depth map of the same view, where darker is closer — Depth

The can separated from its background — Object separated from background

Untextured gray 3D mesh of the can from the same view — Fitted mesh

Photo of the can with the outline of the photo and the outline of the mesh drawn on top of each other — Fit check: photo outline (green), mesh (red)

Appearance matching

This step is important for downstream VLA post-training. We project the training photos onto the mesh and renders it in Blender. The agent reviews the errors and can suggest a material or shape change, which is only kept if it holds up on the validation views.

Flattened texture map of the can built from the photographs — Photos projected onto the mesh

Render of the textured mesh from the same view as the photo — Render

Heat map of the difference between the render and the photo, brightest around the printed text and edges — Error against the photo

Photo of the can from a view withheld from fitting — Photo the agent never saw

Render of the can from that withheld view, with some visible texture seams — Render of that view

Three rows of bottle views in five columns: reference photograph, published final asset, fixed-geometry control, fresh reconstruction with visible surface artifacts, and refined reconstruction with those artifacts removed. — Left to right: photo, an earlier model, a comparison model, a first attempt, and the refined result. The refined outline matches the photos closely (silhouette IoU 0.95 on test views), after some directed refinement.

Object pose and contact point estimation

With the human demo, we track their hand with WiLoR and the cup with FoundationPose. Wherever the hand comes within 10 mm of the cup, we mark that spot on the cup's surface as a contact target, adapting the contact extraction from CHORD.

Human demonstration with tracking overlaid. The first tip begins: one fingertip contact on the base of the upside-down cup. — The recorded demonstration with tracking drawn on top. Blue: tracked cup. Red: cup surface near the tracked hand. Yellow: contact points. These are estimates based on distance, not measured touches or forces.

Human demonstration with tracking overlaid. The cup tips over as two fingertips push on its base. — The recorded demonstration with tracking drawn on top. Blue: tracked cup. Red: cup surface near the tracked hand. Yellow: contact points. These are estimates based on distance, not measured touches or forces.

These targets feed into the reinforcement learning reward. The policy earns credit for reaching them and for turning the cup while it touches them. The largest reward is for leaving the cup upright, released and settled. Penalties cover too much force on the cup, forbidden collisions, and pressing into the table.

Task: flipping a solo cup

Splitting the task using the cup's pose

Two learned policies handle the flip. Once the cup is on its side and nearly still, a scripted step repositions the hands and the second policy takes over.

First policy pushing the base of the upside-down cup — First policy: cup upside down

Cup resting on its side with both grippers clear of it — Second policy: cup on its side

A soft, rotationally symmetric contact reward

We turn each demonstrated contact point into a ring. The reward fades smoothly with distance from that ring, guiding the policy without forcing an exact motion.

Gray cup with yellow dots on its base, marking contact points seen in the human demonstration — Contact points from the demonstration (first tip)

Gray cup with a soft red band covering the base, centered on the demonstrated contacts — Reward target during the first tip

Gray cup with a soft red band around the wall near the base — Reward target during the second tip

Brighter red means more reward for touching that spot.

Results

The full sequence completed 19 of 32 tries in simulation. Here are six tries picked at random, three that worked and three that didn't:

Worked

Failed: the second policy misses the cup

Failed: repositioning stops, and the cup stays on its side

Failed: the cup rolls sideways on the second turn