EmbodyHOI: Embodiment-Aware Generation of Hand–Object Interactions for Humanoid Dexterous Manipulation

Swipe sideways to see the whole figure.

Abstract

Humanoid dexterous manipulation needs reference motions that are both semantically rich and kinematically feasible for the target robot. Text-conditioned hand–object interaction (HOI) generators supply diverse, language-specified demonstrations, but they synthesize motion in human (MANO) space. The embodiment gap is therefore deferred to post-hoc retargeting, where trajectories deployed on the robot can fail to maintain contact with objects and also degrade policy learning. Our key idea is that robot reachability should constrain the interaction trajectories while they are being generated. We present EmbodyHOI, which runs a differentiable inverse-kinematics (IK) solver for a specific humanoid embodiment inside the sampling loop of a frozen flow-matching HOI trajectory generator. We feed back gradients from the embodiment-aware IK solver into the HOI sampler and jointly optimize the position of the humanoid with respect to the object. Because guidance acts only at inference, one HOI trajectory generator trained on human hand poses transfers across humanoid embodiments without retraining. On ARCTIC trajectories with two contrasting humanoids (Unitree G1 and H1-2), our proposed guidance-based method cuts the retargeted wrist error from 8.9 cm to 0.6 cm while leaving text-to-HOI generation quality intact, and raises task success from 68% to 79% on 50 random G1 tasks and from 62% to 76% on a four-task H1-2 suite. Critically, cheaper alternatives for bridging the embodiment gap fail to beat EmbodyHOI. Post-hoc wrist refinement and compute-matched best-of-N selection both reach lower IK energy than our guidance yet train worse policies.

Video

Method

EmbodyHOI makes a frozen HOI generator embodiment-aware at inference. Inside every unmasking step of the generator, a differentiable IK solve on the target humanoid scores the decoded wrist trajectories and places the robot’s base, and the gradient of the resulting IK energy is added to the flow-matching velocity field of the hand tokens. Finger and object tokens generated in later steps therefore form around reachable wrists. We retarget the guided sample and train a PPO tracking policy on it with decoupled arm and hand rewards. Changing the robot changes only the IK module.

Text prompt + target humanoid

“Close the waffle iron with the right hand.”

Only the IK module knows the robot.

1 · Condition

2 · Guided generation · Guided sample overlaid on the seed-matched unguided sample. Object tokens receive no gradient.

3 · Retarget · Guided wrists solved to arm joints from the optimized base; MANO fingers mapped to the XHand.

4 · Track with RL · PPO tracking policy trained on the retargeted reference.

Guided references train better policies

Policies trained on guided references succeed on 79% of 50 randomly drawn G1 tasks, against 68% for unguided references. The gain concentrates where reachability is poor: on the 10 worst-conditioned tasks, success rises from 57.9% to 82.1%.

Only the reference differs between the two policies below. Retargeting, rewards and PPO settings are identical.

Guidance corrects the wrists and keeps the interaction

Guidance moves the wrists of the seed-matched sample by 0.3 cm (median) and leaves penetration, contact and the object trajectory unchanged. Generation quality is intact on the same 400 ARCTIC test prompts (FID 0.545 vs. 0.667 unguided), while the G1’s wrist tracking error after retargeting falls from 8.9 cm and 20° to 0.6 cm and 4°.

Guided sample overlaid on the seed-matched unguided sample. Unguided hands are cyan and green; IK-guided hands are yellow and magenta.

Lower IK energy does not mean a better reference

Guidance during sampling trains the best policies, although two cheaper alternatives reach lower IK energy. Post-hoc IK refinement reaches the lowest energy but moves the wrists by 33 cm after the fingers and object are fixed, and success collapses to 32.8%. Compute-matched best-of-32 selection also beats guidance on energy, yet its policies succeed 5.7 points less often, because selection returns a different interaction instead of correcting the requested one.

One generator and one IK objective serve two humanoids

Switching robots changes only the IK module. From the same initial placement, guidance moves the H1-2 base (68.5 cm arms) away from the object to avoid over-extension and the G1 base (45 cm arms) closer to stay reachable. No term in the objective encodes either behaviour.

The gain carries through to policy learning. Guidance raises H1-2 success on the four-task suite from 62.3% to 75.7%, a 13.4-point gain comparable to the G1’s 11.6 points. The same objective also lowers the IK energy of a second generator, Text2HOI, on 48 of 50 prompts without modification.

Each component raises success

Guidance, in-loop base placement and the decoupled reward each raise success on the four-task G1 suite. Base placement is the larger contributor (4.0 vs. 2.9 points) and the cheaper one, since it adds no sampling cost. Re-solving the base inside the sampling loop rather than once after generation accounts for 2.8 of the 5.7 points that guidance adds over row C.

Limitations

Guidance is kinematic. Better reachability does not imply dynamic or contact feasibility, and the RL stage absorbs what guidance does not model.

BibTeX

@misc{chawla2026embodyhoi,
  title     = {EmbodyHOI: Embodiment-Aware Generation of Hand--Object Interactions
               for Humanoid Dexterous Manipulation},
  author    = {Chawla, Chaitanya and Sharma, Kartik and Bal, Ananya and
               Jeni, L{\'a}szl{\'o} A. and Shi, Guanya},
  year      = {2026},
  note      = {Under review}
}

Acknowledgements

EmbodyHOI builds on MAD-HOI for text-to-HOI generation, PyRoki for differentiable IK, dex-retargeting for finger retargeting, DexMachina for the simulation and RL stack, and the ARCTIC dataset.