HiPHI: A Large-Scale Benchmark for High-Precision Human Motion and Object Interaction

What you will learn about: Why humanoid robot learning, a central problem in embodied AI and Physical AI, needs data that internet video and existing motion capture datasets cannot provide. How FrameNet, a linguistic framework for human action, can guide motion capture collection to systematically cover a broad range of whole-body motion. Why synchronized object trajectories and meshes make human-object interaction data useful for teaching robots real-world tasks such as carrying, pushing, and pulling. How reinforcement learning policies trained on this motion capture data improve with scale, and how sim-to-real transfer carries them onto a physical humanoid robot. Humanoid robots must learn to balance, move, and interact with objects across an enormous range of situations, but the data available for this training has clear limits. Internet video shows diverse behavior but cannot capture precise physical states. Laboratory motion capture systems record accurate movement but usually cover only a narrow set of actions. This white paper examines HiPHI, a 617.5-hour whole-body human motion dataset captured with optical motion capture at sub-millimeter accuracy. It includes 245.7 hours of human-object interaction with synchronized object trajectories and meshes, and organizes coverage using FrameNet, a linguistic framework for human action. The paper also introduces a benchmark suite for measuring motion diversity and interaction grounding, and reports results from policies trained on the dataset and deployed on a physical Unitree G1 humanoid robot.