Manipulation Data Pyramid: From Human Video Pretraining to Physical RL
Scaling laws are now often seen as a key ingredient on the path toward general intelligence. But in robotics, progress is slowed by one major obstacle: the lack of abundant, high-quality data. In this talk, I introduce a data pyramid strategy designed to tackle this challenge by making the most of diverse data sources. The idea is simple but powerful: combine internet-scale datasets, human teleoperation data, and robot-collected experiences so that each strengthens and fills in the gaps of the others. I’ll walk through three examples of this strategy in action:
A system that learns directly from human VR recordings, enabling zero-shot motion generalization.
A study showing how data scaling laws emerge in imitation learning with manually collected demonstrations.
A new reinforcement learning framework that leverages foundation models to speed up learning in the physical world.
The central message is that data is the wellspring of knowledge. By shifting our focus from purely designing algorithms to strategically building and using data, we can start to address some of the biggest challenges on the road to general robotic intelligence.
Bio: Yang Gao is an Assistant Professor at the Institute for Interdisciplinary Information Sciences (IIIS), Tsinghua University, and co-founder of Spirit.AI, a company focused on building general-purpose robots. He received his Ph.D. from UC Berkeley under the supervision of Prof. Trevor Darrell, and later continued as a postdoctoral researcher with Prof. Darrell and Prof. Pieter Abbeel.
His research centers on robotics, with a particular focus on robotic learning. Before Berkeley, he earned his bachelor’s degree in computer science at Tsinghua University, where he worked with Prof. Jun Zhu on Bayesian inference. He has also explored autonomous driving during a research internship at Intel Labs in 2018 with Dr. Vladlen Koltun.
Please visit https://stanfordasl.github.io/robotics_seminar/ for this quarter’s lineup of speakers. Although we encourage live in-person attendance, recordings of talks will be posted also.
Covid-19 related instructions: We recommend wearing a well-fitted, high-quality face covering inside the classroom.
If you’re interested, you’re welcome to join Yang for lunch at Blend Cafe at 12 PM.