Robot-Use Agents: Why General-Purpose Models May Win in Robotics

Video summary

Waddle Labs and RoboCurve argue general-purpose LLMs could control diverse robots, with teenager-level general-purpose robots possible within two years.

Waddle Labs co-founder Jaime describes building robot-control harnesses and training models with collected data, while RoboCurve’s Jay evaluates models across robot types and environments. The guests trace the shift from RT-2, which fine-tuned language models to output robot poses, to coding agents that use tools and write reusable robot policies. They distinguish quick, low-cost in-context learning from training updates, noting it can saturate after roughly 20–40 examples and is limited by context length. In demonstrations, Astra directs robot arms using camera feeds and tool calls; repeated actions can be compiled into faster skills, especially as model latency reportedly improves about 2× per month. The discussion argues that coding, computer-use, CAD, and egocentric data can teach spatial understanding. Jay predicts natural-language robots capable of tasks a competent teenager could do may arrive within two years, while latency, skill compression, and organizing learned skills remain major challenges.

Chapters

1. 0:00Intro + The Rise Of Robot Use Agents: Waddle Labs, RoboCurve, and RT-2
2. 4:07The Bitter Lesson for Robotics: Web Pretraining and General-Purpose LLMs
3. 7:40From Coding Agents to Robot Policies: Voyager and One-Shot DeepMind Methods
4. 10:41In-Context Learning vs Model Training: ICL Saturates After 20–40 Examples
5. 14:22Building a Harness for Robot Control: Ashra’s Block Demo and 2x Monthly Latency Gains
6. 17:50Turning Robot Actions Into Reusable Skills: Astra Tool Calls and Program Induction
7. 20:55How AI Models Learn the Physical World: Astra, CAD Data, and Shared Representations
8. 26:05How Close Are We to General-Purpose Robots? Two-Year Outlook, Astra Latency, and Skill Distillation