Robot-Use Agents: Why General-Purpose Models May Win in Robotics

Video summary

Waddle Labs and RoboCurve argue general-purpose LLMs could control robots, with useful general-purpose robots potentially arriving within two years.

Waddle Labs co-founder Jaime and RoboCurve co-founder Jay explain why general-purpose language models may control robots better than robotics-specific systems. Earlier approaches such as RT-2 fine-tuned models to output robot actions, while coding agents can use existing programming knowledge, tools, and in-context learning to generate policies for new tasks with little robot-specific data. The guests describe RoboCurve’s evaluations across robot types and Waddle Labs’ harnesses for collecting data and building reusable skills. In a demo, Astra uses camera feeds and tool calls to move a block into a bowl; repetitive actions can later be compiled into faster skills, addressing latency that is reportedly improving around 2x per month. They connect computer-use and CAD data to spatial reasoning, and discuss turning task experience into programs or model updates. Jay says frontier labs expect general-purpose robots—able to follow natural-language instructions and handle unfamiliar tasks—to arrive within two years, though latency, skill management, and efficient learning remain challenges.

Chapters

1. 0:00Intro + The Rise Of Robot Use Agents: Waddle Labs, RoboCurve, and RT-2 robot poses
2. 4:07The Bitter Lesson for Robotics: RT-2’s web data and general-purpose LLMs
3. 7:40From Coding Agents to Robot Policies: Voyager and one-shot Python robot policies
4. 10:41In-Context Learning vs Model Training: ICL saturation at 20–40 examples
5. 14:22Building a Harness for Robot Control: compiled skills and 2× monthly latency gains
6. 17:50Turning Robot Actions Into Reusable Skills: Astra tool calls and program induction
7. 20:55How AI Models Learn the Physical World: CAD, computer-use data, and shared representations
8. 26:05How Close Are We to General-Purpose Robots? Jay Predicts They May Arrive Within Two Years

What the full analysis includes

8 chapters · 41 key points · 8 section summaries · mind map

The remaining 38 key points, section summaries and mind map are in the full analysis. Analyze this video to see everything →