Robot-Use Agents: Why General-Purpose Models May Win in Robotics

Video summary

Waddle Labs and RoboCurve argue general-purpose LLMs could control robots within two years.

Decoded examines the rise of robot-use agents with Waddle Labs and RoboCurve. The guests argue that general-purpose language models may outperform robotics-specific systems by combining coding, computer-use, vision, and robotics data. RT2 showed how pretrained language models could output robot end-effector poses, while newer agents such as Astra can use camera inputs and tool calls to pick up blocks, manipulate objects, and create reusable skills. In-context learning adapts quickly but saturates after roughly 20–40 examples and is limited by context length, making skill compilation and later weight updates essential. RoboCurve evaluates models across arms, grippers, humanoids, and quadrupeds; Waddle Labs builds the harness and training pipeline. Model latency is reportedly improving about 2× per month, potentially enabling real-time control by year-end. The guests predict competent general-purpose robots within two years, while warning that latency, data, skill management, and economic deployment remain major challenges.

Chapters

1. 0:00Intro + The Rise Of Robot Use Agents: Philip Isola’s robot-use agent thesis
2. 4:07The Bitter Lesson for Robotics: General-purpose LLMs over robot-specific models
3. 7:40From Coding Agents to Robot Policies: Voyager and one-shot robot control
4. 10:41In-Context Learning vs Model Training: ICL saturates after 20–40 examples
5. 14:22Building a Harness for Robot Control: From camera inputs to real-time control
6. 17:50Turning Robot Actions Into Reusable Skills: Deterministic policy graphs with VLM variation
7. 20:55How AI Models Learn the Physical World: Platonic representations and computer-use data
8. 26:05How Close Are We To General Purpose Robots? General-purpose robots may arrive within two years