Robot-Use Agents: Why General-Purpose Models May Win in Robotics
A discussion with the founders of Waddle Labs and RoboCurve on how general-purpose language models are beginning to control robots. The conversation covers the lineage from RT-2 and code-as-policies to in-context learning, harness design, skill consolidation, Platonic representations, and the path toward general-purpose robots.
The conversation opens with a claim that is hard to ignore: coding agents generalize far beyond software, and frontier researchers are now showing that this extends to controlling robots. The speakers build on a recent essay by MIT professor Philip Isola arguing that we may be entering an era of robot-use agents, general-purpose models that can drive different robots, write policies, and learn new physical tasks with little or no robot-specific training. Two startups anchor the discussion. Waddle Labs builds LLMs that control robots through a harness and uses the data it collects to train better models. RoboCurve is an evaluation company for physical AI that measures arms, grippers, humanoids, and quadrupeds across models ranging from LLMs to VLAs and world action models.
From RT-2 to the bitter lesson for robotics
The early lineage starts with RT-2, which fine-tuned a pre-trained vision-language model on web text and images and had it output end-effector poses instead of English. Those poses could then be translated into joint commands. The speakers draw a parallel to chain-of-thought prompting. A VLA that must emit an action directly has no room to spend more compute on a harder problem, whereas a model that writes intermediate reasoning or code can think before it acts. Code is described as a compact, low Kolmogorov complexity form of that reasoning: the model states a plan in a few lines and then executes it.
The bitter lesson gets a robotics twist. One speaker argues that architecture matters less than data. VLAs are already built on language models and can reason and write code, so the real bottleneck is the VLA data problem, which has progressed slowly. The better strategy is to pick a modality that is abundant and well understood by language models and transfer it into robotics, turning an out-of-distribution domain into an in-distribution one. RT-2 benefited from inheriting broad knowledge from web-scale pre-training. Since then, models have improved at tool use, at writing complex policies as code, and at exploring their environments, such as discovering which arms they can access. Much of this comes from in-context learning rather than architectural change. The conclusion is that a strong general-purpose LLM may be a better foundation for robots than a robotics-specific model that depends on robot data.
Code as policies and Voyager
Code as policies is presented as an early signal of this trajectory. Voyager, which played Minecraft by writing and storing reusable skills as new functions, was an early example of on-the-fly tool creation. The speakers suggest that in 2024 the frontier labs bet heavily on coding as a route to automating ML engineering, and that the robotics implications were more a side effect than a deliberate insight. Google DeepMind’s code-as-policies work gave coding agents Python functions such as pick up an object or move to a pose. The model then composed these into programs that executed complex manipulation tasks in a single shot, without additional robot data. That one-shot, in-context exploration ability motivated much of the later work, including the speakers’ own.
In-context learning versus weight updates
The host, Francois, lays out a spectrum of learning: knowledge stored in weights, in-context learning, LoRA at various ranks, and full SFT or RL. He describes an experiment in which a trained LLM is given a held-out task (GSM8K) and improved through in-context examples. ICL improves performance cheaply, with no gradient descent and no FLOPs spent on training. However, the improvement is non-monotonic, getting worse and better in turns, and it saturates after roughly 20 to 40 examples. It is also bounded by the context length the model was trained on, and by whether the model was post-trained to use multiple turns for self-reflection. Retrieval into active memory is seen as a promising extension, closer to how an agent with a textbook can look things up. The key takeaway is that ICL is powerful in low-data regimes but cannot replace training when data is abundant. The open question is how to compress in-context experience into tools or learned skills.
The guests respond that a harness can consolidate context into packaged skills, which is a form of distillation from past experience. They relate this to meta-learning, where a large model programs smaller models that learn in context inside the harness. Those smaller models can be made faster and smaller, as long as the big model handles domain specialization well. How far ICL can go is described as anyone’s guess. Some evidence suggests transformers can approximate gradient descent in context, which blurs the line between weight space and symbolic space.
Harnesses, skills, and what the demos show
The demo shows Astra (as transcribed) controlling arms to pick a block off the table and place it in a bowl. It uses camera feeds and issues tool calls that set end-effector poses over multiple turns, which the speakers describe as closer to tool calling than to code as policies. The speakers contrast direct use of the model with routing the task through the Waddle harness. Having the model command every pose directly is inefficient for repetitive tasks. For those, one should write code that runs quickly or call a compiled skill for a task seen before, which also handles edge cases better. Latency is the visible bottleneck, and the speakers note that for certain LLMs it has been improving about 2x per month. If that trend holds, real-time control could arrive by the end of the year.
The speakers’ view of program structure is that much of manipulation becomes deterministic once a task has been performed a few times. Variation enters at specific points, such as detecting an object, checking whether a step failed, or deciding how to recover. A VLM is placed inside the loop at those points, so the overall code graph stays largely deterministic while still handling variation.
A related idea comes from program synthesis. Transduction learns a function that maps inputs to outputs directly, which is data-hungry and information-inefficient. Program induction instead emits a function from a few examples, which is far more sample-efficient, provided the inductive biases are right. The speakers argue that this is what LLMs buy: the emitted program is code, and the hard problem shifts to finding the right inductive biases, which earlier program synthesis methods struggled with. They connect this to neurosymbolic methods, where neurons emit symbols in the form of code.
Platonic representations and why spatial intelligence improved
The discussion moves to Isola’s Platonic Representation Hypothesis, which holds that models trained on increasingly large and overlapping data converge toward similar representations of the world. If that is true, a very strong language model implies a strong robotics model. The bitter lesson then points toward one sufficiently strong general model outperforming specialized models that are slightly weaker.
The host raises a puzzle: why the newest model seems markedly better at spatial intelligence, with a step-function improvement on certain benchmarks, even though it still benefits from scaling in code and math. The guests point to vision capabilities and to pre-training on much more computer-use data and CAD data. Computer-use data is not obviously relevant to robotics, but a speaker explains that dragging a cursor to orbit a CAD object teaches a model about space, including top-down, left-right, and other concepts needed to control a robot. Robotics researchers have been broadening their data too, moving from teleoperation to egocentric video, so the logical extreme is to feed coding, computer-use, and egocentric data into one model. One speaker notes the historical irony: graphical interfaces were built to resemble the physical world, and now computer-use environments have become useful training grounds for physical robots. Harnesses that make the robot’s tools look like computer-use tools, such as an agent dragging a cursor to direct a robot, improve LLM performance on physical tasks.
How close are general-purpose robots
There is reported consensus among frontier labs and robotics companies that general-purpose robots could arrive within two years, or even sooner. The speakers define this as a system that can follow any natural language instruction and do what a competent teenager could do with their hands, generalizing to unseen tasks and environments. They describe it as a ChatGPT-style moment for robotics, and they suspect society is largely unprepared for it.
For Waddle Labs, the near-term challenge is latency and economics. Running a frontier model in the loop at every step is too slow to be useful, so the goal is to consolidate a first pass of in-context learning into a faster skill or policy that can run repeatedly at high throughput. The speakers draw an analogy to sleep. Contextual learning is fast, but consolidation happens offline, as memories compress into weights during sleep. A workable pattern would collect data, reflect on it, and distill it into an updated weight file, which might be a model other than the original. The discussion references DreamCoder, which refactored its skill library during a sleep phase into more compact representations. Similar refactoring may be needed for traces and skills in robotic code-as-policy systems, where growing skill libraries need pruning and organization. Managing that growing set of skills and deployment data is seen as a central open problem for the next few years.