Robot use means executing tasks, not describing them
RobotWorld asks how far the tool-using, code-writing abilities of general-purpose agents carry into the physical world. The paper defines robot use as turning instructions and observations into physical task execution through robot interfaces. That definition matters because it makes success depend on the final physical state, not on whether an agent produced a plausible plan or a working piece of code.
The testbed contains 84 tasks spanning manipulation, mobile manipulation, locomotion, driving, and aerial control. Each task comes with an explicit interaction budget and an executable success check, so an episode is scored by whether the environment's checker confirms the goal within the allowed steps. Agents receive the task goal, permitted observations, and tool documentation, and they issue direct robot-tool calls while replanning from returned feedback. Environment-side code control was disabled in the reported evaluation, and agents could not read hidden checker state or reference solutions.
The evaluation covered GPT-6 Astra, Claude Opus 5.5, Kimi K3, DeepSeek V4.1 Flash, and Gemini 3.8 Flash on all 84 tasks, with one episode per model–task pair. That produced 420 scored episodes, collected from 30 September to 6 October 2026, using the same initialisation protocols, control-step horizons, and scoring rules across models.
Agents build real workflows but do not compose them
The paper's central finding is not that agents lack robot-relevant skills. Current agents construct workflows that include image segmentation, camera calibration, spatial estimation, and dynamics-based computation. In the stacking task, Opus 5.5 wrote colour-segmentation code before moving, then sampled robot motions to fit camera geometry and compute targets. Astra instead approached the blue block from successive camera observations, closed the gripper at step 157, and lifted at 179 before introducing a camera-projection fit at step 204.
Both routes are genuine robot-use behaviour, and both can support progress. The gap appears when these components must be held together across an episode. The abstract summarises the pattern: agents lose task-relevant object states despite reaching commanded poses, fail to correct ineffective actions, recover too late, or mistake unfinished tasks for completion. Reaching a pose is therefore a different measurement from securing an object, and a favourable final frame is a different measurement from satisfying a condition for the full required interval.
Execution traces show where task progress stops
The case studies connect outcomes to recorded execution rather than only to final scores. In Kimi K3's pouring episode, the initial approach knocked the bottle onto its side. Repeated left-arm attempts did not establish a grasp, and the agent switched to the right arm at step 240; the remaining budget went to further approach motions, and the 400-step episode ended without success. Successive arm poses did not restore the object state that pouring required.
Preparation can also consume the whole horizon. All 69 positive-step calls in K3's vegetable-chopping episode concerned surveying, navigation, or clearance around furniture, and no cutting command appeared before the 2,000-step limit. The authors note that this failure diagnoses reaching the workspace, not the unattempted cutting skill. In block sweeping, K3 described the task as complete at step 861 after returning the empty left arm home; its final 69 calls changed only that arm's vertical target, advancing another 139 control steps without resuming sweeping, and native success remained false at the 1,000-step horizon.
Timing produces a separate failure. Astra's inverted-pendulum tracking ended with only 2.45 cm final tip error, but it violated the tracking tolerance when checking began at 0.5 s, and its final stable interval was 0.565 s rather than the required second. The paper's takeaway is direct: motion is not task progress.
Feedback helps only when it changes the next action
Successful episodes show feedback being used to alter execution, not merely to observe it. In K3's drawer-closing episode, reduced end-effector displacement prompted shorter pushing segments, followed by withdrawal to check closure; native success occurred during withdrawal. In charger insertion, Astra tried rotating the idle arm's wrist camera downward to inspect alignment, and the planner rejected the command at step 196. Astra then lowered and brought the idle arm nearer before retrying the same requested pitch, which executed successfully; further adjustments produced native success at step 323. The authors present this as feedback-dependent reconfiguration but do not isolate that change as the cause of final success.
Sustained recovery adds a temporal requirement. Opus 5.5 combined offline linear-quadratic regulator calculations with repeated state feedback in drone payload hover. Payload error decreased, and the final interval met the joint position, velocity, swing, and attitude conditions, with all stability conditions jointly satisfied for 2.15 s against a two-second requirement. The contrast with the pendulum case is the point: recovering for an instant does not satisfy a sustained-stability goal.
Model profiles split by spatial and dynamic demands
The paper reports exploratory task-demand profiles using five overlapping goal labels that cover all tasks: spatial placement/navigation, constrained contact, continuous stabilisation, timed interaction, and multiple subgoals. Astra succeeds on 13 of 45 spatial tasks versus Opus 5.5's 6 of 45, and on 6 of 41 constrained-contact tasks versus 1 of 41. Opus 5.5 leads on continuous balance/tracking (4 of 20 versus 1 of 20) and timed interaction (4 of 7 versus 2 of 7), including quadruped push recovery, cup-based ball catching, and drones.
The drone juggling comparison shows what sustained contact coordination looks like. Both models performed offline dynamics calculations, and both had a median of three executed steps per call. Opus 5.5 sustained 15 valid hits, including 14 height-qualified hits, over 800 steps; Astra recorded three valid hits, one height-qualified, before ball-too-low termination at step 188. The authors attribute the difference to sustained contact coordination rather than to building a model or using shorter segments.
Interaction cost and physical execution cost also diverge. Among eight tasks solved by both models, Astra used fewer tool calls in seven but fewer control steps in only six. These paired costs condition on shared success, so they describe routes through solved tasks rather than overall efficiency. The authors treat the profiles as an exploratory hypothesis about how visual-spatial reasoning, program construction, and feedback control are composed, not as an isolated measurement of any one component.
Simulation results do not settle physical deployment
The reported numbers describe performance on this suite under the observations and control interfaces available in each task. Those interfaces provide different degrees of assistance, and pausing simulation during model inference removes a real-world timing constraint. Physical robots must act despite inference delays, sensor noise, and discrepancies in contact and dynamics, so the paper frames real-time loops and hardware as an open test rather than a settled extension.
Coverage is also finite and uneven, with related tasks inherited from shared source projects. Results therefore characterise this suite rather than a representative distribution of all robot use, and public task descriptions, code, or demonstrations may have appeared in model training data. The task-demand comparisons are descriptive: group membership comes from task requirements, not from outcomes, and the differences do not isolate component abilities or identify training-data causes.
Preserve state, revise actions, and hold conditions
The paper's most useful priority follows from the failure traces rather than from aggregate scores. Reliable robot use requires three concrete things: preserve the object and subgoal state across motions, turn failure feedback into a changed task-directed action, and maintain the required conditions for the full evaluation interval. The pouring, chopping, sweeping, and pendulum episodes each violate a different one of these requirements, which is why local pose accuracy alone is insufficient evidence of competence.
The model comparison sharpens where to look next. Astra's advantage concentrates on spatial and constrained-contact goals, while Opus 5.5's concentrates on continuous-balance and timed-interaction goals, and the drone juggling case shows that the decisive difference was sustaining contact over hundreds of steps rather than constructing a dynamics model. For readers comparing agents, the practical split is therefore task demand: an agent that estimates space well may still fail a goal that requires holding a condition for a required duration, and tool-call counts will not reveal that difference.
Sources used in this answer
RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments
Across 84 simulated robot tasks and 420 scored episodes, five multimodal agents built sophisticated perception and control workflows but failed to compose them reliably, losing object state, not revising ineffective actions, recovering too late, or mistaking unfinished tasks for completion.