LLMs are becoming capable of controlling various robot embodiments out of the box. To examine how they handle physical tasks, we extended our real-world evaluation from the MolmoAct2 paper. Here, we compare GPT-6 Astra, Gemini 3.8 Flash, and Fable 5.1 on two tasks: loading a cold sandwich into a microwave and closing the door, and attaching a 5 mm tip to a pipette. Both require the robot to translate what it sees into precise movement.

GPT-6 Astra scored highest on both tasks in this evaluation.

Microwave task scores: GPT-6 Astra 80%, Gemini 3.8 Flash 13.3%, Fable 5.1 6.7%.
Figure 1: Microwave task scores.
Pipette task scores: GPT-6 Astra 70%, Gemini 3.8 Flash 30%, Fable 5.1 20%.
Figure 2: Pipette task scores.

We’ve evaluated all the models in a bi-manual YAM BOX setup with an overhead camera and a camera on each wrist. Each model used medium reasoning effort and had up to 10 minutes to complete each task. An evaluator scored each attempt, awarding partial credit for completed steps. We also provided depth images alongside the camera views to help the models judge distances.

The video below shows a successful Astra microwave episode. It picks up the sandwich, moves it into the microwave, pushes the protruding tray farther inside, and closes the door. At the end, it withdraws its hand to check that the door stays shut.

Video 1: Loading the microwave and closing the door.

Attaching a 5 mm tip to a pipette requires precise alignment. In the video below, Astra succeeds after a few failed attempts. Its reasoning notes that the tip has not attached, even though it lowered the pipette into the tip container, and it tries again. In contrast, in the failed attempts we observed across all the VLAs in MolmoAct 2 evals, the robots continued as if the tip had been fitted, carried out the remaining steps, and returned to the rest position without correcting the failure.

Video 2: Astra attaches a pipette tip after retrying.

Astra exhibits behaviors unseen in prior generalist robot models

In one of the rollouts, the sandwich is inside the microwave, but its plastic tray still extends beyond the sill. Astra identifies the remaining obstruction and proposes using the empty gripper as a small pusher against the tray's end. This kind of behavior is usually not seen in VLAs from our previous eval rollouts.

Astra reasoning about narrowing the gripper to push the sandwich tray.
Figure 3: Astra pushing the tray using its gripper as a 'small pusher'.

In a separate microwave task rollout, Astra realizes that the door is not fully closed and proposes a final commanded push of 1.5 centimeters, followed by a check that the door stays closed. This is particularly interesting as it suggests Astra understands that closing the door requires enough force to overcome the latch's resistance.

Astra reasoning about a final 1.5-centimeter push and checking the microwave door.
Figure 4: Astra plans a final 'commanded' push to latch the door.

Gemini and Fable scored lower on these tasks. Our working hypothesis is that weaker depth perception contributed to the gap. Both tasks require accurate judgments about relative position: how far a tray extends beyond a sill, or how a pipette lines up with a small tip. Errors in those judgments can make an otherwise sensible action fail.

Reasoning its way through to better visibility

In a previous X post, we shared how gpt-6-Astra is using its free arm’s camera to get a better view on the scene. We haven't seen this behavior in the VLAs trained and fine-tuned on teleoperated data that we've evaluated. In teleop data, operators typically move the arms to perform the task. Demonstrations rarely include deliberately repositioning a free arm to give the robot a better view.

Astra uses its free arm camera to get a better view of the scene.
Figure 5: Astra uses its free arm’s camera for a better view.

We’re continuing our work in this area and will share updates soon.