Gemini Robotics ER 2: powering robotics with video understanding, task orchestration, and multi-robot collaboration

What it claims

Gemini Robotics ER 2 is an orchestrator, not a controller. It plans and monitors robot tasks and calls your low-level controllers as tools. The page positions it as an upgrade over Gemini Robotics ER 1.6 in four areas: tool orchestration, video-based progress tracking, spatial reasoning (success detection, instrument reading, VQA) and safety behaviour. The headline numbers are 57.4% on progress classification and, on moment-finding, 91.3% accuracy with 0.96s mean absolute distance. The page says the moment-finding result is close to "much larger model categories" at a fraction of the compute and 4x the execution speed.

How it works

• Integration shape. You declare low-level control interfaces as tools. Examples are VLA models, navigation APIs, or any user-defined function; Google Search is also callable natively. You then stream video, audio or text into the model. ER 2 decides which tool to call next and checks the result from the video feed.
• Latency path. The model runs through the Gemini Live API, a bidirectional streaming endpoint meant for latency-sensitive use. It is designed to plan the next step while the current action is still running, which avoids "stop-and-think" pauses.
• Progress classification. Each video frame is placed into one of five progress bins: 0–20%, 20–40%, 40–60%, 60–80% and 80–100%. The robot can use this to retry a failed step without restarting the whole workflow. A random guess on this task scores 20%, which is the right context for the 57.4% figure.
• Moment-finding. The model locates the frame where a key event happens, for example when to stop pouring coffee. This is what triggers moving to the next step.
• Success/failure detection. This now runs on raw video instead of single snapshots, so it can catch failures mid-execution such as spills, slips or misalignment.
• Multi-robot collaboration. Different robots hand off tasks through a "shared semantic understanding." The page gives no mechanism beyond that phrase.

Evidence

• Tool orchestration. ER 2 beats ER 1.6 in three control modes: a real VLA, a simulated VLA, and a human teleoperating the robot. The page gives no numbers.
• Progress classification. 57.4% on the five-bin task. The page says this beats previous-generation and "competing frontier models" but names neither those models nor their scores.
• Moment-finding. 91.3% accuracy and 0.96s mean absolute distance. The comparison to larger models is qualitative only (close performance, 4x speed).
• Instrument reading. Tested on 10 instrument types: digital displays, linear scales, rulers and liquid thermometers, in addition to the dials and sight glasses covered previously.
• Safety. ER 2 beats ER 1.6 and other frontier models on two benchmarks, Safety Instruction Following and Human Proximity. In a demo it stopped a humanoid robot when a person was nearby and resumed once the area was clear. The page also introduces a new benchmark for safe VLA orchestration covering constraint enforcement, environment monitoring, physical feasibility and asking a human for clarification. Details are in a separate safety technical report.

How to use it

• Access. Public through the Gemini API and Google AI Studio. Private preview only on Gemini Enterprise Agent Platform.
• Streaming. Use the Gemini Live API endpoint if you need streaming orchestration.
• Example code. The Boston Dynamics Spot fetch demo, which orchestrates Spot's navigation and manipulator APIs, is on GitHub alongside other examples. The page does not give the repository URL.
• What you supply. ER 2 does not do motor control, so you need your own VLA model or robot API to expose as tools.

Limits

• The page gives no model size, pricing, rate limits or model ID string.
• Competitor models are unnamed throughout. Most comparisons, including all orchestration and safety results, are given without numbers.
• Progress classification is 57.4% on a five-way task, so it is wrong more often than it is right. Do not use it as a sole completion gate without your own validation.
• Multi-robot collaboration is shown only as a demo with Apptronik Apollo 2 and Franka F3 Duo. There is no metric and no protocol description.
• Enterprise deployment is private preview only.