A Brain Too Big to Carry — On-Device vs Datacenter Inference

What it claims

SemiAnalysis argues that the planning layer of a generalist robot should move to shared datacenter GPUs, while action and servo loops stay onboard. The split is set by loop frequency. Loops at about 100 Hz have a 10 ms budget, and a typical wireless round trip of 10 to 50 ms consumes it. A planner below about 20 Hz (200 ms per decision at 5 Hz) can absorb the round trip. The authors name jitter, not mean latency, as the real blocker. Buffering to the worst case removes jitter but charges every action the worst-case delay. Jetson Thor offers about 1/10th the FLOPs and 1/30th the memory bandwidth of a GB200.

How it works

The serving cost comes from a timing-only reconstruction of NVIDIA's RoboTTT, which has no public code or weights. The authors start from real GR00T N1.7 checkpoints and insert 16 test-time-training modules into the DiT action head across its 32 blocks. Eight follow cross-attention blocks and eight follow self-attention blocks. They add 16 register tokens, which gives a 57-token sequence (16 registers, 1 state, 40 actions). Each block takes one inner gradient step per denoising pass. Each robot carries 151 MB of fast-weight state that is gathered before every batched call and scattered back after it. A shared-weight VLA does not pay that per-robot cost. Under a 500 ms action-chunk deadline at p99, one B300 serves 12 robots and one RTX PRO 6000 serves 4.

Boston Dynamics is the main example of an offload deployment:

flowchart LR
  MES[Work order from MES] --> S2
  subgraph DC[Off-robot: Google TPUs via DeepMind]
    S2[System 2 planner]
  end
  S2 -- "wireless link (Orbit)" --> S1
  subgraph R[On-robot: Jetson Thor]
    S1[System 1 VLA] --> M[Motor actions]
  end
  S1 -- "camera frames, progress" --> S2

System 2 is pictured as Gemini Robotics ER, built on Gemini 3.5 Flash. SemiAnalysis estimates it at hundreds of billions of parameters, possibly 1T. It turns abstract targets such as "inventory bin 5" into cues the VLA can act on, for example a green dot marked in the camera view. It also supervises execution and is queried every few to ten seconds, sometimes once or twice a second.

Evidence

The TCO model covers a fleet of 96 robots with owner/operator economics. Shared robot parts are excluded.

• B300 offload: one server plus 96 wireless boards, about $554K capex, $18.63/hr.
• RTX PRO 6000 offload: three 8-GPU servers, $436K capex, $15.61/hr.
• Onboard Jetson Thor: 96 Thors at $3,500 each, about $394K capex, $14.97/hr.

The model assumes a four-year life for Thor and the same WACC for all three options.

Before utilization, the cost per FP4 dense PFLOP is $0.15/hr for B300, $0.16/hr for Thor and $0.39/hr for RTX PRO 6000. With 90% server utilization and 40% robot utilization, B300 costs $0.17/hr per PFLOP against $0.37/hr for Thor, or about 46% of the onboard cost. At home utilization the figure falls to about 12%. Offload wins for industrial fleets at 5 or more robots per B300.

The 40% figure comes from Figure's BMW deployment: about 1,250 hours over about 11 months of weekday shifts, at about 10 hours a day. One home-robot company reported 1 to 2 hours of use a day (4 to 8%).

Supply arguments:

• Wafer crossover is about 7 robots per GPU and DRAM crossover is about 5.
• One million 400 mm² Jetson-class chips in 2030 would need only about 10,000 wafers.
• Thor uses 128 GB of LPDDR5X and runs on TSMC N4. Orin used Samsung SF8.
• Blackwell gross margins are in the mid-to-high 70s percent, against the mid-60s for Jetson.

Power rules out carrying datacenter GPUs. A datacenter GPU draws 1.2 to 1.4 kW. A humanoid carries about 2 kWh of battery and uses a 40 to 130 W Thor.

Sunday's ACT-2 folded laundry with 99.1% zero-shot success across nine garment types. Without pretraining the model scored 96% in the lab and 14% in unseen homes. With full pretraining it scored 100% in both.

How to use it

Deployment patterns:

• Agility: runs inference on a Jetson-class chip onboard. Only orchestration runs in Agility Arc, which works because each robot holds 30 to 60 seconds of queued work. Safety stays onboard because losing the link forces a blind fail-safe.
• Sunday: planned cloud inference, then moved fully onboard two days into home testing of ACT-2. Home WiFi jitter was the cause, and Starlink and 5G also dropped out.
• Verne and Weave: run few-billion-parameter policies locally.

The networking prescriptions:

• Robot-aware uplink scheduling with a reserved slot per robot.
• Location-aware beamforming.
• One timing authority per site, with sub-millisecond sync.
• Multi-link operation across WiFi and 5G.
• 6 GHz spectrum.
• Access point handoffs that finish within the observation cadence. Typical routers go silent for 100 ms to several seconds during a handoff.
• 4x4 uplink MIMO on the robot.
• An RF survey along task paths.
• Batching on the GPU side aligned to the shared clock, with late frames moved to the next batch.

Limits

The RoboTTT benchmark measures serving cost only. Its untrained fast weights produce meaningless actions, so the benchmark says nothing about task success. The utilization inputs rest on single anecdotes. No company in the article is at mass production, so TCO does not yet drive real deployment choices. The System 2 parameter count is SemiAnalysis's own estimate. The "shrink the load" section says that image sensors dominate uplink but gives no compression method.