Putting sign language AI into users’ hands

What it claims

SL2T is a sign-language-to-text translation model from Google DeepMind and Android. It is trained on more than 100,000 hours of data across more than 50 sign languages, and roughly a quarter of that data is ASL. It ships as a consumer feature: ASL-to-English sign dictation in Gboard and Live Transcribe on Pixel 11. The page offers no model, weights, API or paper to build on. What an engineer gets from it is the system design and the deployment problems the team chose to solve.

How it works

Pose extraction runs on the device, and translation runs on a server.

flowchart LR
  A[Camera video] --> B[MediaPipe Holistic<br/>on-device]
  B --> C[Pose landmark coordinates]
  A -.-> X[Video discarded]
  C -->|sent over network| D[SL2T on server]
  D --> E[Text into Gboard /<br/>Live Transcribe]

• Privacy boundary. Only geometric landmark coordinates leave the phone. The raw video "can be discarded immediately." The privacy claim rests on landmarks being less identifying than video. The page does not analyse whether that holds.
• No glosses. The model maps landmark sequences directly to text. The authors argue that glosses lose non-manual markers and spatial constructions and impose a fixed vocabulary. They say skipping glosses lets quality "scale directly with data."
• Joint multilingual training. One model is trained across languages, dialects and signer proficiency levels. The authors report that it outperformed single-language models "in our experiments," but give no numbers.

Evidence

• FLEURS-ASL (sd-test), ASL to English: 70 BLEURT, zero-shot. The page calls this "significantly higher than any previously reported score" but does not name or quantify the prior baselines.
• User testing: testers reportedly found signing faster and more natural than typing English. This is anecdotal, with no metrics.

The page gives no model size, architecture, compute, latency figures or per-language results.

Limits

• Not something you can adopt. There are no weights, API, licence or benchmark code. Access is via Pixel 11 apps only, "more devices coming soon," at no extra cost.
• ASL to English only. Other sign languages are trained on but not shipped. Sign language generation is listed as future work.
• Server-dependent. Translation is not on-device, so it needs connectivity. The page does not address offline use.
• Hard cases the team names. The page lists the following as areas they "worked hard on," without reporting results for any of them:
• streaming latency
• hallucination when the input contains no signing
• fairness for left-handed signers (about 10%)
• one-handed signing while holding the phone
• Limitations documented elsewhere. Capabilities and limitations are detailed in a joint impact report written with the AI Sign Language Advisory Committee (AISLAC), a group of Deaf organisations and experts. That report is not reproduced on this page. Read it before relying on the system for anything beyond dictation.
• Thin evaluation. The only quantitative result is a single benchmark measured with a single learned metric.