deepmind.googlearticlePiloting the world's first double-blind AI evaluations
Google DeepMind describes a pilot of double-blind AI model evaluation, testing a Gemini Flash Lite model against confidential benchmarks from the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons. Using Google Cloud's Confidential Space (confidential computing), neither the evaluator's test prompts nor the model's weights are exposed to the other party, cryptographically preventing benchmark contamination while preserving data sovereignty for both sides.
TL;DR DeepMind piloted cryptographically-verified double-blind evaluations of Gemini Flash Lite to stop benchmark contamination, using Confidential Space so evaluators and Google never see each other's prompts or model weights.
Also seen from another angle🛡️ Security →
Organisations named here
- Google DeepMind
- Google Cloud
- Singapore AI Safety Institute
- OpenMined
- AVERI
- MLCommons
Technologies named here
- Confidential Space
- Gemini Flash Lite
- Gemini
Techniques named here
- Double-blind evaluations
- Zero-logging protocols and contractual safeguards
Related in evals
