Skip to content

Back to the catalog

deepmind.googlearticle

Piloting the world's first double-blind AI evaluations

DeepMind describes piloting a "double-blind" evaluation of a proprietary frontier model (Gemini Flash Lite), using Google Cloud's Confidential Space to cryptographically prevent either party from seeing the other's assets: the evaluator's benchmark prompts stay hidden from Google, and Google's model weights stay hidden from the evaluator. The pilot was run with the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons to address benchmark contamination, where a model provider having advance visibility into test prompts can artificially inflate reported capability/safety scores. This establishes confidential-computing-based evaluation as a technique for trustworthy third-party AI model assessment, particularly relevant for high-stakes cybersecurity and government evaluations.

TL;DR DeepMind piloted double-blind AI model evaluation using Google Cloud Confidential Space to cryptographically hide evaluator prompts from the model provider and model weights from the evaluator, preventing benchmark contamination.
Open original ↗
Also seen from another angle🧠 AI →

Organisations named here

  • AVERI
  • MLCommons
  • OpenMined
  • Singapore AI Safety Institute

Technologies named here

  • Gemini Flash Lite
  • Confidential Space
  • Google Cloud
  • Gemini

Techniques named here

  • Double-blind AI evaluation
  • Zero-logging protocols and contractual safeguards

Related in AI Security