● Covered by 1 source · 1 reportMedium impact1 neutral
Google DeepMind introduces double-blind evaluation for AI models using confidential computing
Key points
- Google DeepMind developed a double-blind AI model evaluation.
- Neither Google nor evaluators see the other's private data.
- Confidential computing protects model weights and test questions.
- Pilot tested Gemini 2.5 Flash Lite against private benchmarks.
New Double-Blind Evaluation Method
Google DeepMind has introduced what it calls the first double-blind evaluation for a proprietary frontier-class AI model. This new setup ensures that Gemini's model weights remain hidden from evaluators, while Google does not have access to the test questions. This method was demonstrated in a pilot test of Gemini 2.5 Flash Lite.
Addressing Benchmark Leakage
The initiative directly addresses the problem of benchmark leakage, where models may be inadvertently tested on data they have already seen, leading to inflated scores. Previous research indicates that benchmark leakage has affected approximately half of tested models, particularly impacting larger models and raising questions about the true source of post-training performance gains. The new method aims to prevent this contamination by keeping both model and test data private.
Challenges with Closed Models
Evaluating closed models presents a challenge because evaluators typically use the provider's API, exposing test data to the company. Conversely, independent evaluation would require the provider to share model weights. Google's double-blind setup offers an alternative, allowing both parties to maintain the confidentiality of their respective assets during the evaluation process.
How the Enclave Works
The pilot utilizes Google Cloud Confidential Space, an NVIDIA H100 Confidential GPU, and Intel TDX host memory encryption. Google DeepMind provides Gemini's weights and inference code, while the evaluator supplies benchmark prompts and evaluation code. These are transmitted over encrypted connections into a secure enclave. Within this enclave, the evaluation occurs without either party accessing the other's protected assets. Model weights are stored in hardware-encrypted GPU memory, and evaluation prompts are kept in encrypted host memory. After the test, the evaluator receives permitted results, and the temporary environment is destroyed. Remote attestation verifies the enclave's software before data transmission.
✨ This summary was generated by AI from the outlets' reporting listed below. It is not independently verified and may contain errors — check the original sources. How BrevFeed works →
The daily brief
One email each morning: the day's tech stories, clustered across outlets and summarized. No account needed.
One email a day. Unsubscribe in one click, any time.
Today's brief
Spend a few minutes, get the whole day. Every topic's top stories in one hands-free rundown — listen, watch, or read the transcript.
▶ Play today's brief
New every morning, and the back catalogue is archived by date.
Reporting from
Google DeepMind has developed a double-blind evaluation method for AI models, allowing testing without either the model provider or the evaluator seeing the other's proprietary data. This method addresses benchmark leakage concerns by using confidential computing to protect both model weights and test questions during evaluation. The approach aims to provide a more secure and unbiased way to assess AI model performance.