Google DeepMindSat, Aug 1, 2026

Piloting the world's first double-blind AI evaluations

William Isaac, Sol Messing and Kristian Lum Building trust in proprietary model benchmarks using cryptographically secure environments Imagine a student is set to take a high-stakes exam. If they accidentally peek at the test questions in advance, achieving a perfect score is influenced by this knowledge, making it a meaningless accomplishment. To truly measure what they know, they must have no visibility of the test questions until it's time to take the exam. That is the exact challenge the industry faces when evaluating advanced AI models. If a model has already seen the test questions - a problem known as benchmark contamination - the results can only be trusted to an extent. Today, we’re introducing the world’s first double-blind evaluation of a proprietary, frontier class AI model, which keeps external evaluations confined to a cryptographic “box” where they can’t be used by models later to optimize performance ahead of testing. We're partnering with the Singapore AI Safety Institute, OpenMined, AVERI, and MLCommons, to test a Gemini Flash Lite model against confidential benchmarks in a privacy-preserving environment, increasing evaluation integrity. At Google, we assess our AI systems using a broad spectrum of evaluations throughout model development and deployment, but we don’t rely on internal testing alone. To identify potential blindspots, we work with a diverse group of external partners, including specialized research labs, civil society and national AI Safety and Security Institutes (AISIs), using their unique expertise to stress-test our models. As AI models become more capable, ensuring the model has not seen the test questions or prompts in advance is critical, as this can skew the results. Policymakers, researchers, and enterprises need to trust that AI benchmarks accurately reflect a model's true capabilities and safety, but if models are able to “peek” at the evaluation questions in advance, it can artificially inflate scores and undermine this trust. Although zero-logging protocols and rigorous contractual safeguards have long kept external test prompts confidential, incorporating technical and cryptographic safeguards marks a major step forward in secure model evaluation.

What shipped & dropped across AI labs, today.

·last sync 53m ago

Yesterday

Wed · Aug 26 · 7 posts

GitHub1d ago

Enterprise-managed settings now support autoUpdate for plugin marketplaces

You can now opt individual plugin marketplaces into automatic updates by setting autoUpdate: true on an extraKnownMarketplaces entry in enterprise managed settings. Supported clients automatically check the marketplace and update…

Businessgithub.blog/changelog/2026-08-26-enterprise-managed-settings-now-support-autoupdate-for-plugin-marketplaces

xAI1d ago

Grok 4.6 on Microsoft Foundry

Grok 4.6 is now available via Microsoft Foundry.

Modelsx.ai/news/grok-4-6-microsoft-foundry

GitHub1d ago

Global model policy generally available

In July, we announced a default model policy for generally available GitHub Copilot models on Copilot Business and Copilot Enterprise plans. Starting today, we’re gradually rolling out enforcement of the…

Toolsgithub.blog/changelog/2026-08-26-global-model-policy-generally-available

Claude Blog1d ago

Claude gets its own browser in Cowork

Claude now has its own browser inside the Cowork desktop app. It opens sites, reads pages, and works in them while you keep working. No extension, and your own tabs and logins stay out of it.

Modelsclaude.com/blog/cowork-built-in-browser

Claude Blog1d ago

Claude in Chrome is generally available

Claude in Chrome is now available on every paid plan and can work through browser tasks without approving each step, with a safety check on every action. Here's how we tested it against prompt injection.

Practicesclaude.com/blog/claude-in-chrome-generally-available

Google AI for Developers1d ago

Enterprise-Grade Precision for Long-Context Multimodal Embedding Inference on Cloud TPU

Google Cloud has natively integrated TPU support into the vLLM serving engine, allowing developers to elastically scale high-demand embedding pipelines using Google Kubernetes Engine (GKE). To handle massive 15K+ token contexts for models like Qwen3-Embedding-8B, the engineering team implemented TPU-specific optimizations such as hardware-safe tensor alignment, JAX/XLA compilation pre-warming, and a hybrid StepPool architecture for chunked prefill management. These enhancements achieve near-perfect numerical parity with reference GPU baselines, and developers can immediately leverage the open-sourced setup recipes on the AI-Hypercomputer GitHub to build their own high-throughput semantic retrieval applications.

Modelsdevelopers.googleblog.com/enterprise-grade-precision-for-long-context-multimodal-embedding-inference-on-cloud-tpu/

Claude Blog1d ago

How Warp builds self-improving agents on Claude

How Warp devised a simple development pattern that anyone can use to create self-improving agents.

Modelsclaude.com/blog/how-warp-builds-self-improving-agents-on-claude

Tuesday

Tue · Aug 25 · 4 posts

GitHub2d ago

GitHub Copilot app Customize tab is generally available

Customize tab in the GitHub Copilot app brings MCP servers, plugins, skills, and canvases together in one place.

Toolsgithub.blog/changelog/2026-08-25-github-copilot-app-customize-tab-is-generally-available

Claude Blog2d ago

Claude's memory works everywhere, and you decide what's in it

Wherever you work with Claude from Anthropic, it starts from what it already knows about you.

Modelsclaude.com/blog/claudes-memory-works-everywhere-and-you-decide-whats-in-it

Anthropic2d ago

Funding better evaluations of AI’s impact on wellbeing

Researchanthropic.com/news/wellbeing-research-grants

Claude Blog2d ago

Bain & Company joins the Claude Partner Network as a Global Premier partner

Anthropic and Bain & Company are partnering to help enterprises deploy AI, building on Bain's rollout of Claude to its 19,000 employees.

Businessclaude.com/blog/bain-company-joins-the-claude-partner-network-as-a-global-premier-partner

Monday

Mon · Aug 24 · 2 posts

Google AI for Developers3d ago

How to Evaluate Live & Voice Agents in ADK

Moving live voice agents from demo to production requires rigorous, automated testing to handle the unpredictability of real multi-turn conversations. ADK now provides native live evaluation, allowing developers to test graph-based agent workflows against LLM-driven simulated users that generate actual audio via Gemini TTS. By defining evaluation scenarios and natural-language rubrics, you can automatically score audio responses and tool executions, inspect the resulting transcripts in ADK Web, or run the CLI directly in your CI/CD pipeline.

Toolsdevelopers.googleblog.com/how-to-evaluate-live-voice-agents-in-adk/

Claude Blog3d ago

How an Anthropic field marketer uses Claude Code to send weekly personalized updates to every sales rep

Adam Ward, on Anthropic’s marketing team, shares how he uses Claude to turn one weekly sales report into a personalized Monday briefing for every account executive he supports.

Modelsclaude.com/blog/how-an-anthropic-field-marketer-uses-claude-code-to-send-weekly-personalized-updates-to-every-sales-rep

Friday

Fri · Aug 21 · 6 posts

xAI6d ago

Grok 4.6 on Gemini Enterprise Agent Platform

Grok 4.6 is now available via Gemini Enterprise Agent Platform.

Modelsx.ai/news/grok-4-6-vertex-ai

xAI6d ago

Grok Bot is now included with more plans

Grok Bot is now available for SuperGrok, Cursor Pro, and all Cursor Teams plans.

Modelsx.ai/news/grok-bot-more-plans

Claude Blog6d ago

Bringing the cybersecurity capabilities of Claude Mythos 5 to more defenders

We’re sharing an update on our efforts to help more teams leverage frontier capabilities for cyber defense.

Businessclaude.com/blog/bringing-claude-mythos-5-to-more-defenders

GitHub6d ago

Shared agentic work with GitHub Copilot in Microsoft Teams

Turn a Microsoft Teams discussion into a collaborative agent session everyone can see and help direct. Mention @GitHub in a channel, thread, or direct message to start a GitHub Copilot…

Toolsgithub.blog/changelog/2026-08-21-shared-agentic-work-with-github-copilot-in-microsoft-teams

GitHub6d ago

The new GitHub Copilot experience in Slack

The GitHub integration in Slack now brings the agentic capabilities of GitHub Copilot CLI and the GitHub Copilot app into Slack in public preview. You can work with @GitHub to…

Toolsgithub.blog/changelog/2026-08-21-the-new-github-copilot-experience-in-slack

Claude Blog6d ago

The AI-Native SDLC playbook

Anthropic's stage-by-stage playbook for the AI-native SDLC: how teams plan, design, build, test, deploy, and maintain software with Claude.

Practicesclaude.com/blog/the-ai-native-sdlc-playbook

August 20

Thu · Aug 20 · 1 post

Claude Blog7d ago

Build production agents with computer use, the Skills API, and the Files API

Computer use, the Skills API, and the Files API are generally available on the Claude Platform today. Computer use also adds a new browser use tool for agents that work in web applications. Together they let you build agents that operate software, apply your team's expertise, and return finished files.

Toolsclaude.com/blog/computer-use-skills-api-files-api

trigger