At Waymo, an AI project isn't ready until its evals are — not when the model performs well

Waymo has developed a rigorous evaluation framework that prioritizes testing over raw model performance metrics. The autonomous vehicle company, owned by Alphabet, treats AI deployment as incomplete until comprehensive evaluations confirm safety across real-world driving scenarios. This approach diverges from typical machine learning workflows where strong benchmark numbers often signal project readiness.

Manasi Joshi, Waymo's director of engineering for systems intelligence and machine learning, outlined the company's methodology at VB Transform 2026. Waymo relies on continuous evaluation, carefully curated datasets, human oversight, and clearly defined business outcomes to manage deployment risks. Unlike text generation or back-office automation, self-driving decisions carry immediate physical consequences. A model failure means vehicles fail to navigate unpredictable streets, misread human drivers, or make dangerous split-second judgments in traffic.

Waymo's framework addresses a fundamental tension in AI development. Models that excel on standard benchmarks sometimes fail in deployment. Waymo inverts this priority by treating evaluation as a release gate rather than a post-hoc validation step. The company builds testing into the development cycle from the start, not as an afterthought.

This approach scales beyond autonomous vehicles. Any AI system operating in the physical world or making consequential decisions benefits from Waymo's playbook. Robotics companies, industrial automation firms, and healthcare providers all face similar stakes. Continuous evaluation catches edge cases benchmark tests miss. Human oversight preserves decision accountability. Business outcome alignment ensures models solve actual problems rather than optimizing abstract metrics.

Waymo's methodology reflects maturation in enterprise AI deployment. Early machine learning projects often treated evals as optional polish. Waymo treats them as foundational. This shift acknowledges that model performance and system reliability differ fundamentally. A 95 percent accurate vision model provides no comfort if the remaining 5 percent occurs during a critical intersection decision.

The