Zillow's SVP of Engineering Toby Roberts revealed at VB Transform 2026 that measuring AI return on investment requires baseline metrics established before deployment, not after. This insight emerged while discussing how Zillow and Glean built AI systems to maintain context across fragmented customer journeys in real estate transactions.

The challenge Zillow faces is architectural. Customers interact with the company through multiple touchpoints over months or years: phone screens, loan officers, real estate agents. A single chatbot cannot maintain continuity across these interactions. Roberts and Glean co-founder Arvind Jain explained that building AI systems to thread context through an entire customer journey proved harder than processing raw data.

Zillow's scale amplifies the complexity. The company's products touch roughly 80 percent of U.S. real estate transactions annually. Roberts noted that Zillow deployed AI systems years before ChatGPT became mainstream, giving the company institutional knowledge about what actually works versus what generates hype.

The ROI measurement framework Roberts advocated is counterintuitive to how many companies approach AI adoption. Organizations typically measure results post-deployment and compare them to the previous state. Roberts argues this approach fails because it lacks a true baseline. Establishing metrics before building AI systems captures the real starting point, making improvement measurable against actual performance rather than assumptions.

This distinction matters operationally. Without pre-build measurement, companies cannot distinguish between AI's actual impact and other variables affecting outcomes. They also cannot quantify what problems the AI actually solves or whether it solved the right problems.

Zillow's emphasis on context over raw computational power reflects a broader market shift. The housing transaction involves complex human interactions, regulatory requirements, and decision-making that raw processing power alone cannot handle. AI must understand and preserve the customer's position within a multi-month journey, not just respond to individual queries.

Roberts'