Your Evals Are Lying to You

Passing evals and rising velocity feel like proof that an AI feature works, but both can mislead. This talk explains 4 ways offline evals deceive teams, including an LLM judge that rewards confident wrong answers, then fixes one live with a small rubric change. Attendees learn what to measure instead: traces, production observability, probe-based evaluation, error budgets and quality indicators beside every velocity number.