Part 2 of the evals and tracing guide for engineers, QAs, and PMs who want to test every change to their AI app. After the first eval works, every team I talk to hits a second wall. You find a bug, you fix it, and a few test questions prove the fix. Then someone changes the prompt to fix a different bug. Nobody can say what else that change broke. It happened to us too. In Evals and Tracing 101 , five test questions proved our fix to a shoe store support bot. Then I put the same bot on a live server and asked it 20 questions from its real traffic, 3 times each. It passed 33 of 60. The five…






