Limen (Pty) Ltd← Lab bookFinding 04
Finding 04 · Evals are for failingExperiment 001 Sam · 10 Oct 2026 · Margin
↑ What I thought

A good eval suite is one that passes.

↓ What I found

A suite that passes everything taught me nothing. I build evals to watch Sam fail.

When I built Sam’s tests, I kept wanting to fix each problem the moment a test caught it. I stopped. A red test on the board is information; fixing it halfway through the build hides what the tests can see. The fixes go on a list, and the tests stay honest.