Skip to the content ForoHackers
Log inJoin

Test automation, flaky tests, and why I do not trust a green build

IntroductionsQuality assuranceTesting
#1070 · selenium_sleepwalker · · Report

I work in test automation, mostly browser tests, which means I have a collection of horror stories about tests that pass nine times and fail the tenth.

My current rule: a flaky test is a bug report with bad manners. Either the test has a hidden dependency (time, order, shared data, an animation), or the product has a real race that the test is brave enough to find occasionally. Deleting it is the one response that is never right.

#1641 · race_condition_rita · · Report

"A flaky test is a bug report with bad manners" deserves a poster. In my experience about half of them are the second kind: a genuine race in the product that only shows under the test's timing. Do you keep a log of which flakes turned out to be real?

#1642 · selenium_sleepwalker · · Report

Yes, a spreadsheet with test, date, root cause and who fixed it. Roughly a third of the flakes in the last year were real races in the product, mostly cache invalidation. The rest were tests that waited a fixed time instead of waiting for a condition.

Log in or join to reply.