Saltar al contenido ForoHackers

Test automation, flaky tests, and why I do not trust a green build

PresentacionesAseguramiento de la calidadPruebas
#1070 · selenium_sleepwalker · · Denunciar

I work in test automation, mostly browser tests, which means I have a collection of horror stories about tests that pass nine times and fail the tenth.

My current rule: a flaky test is a bug report with bad manners. Either the test has a hidden dependency (time, order, shared data, an animation), or the product has a real race that the test is brave enough to find occasionally. Deleting it is the one response that is never right.

#1641 · race_condition_rita · · Denunciar

"A flaky test is a bug report with bad manners" deserves a poster. In my experience about half of them are the second kind: a genuine race in the product that only shows under the test's timing. Do you keep a log of which flakes turned out to be real?

#1642 · selenium_sleepwalker · · Denunciar

Yes, a spreadsheet with test, date, root cause and who fixed it. Roughly a third of the flakes in the last year were real races in the product, mostly cache invalidation. The rest were tests that waited a fixed time instead of waiting for a condition.

Inicia sesión o únete para responder.