What are Flaky Tests?
A flaky test is one that might pass or fail for the same configuration. Their existence confuses developers because in this case, test failure does not always signal the existence of bugs in the code. They are particularly disruptive to tests with a broad scope.
However, flaky tests actually help reveal gaps in the testing pipeline, both on the technical and human aspects.
The Technical Aspect
In Selenium WebDriver tests, flakiness comes from:
Lack of synchronization: Web apps contain multiple layers, The way these layers interact influences web app performance. Because of this, some operations may have varied timings when the website is put through different end-to-end tests.
This can be solved by including Selenium wait commands. If some actions more time to execute, Selenium wait commands halt test execution until the action is complete. But if certain areas of the web app consistently need waits, it could suggest poor performance.
Accidental load testing: As automated test suites grow, the number of lines in test code increases. Software is also put through more tests. Naturally, test suites are reorganized to be executed at the same time, usually through parallel testing to reduce test runtime. However, this can impose large loads on the software, resulting in an unintentional load test.
Certain tests may run perfectly fine in a series but may be flaky when running simultaneously. If the failures are seemingly random, perform some debugging.
The Human Aspect
Here’s what flaky Selenium tests can reveal about testers:
Teamwork and Communication: It can be challenging to get every team member to take an interest in end-to-end tests. Since flaky tests appear to pass sometimes and fail at other times, it’s safe to assume that anyone asking about them has been looking at test results consistently. They are the people more likely to be receptive to collaborative agile practices. If flaky tests appear and are flagged, this means the team is either not receiving information about the tests, or they are not interested.
Test results fatigue: This refers to a condition in which teams have been so saturated with unreliable test results that they start ignoring end-to-end results altogether. A prime reason for test results fatigue is flaky tests. If team members start ignoring flaky tests and all results related to flaky tests, they end up disregarding large portions of the automation testing pipeline. Reaction to flaky tests can help you gauge how much your team is invested in automation.
Flaky tests can be an indicator of the quality of the Selenium automation test setup. Dive deeper, examine flaky tests, and find where they come from. It could be an anomaly with performance, speed, or with the nature of software. The point here is: use flaky tests as a friend.
