Deep dive handbook

Testing

Flakiness

Intro

What causes flakiness?

Flakiness often happens when tests rely on conditions that can change or behave unpredictably.

Common causes include:

• Timing issues
  Tests that check for elements before they’re fully loaded or actions before they’re complete.
• Race conditions
  Situations where multiple processes or requests interfere with each other in unpredictable ways.
• Unstable environments
  External factors like network instability, slow servers, or shared test infrastructure.
• External dependencies
  Services, APIs, or databases that are outside your control and may respond inconsistently.
• Poor test design
  Tests that are too fragile, rely on hard-coded assumptions, or depend on a specific test order.

These factors can cause tests to fail randomly even when the application itself works correctly, making it harder to trust test results.

How can flakiness be avoided?

Reducing flakiness requires careful test design, stable environments, and good practices.

Key strategies include:

• Use proper waits
  Ensure tests wait for conditions like element visibility or network responses instead of relying on fixed delays.
• Control the environment
  Run tests in stable, predictable environments to minimize variability.
• Isolate tests
  Design tests so they don’t depend on state left by other tests. Each should set up and tear down its own data and environment.
• Review and fix flaky tests
  Track flaky tests and regularly review their causes. Rewrite fragile tests as needed.
• Mock external dependencies
  Mock APIs or services when possible to avoid external instability.
• Be careful with parallelization
  Avoid running tests in parallel when they modify shared state or resources.
• Use retries as a last resort
  Retries can hide real problems and should only be used temporarily or when root causes can’t be eliminated.

By following these practices, teams can improve reliability and trust test results.