Project Name
Eliminated Flaky Test Failures in a Spring Boot CI/CD Pipeline
Our client is a mid-market software organisation running a Java 17, Spring Boot 3.x microservices platform across 12 services, using RabbitMQ for asynchronous messaging and a CI/CD pipeline supporting multiple deployments per day across a 4-person engineering team. As the test suite grew to approximately 1,800 tests, intermittent CI failures became a persistent operational drag, not from genuine regressions but from structural flakiness across 23 tests that passed reliably on local machines and failed unpredictably in CI.
With 45% of builds failing on first attempt and 82% of those failures caused by flaky tests rather than real bugs, the team was losing 16 developer hours per week to retries, and had begun treating red builds as probably flaky rather than signals worth investigating. The organisation undertook a structured six-week initiative to identify, classify, and eliminate every source of flakiness, and to build the pipeline guardrails that would prevent re-accumulation.
The CI test suite exposed flaky failures caused by timing, shared state, parallel execution, environment differences, and inconsistent test isolation.
- Shared Mutable State and Test Order Dependency: Seven tests passed in isolation but failed after specific tests due to execution-order dependencies and leftover data that local module-level runs never exposed.
- Async and Timing Issues: Six tests relied on hardcoded Thread.sleep durations that worked on faster local machines but failed consistently on slower CI agents.
- Spring Context Pollution: Four tests failed unpredictably due to inconsistent @MockBean/@SpyBean usage, unnecessary @DirtiesContext, and static bean or cache state leaking between tests.
- Fixed Port Collisions in Parallel Runs: Three tests used fixed ports that collided during parallel CI execution, causing Address already in use failures absent from local runs.
- Time and Timezone Assumptions: Two tests relied on timezone or clock assumptions that failed near midnight UTC or when running on CI agents configured for UTC.
- Unseeded Random Test Data: One test generated random data without a fixed seed, causing rare but reproducible constraint violations that appeared as intermittent failures.
- Why Local Always Passed: Faster local machines masked timing issues, while module-level runs, warm databases, and single-threaded execution prevented order, state, port, and data conflicts from surfacing.
The remediation followed three phases: make flakiness measurable, classify root causes, and fix issues by category rather than test by test.
- Phase 1 - Making Flakiness Visible: The full suite ran 20 times overnight on CI to reproduce failures. CI test history tracked results, while failing tests were tagged and quarantined into a separate non-blocking job. This identified 23 flaky tests from approximately 1,800.
- Fixing Timing-Dependent Tests: All Thread.sleep calls were replaced with Awaitility polling, validating actual outcomes instead of relying on fixed delays. Clock injection and Clock.fixed() were also introduced for deterministic time-based tests.
- Eliminating Shared State and Order Dependency: Random test ordering was enabled locally and in CI to expose ordering issues early. Static state was reset between tests, unique test data was generated with UUIDs, and databases were cleaned using @Sql or transaction rollback.
- Stabilising the Spring Context: @MockBean and @SpyBean usage was audited to reduce unnecessary context variations. Shared test configurations were consolidated, while unnecessary @DirtiesContext usage was removed or scoped to AFTER_CLASS.
- Eliminating Environment Differences: H2 was replaced with Testcontainers using PostgreSQL and RabbitMQ to match production. Fixed ports were replaced with random ports, CI timezone was pinned to UTC, date assertions were made explicit, and random data generators were seeded.
- Hardening the Pipeline: Unit tests run in 3 minutes on every push, while integration tests run in parallel with isolated containers. A nightly flake detector runs the suite five times with random ordering and automatically flags failures. Retries are not used to mask flaky tests; quarantined tests must be fixed or removed within two days.
Technology Stack
| Category | Technology |
|---|---|
| Language | Java 17 |
| Framework | Spring Boot 3.x |
| Testing | JUnit 5 and Mockito |
| Testing | Awaitility |
| Testing | Testcontainers – PostgreSQL and RabbitMQ |
| Messaging | RabbitMQ with Spring AMQP |
| Pipeline | CI/CD — Jenkins or GitLab |
The CI pipeline moved from unreliable builds and recurring retries to 98% first-attempt success, zero flaky tests in the main pipeline, and greater developer confidence in build results.
- First-Attempt Build Success Improved From 55% to 98%: Root-cause classification and category-level fixes eliminated recurring failures and reduced the need for build retries.
- Time-to-Deploy Cut From 45 to 22 Minutes: Parallelised testing and the removal of retry cycles more than halved average push-to-deployment time.
- Developer Trust in Red Builds Restored: Red builds are now treated as genuine regression signals rather than failures likely to disappear after a retry, improving confidence across the development team.
- Flakiness Prevention Automated Permanently: Nightly random-order testing and the two-day fix-or-quarantine rule continuously detect new flaky tests before they become recurring pipeline issues.
Flaky tests are rarely just isolated test failures; they are often symptoms of deeper issues in test design, environment consistency, and pipeline configuration. Ksolves, an AI-first DevOps consulting services company, addressed these root causes systematically instead of masking them with automatic retries. By replacing timing assumptions, eliminating shared state, standardising test environments, and strengthening CI practices, the testing pipeline was transformed into a more reliable and maintainable foundation for continuous delivery.
Is Your CI/CD Pipeline Producing Red Builds Your Team Has Learned to Retry Rather Than Investigate?
