Original source: TechTarget / LinkedIn Engineering Blog (January 2017)
The 3×3 Release Challenge
LinkedIn operates on what it calls a 3×3 mobile release schedule: three releases per day, with a goal of never allowing code to sit uncommitted for more than three hours. At the scale LinkedIn operates, that cadence demands that every stage of the pipeline — including automated testing — runs fast, reliably, and without human bottlenecks.
For most engineering teams, continuous integration pipelines already feel stretched when pushing one release per day. Tripling that frequency requires rethinking not just deployment, but every upstream gate, particularly automated UI testing on mobile.
Where iOS UI Testing Became the Bottleneck
Automated iOS UI testing is notoriously slow. Running a full UI test suite sequentially on a single simulator can take far longer than the three-hour window LinkedIn had set for its release cycle. Off-the-shelf solutions available at the time could not adequately parallelize iOS simulator runs to fit within that window, and LinkedIn’s engineering team found themselves at a wall: without faster test execution, the 3×3 schedule simply was not achievable.
The options were limited. Compromise the release cadence, accept reduced test coverage, or build something new.
The Solution: Bluepill — An Open-Source Parallel iOS Test Runner
LinkedIn’s engineering team chose to build their own tool. Named Bluepill — a nod to The Matrix — it is an open-source iOS test runner designed to execute tests across multiple simulators in parallel. Key capabilities include:
- Parallel simulator execution: Bluepill spawns multiple iOS simulators simultaneously, distributing the test suite across them to cut overall wall-clock execution time.
- Intelligent test packing: Tests are grouped into balanced batches so that simulators finish at roughly the same time, avoiding idle resources waiting on a slow final batch.
- JUnit report output: After each run, Bluepill produces a standard JUnit XML report, making it straightforward to integrate with existing CI dashboards and reporting tools.
LinkedIn open-sourced Bluepill in January 2017. Full details are available on the LinkedIn Engineering Blog.
How Long Did It Take to Build?
LinkedIn engineer Oscar Bonilla described the development timeline directly:
“The first commit happened on Aug. 12 and we had it working on the LinkedIn Continuous Integration system by Nov. 1. From there we continued to polish the project and address a few bugs. So in total it took about 10 weeks to build Bluepill and another 10 to refine it.”
— Oscar Bonilla, LinkedIn Engineering
That is roughly five months from initial commit to a production-hardened open-source release — a reasonable timeline for a tool that reportedly saved LinkedIn thousands of developer hours.
Lessons for DevOps and Mobile Engineering Teams
The Bluepill story illustrates several principles relevant to any team dealing with CI bottlenecks at scale:
- Identify the constraint precisely. LinkedIn did not try to speed up their entire pipeline at once. They identified iOS UI testing as the specific gate blocking their release cadence and focused engineering effort there.
- Evaluate build-vs-buy honestly. Off-the-shelf tools failed to meet the requirement. Rather than forcing an inadequate tool into production, the team invested roughly ten weeks to build a fit-for-purpose solution.
- Parallelisation is often the right lever for test speed. Distributing a test suite across multiple simulators or machines is frequently more practical than trying to make individual tests faster.
- Open-sourcing internal tooling creates compounding value. By releasing Bluepill publicly, LinkedIn benefited from external contributions and bug reports while giving the broader iOS community a tool to address a common pain point.
- Standard output formats matter. JUnit XML is a lingua franca for CI test results. Tools that produce it integrate immediately with Jenkins, CircleCI, GitHub Actions, and most other CI systems without custom parsing.
Further Reading
For the full technical details of Bluepill’s architecture and configuration options, see the original announcement on the LinkedIn Engineering Blog.