This article summarizes findings from the “Testing Trends In 2017” survey conducted by Dimension Data and sponsored by Sauce Labs, as reported by InformationWeek. The survey polled 732 software professionals, primarily in North America and Europe.
Agile Adoption Is Near-Universal — But Depth Varies
Enterprise software development has broadly embraced agile methodology. According to the 2017 Testing Trends survey, 94% of respondents said that many or some of their software teams were committed to agile practices. Only 32% reported that all their teams had fully adopted agile, however, highlighting a gap between organizational intent and consistent implementation. Just 6% said agile was either unused or still in a pilot phase — a clear signal that agile has become the dominant development philosophy across the industry.
Continuous Integration: Popular in Theory, Inconsistent in Practice
Despite agile’s dominance, continuous integration (CI) — the practice of frequently merging code changes into a shared repository and validating them against production-like systems — remains unevenly implemented. The survey data paints a mixed picture of integration frequency:
- Daily integration: 35% of respondents could integrate updates daily.
- Weekly integration: 17% integrated on a weekly cadence.
- Less than weekly, but more frequent than legacy cycles: 20% integrated more often than the traditional six-month or annual release windows.
- Hourly integration: This category grew by 10%, indicating a meaningful shift toward higher-frequency deployments.
While the trend is clearly moving toward more frequent integration, a significant portion of teams are still far from the continuous delivery ideal that CI is designed to enable.
Bug Fix Velocity: Improving, But Room Remains
One practical measure of CI maturity is how quickly teams resolve bugs. The survey found incremental improvement between 2016 and 2017:
- Fixed immediately: 23% in both 2016 and 2017 — no change.
- Fixed within a few working days: Improved from 48% to 51%.
- Fixed in more than a few days: Dropped slightly from 5% to 4%.
The survey report noted: “This is clearly an area where more work can be done by testing and development teams, since the benefits of fixing bugs immediately are well established.” Deferred bug remediation is a known risk in CI pipelines — bugs that linger increase integration complexity and can compound downstream quality issues.
Test Writing Practices: A Waterfall Hangover
One of the foundational practices of CI is writing tests alongside code — or ideally, before code (test-driven development). The survey reveals that this practice has not yet become the norm:
- 19% write tests when code is checked into the build system — the CI-aligned approach.
- 54% defer testing to a dedicated quality assurance phase.
- 22% write tests only after development is complete — a pattern more consistent with waterfall development.
- 5% do not write automated tests at all.
This means roughly 81% of teams are not integrating tests at the point of code commit, which limits the feedback value of a CI pipeline. The ideal CI setup catches regressions at commit time, not in a downstream QA phase.
Test Automation: Broad Adoption, Uneven Execution
Test automation adoption is high at the surface level, with 87% of development teams using some form of automated testing. However, the tooling landscape and actual automation depth vary considerably:
- Open source tools (e.g., Selenium, Appium): Used by 70% of respondents.
- Proprietary tools: 12%.
- Manual testing only: 13%.
- Automated test framework in use: 52%.
- Mostly or entirely manual test setup and execution: 42%.
The coexistence of high automation adoption with 42% still relying predominantly on manual processes suggests that many teams are in a transition state — applying automation selectively rather than systematically throughout the pipeline.
Cloud-Based Testing Is Growing Steadily
The use of public cloud infrastructure for testing has shown consistent year-over-year growth:
- 2015: 43%
- 2016: 45%
- 2017: 52%
A 9-percentage-point increase over two years indicates that cloud-based test infrastructure is becoming a mainstream choice, offering on-demand scalability that supports the higher deployment frequencies that CI demands.
DevOps: High Awareness, Low Full Adoption
DevOps has clearly moved beyond buzzword status. The survey found that 88% of respondents had adopted DevOps practices, were in the process of adopting them, or were actively considering adoption. The survey used the following working definition:
“A practice that emphasizes the collaboration and communication of both software developers and other IT professionals while automating the processes of software delivery.”
Despite the high interest, full organizational adoption remains rare:
- Fully adopted across the company: Only 10%.
- A few teams using it: 24%.
- Just started: 25%.
- Still considering: 15%.
- No plans to adopt: 6%.
This pattern — broad organizational awareness but shallow implementation — mirrors the agile adoption curve. The data suggests that DevOps, like agile before it, is spreading team by team rather than transforming entire organizations at once.
Who Responded to the Survey?
The 732 respondents were drawn from web and application testing professionals. The breakdown by company size was roughly even across small (<100 employees, 32%), mid-size (100–1,000 employees, 33%), and enterprise (>1,000 employees, 35%) organizations. Eighty-seven percent were based in North America or Europe. By role:
- Developers: 23%
- Test engineers: 19%
- QA engineers: 17%
- QA team managers: 9%
- DevOps advocates or specialists: 8%
- Architects: 6%
- Development team managers: 6%
Key Takeaways for DevOps and Engineering Teams
The 2017 data tells a consistent story: the industry knows where it needs to go, but closing the gap between aspiration and practice takes time. For teams looking to move the needle on CI maturity, the survey highlights three practical focus areas:
- Shift tests left. Writing tests at commit time rather than in a downstream QA phase is the single most direct way to realize the feedback benefits of a CI pipeline.
- Reduce bug fix latency. Immediate bug remediation prevents accumulation of technical debt that slows future integration cycles.
- Standardize automation tooling. High-level automation adoption masks significant inconsistency in how automation is applied. Teams benefit from aligning on a framework that integrates with their CI/CD pipeline end to end.
The “Testing Trends In 2017” survey was conducted by Dimension Data and sponsored by Sauce Labs, a cloud-based automated testing platform founded in 2008 in San Francisco. Original coverage appeared on InformationWeek.