This article is based on a piece originally published on computerweekly.com and summarised here for a DevOps engineering audience.
The DevOps Tooling Challenge
Organisations are moving away from traditional waterfall and cascade software development — where code is produced over defined time periods and handed off between teams — toward more agile, iterative approaches. The demand for rapid delivery via continuous integration (CI), continuous development, and continuous delivery (CD) has pushed many businesses to adopt DevOps practices.
DevOps aims to bring together development, testing, and operations teams to streamline how code moves through the software delivery pipeline. Done well, it delivers frequent, reliable releases with minimal disruption. Done poorly — especially when tooling is left ungoverned — it creates fragmented environments that are difficult to audit, support, or scale.
Developer-Led Tool Adoption: Bottom-Up Growth
One of the defining characteristics of DevOps tooling adoption is that it tends to grow from the bottom up rather than being driven by top-down mandates. Because many DevOps tools are open source and freely downloadable, individual developers and system administrators often adopt them without any formal procurement process or organisational approval.
Developers look for tools that integrate easily with the environments they already use — commercial IDEs such as IBM Rational Software Architect or Microsoft Visual Studio, open source IDEs such as Eclipse or Anjuta, and project management tools such as Atlassian Jira or CA PPM.
This organic adoption has driven widespread use of tools such as:
- Jenkins — An open source automation server that manages software versioning and supports continuous delivery pipelines. It integrates with version control systems including Git, Subversion, Perforce, CVS, and Accurev via its extensive plugin ecosystem.
- Chef — A configuration management tool originally focused on the operations side of DevOps, used to define infrastructure as code and automate provisioning.
- Puppet — Another configuration management system that enforces desired infrastructure state and automates repetitive operational tasks.
These tools began as point solutions targeting specific roles or workflow stages. Over time — Puppet having been released around 2005, Chef around 2009, and Jenkins in 2011 as a continuation of the Hudson project started at Sun in 2004 — their capabilities have expanded significantly and increasingly overlap.
An Expanding Ecosystem of Options
Jenkins, Chef, and Puppet are no longer the only significant players. The broader DevOps tooling ecosystem now includes a wide range of open source and commercially supported options:
- Ansible and Salt — Agentless or lightweight configuration management and orchestration tools that compete with Chef and Puppet in many environments.
- Docker — Container runtime and image management tooling that has become foundational to modern application packaging and delivery.
- Kubernetes — A container orchestration platform backed by the Cloud Native Computing Foundation (CNCF), with member organisations including AWS, Google, Microsoft, IBM, Intel, and Twitter. Kubernetes has rapidly become a dominant force in managing containerised workloads at scale.
- Cloud Foundry — An open source platform-as-a-service with strong automation capabilities and broad integration support for upstream and downstream systems. It goes beyond containerised environments and has a wider application runtime reach than Kubernetes alone. The KuBo project (Kubernetes on BOSH) provides an integrated Cloud Foundry and Kubernetes stack for heterogeneous environments.
- HashiCorp tools (Terraform, Vagrant) — Terraform enables infrastructure provisioning as code across multiple cloud providers; Vagrant simplifies the creation and management of reproducible development environments.
- Perforce Helix — A portfolio of DevOps tools that extends beyond Perforce’s version control origins to cover a broad range of delivery pipeline functions.
Canonical also offers an enterprise-supported Kubernetes distribution via its Juju tooling, providing an Ubuntu-integrated option for teams standardising on that ecosystem.
The Problem of Too Many Tools
When individual engineers freely adopt tools of their own choosing, organisations can quickly end up with a fragmented toolchain — multiple incompatible systems, no unified visibility, and significant gaps in audit trails and feedback loops. Managing this situation typically follows one of two approaches:
Approach 1: Prescriptive Standardisation
Mandate a single approved toolset and require everyone to use it. In theory, this creates consistency and simplifies support. In practice, it frequently fails. Developers and system administrators are resourceful; if a tool they prefer is banned, they tend to continue using it informally while outwardly complying. The result is a false sense of control — organisations believe they have standardised their environment, only to discover fragmented tool usage when something breaks and a postmortem is conducted.
Approach 2: Pragmatic Integration
Accept that ad hoc tool adoption has already occurred and focus on creating an integration layer that brings structure to the existing mix. Most mature open source DevOps tools now expose open APIs or support plugin architectures that allow them to interoperate with other tools at a reasonable level of fidelity. This makes it feasible to build an orchestration or governance layer on top of a heterogeneous toolchain rather than attempting to eliminate it.
Where stricter controls and enterprise support are required, commercially supported distributions of open source tools can provide the additional reliability and accountability organisations need. CloudBees, for example, offers enterprise support and additional components built on top of Jenkins.
Open Systems and Long-Term Flexibility
When evaluating DevOps tooling — particularly umbrella or orchestration platforms — openness should be a primary criterion. The tooling landscape changes rapidly. Kubernetes did not exist a decade ago; tools that are central to many pipelines today may be superseded within the next five years. If an overarching platform is tightly coupled to specific underlying tools, organisations risk being locked into a vendor’s roadmap and unable to adopt newer, more capable alternatives as they emerge.
Platforms that support plugin-based or API-driven integration allow teams to swap underlying tools as the ecosystem evolves. This architectural flexibility is a key factor in long-term viability.
Several commercial platforms aim to provide this kind of open orchestration layer:
- CA Automic — Offers automated DevOps process orchestration designed to work with a range of underlying tools in an open manner.
- BMC BladeLogic — An automation suite rooted in traditional systems management that has been modernised to support cloud and container environments. When combined with other BMC products (Atrium Orchestrator, Control-M, TrueSight), it can provide broad DevOps workflow coverage.
- HPE OneView — Bridges the hardware/software boundary by enabling logical platform configuration and provisioning of physical, virtualised, or containerised software workloads.
Cloud-Native DevOps Options
Major cloud providers have invested heavily in native DevOps tooling that removes the need to self-manage infrastructure for CI/CD pipelines:
- AWS — Offers CodePipeline, CodeBuild, CodeDeploy, and related services integrated with its broader cloud platform.
- Google Cloud — Provides Cloud Build and related services, with strong Kubernetes integration through GKE.
- Microsoft Azure — Azure DevOps (formerly VSTS) provides end-to-end pipeline management tightly integrated with the Azure ecosystem.
- IBM Cloud (formerly BlueMix) — Offers a broad set of DevOps-related services and toolchain integrations.
Most of the open source tools discussed in this article are also available for self-hosted deployment on these cloud platforms, giving teams flexibility in how they manage their toolchains.
Key Takeaways for DevOps Engineers
- Governance matters. Ungoverned tool proliferation creates invisible technical debt and audit gaps. Establish visibility into what tools are in use before mandating what should be.
- Pragmatism beats prescription. Attempting to enforce a single toolset by fiat rarely succeeds in engineering organisations. Focus instead on integration layers and shared standards.
- Openness is a non-functional requirement. Evaluate any orchestration or CI/CD platform on its ability to integrate with a range of tools via open APIs or plugins — not just its current feature set.
- Business requirements drive tooling, not the other way round. DevOps tooling should ultimately serve business outcomes: faster delivery of reliable software. Tool selection should follow from that goal, not precede it.
- The landscape will keep changing. Invest in architectural flexibility — the ability to swap tools as the ecosystem evolves — rather than optimising for any specific tool combination that is popular today.
The organisations that get the most out of DevOps are those that apply rigorous thinking to their toolchain strategy: providing umbrella systems that impose order on existing complexity, while giving engineers the autonomy to use the tools that make them most effective.