In the DevOps world, we tend to speak of the pipeline in singular terms. In most real-world scenarios, however, achieving continuous delivery requires working with multiple distinct pipelines that must be coordinated into a coherent tool chain. Making good use of a continuous delivery tool chain becomes critical when effective DevOps is the end goal.
Software delivery involves more than a single team or workflow. You have to integrate your teams and workflows — and keep them in sync with one another — if you want to embrace effective DevOps fully and succeed in building a reliable, repeatable delivery process.
Identifying Your DevOps Pipelines
The path to effective DevOps starts by identifying the various pipelines you have to work with. The exact type, number, and nature of your pipelines will vary from organization to organization, but a typical software delivery process might include the following:
- Architecture pipeline: Designs new features and defines system structure.
- Coding pipeline: Handles the implementation of new features.
- Quality assurance pipeline: Tests new features for technical stability and correctness.
- User experience pipeline: Evaluates changes for usability and user impact.
- Staging and build pipeline: Compiles, configures, and packages new code.
- Deployment pipeline: Pushes validated builds into production.
While some of these pipelines occur earlier in the software delivery lifecycle than others, they cannot always be neatly arranged in strict chronological order. Design may precede coding, but different kinds of testing often occur simultaneously. Staging and building may overlap with testing phases. There is rarely a straight line running from the first pipeline to the last.
For that reason, rather than thinking of your continuous delivery tool chain as a series of pipes connected end-to-end, think of it as one large pipe into which multiple smaller pipes feed — each contributing at different times and in different volumes.
Microservices and Continuous Delivery Tool Chains
In a microservices and container-based environment, the pipeline picture becomes even more complex. You may have multiple pipelines of the same type spread across your different microservices. For example, there might be several coding pipelines, each one associated with a particular microservice or bounded domain.
At the same time, other pipelines in a microservices environment might span all services. Usability testing is a good example: even if your application is decomposed into many microservices, end-to-end usability testing typically relies on a single pipeline that evaluates the complete user experience. Testing a single microservice in isolation does not make sense for usability purposes, since users interact with the full application — not individual services.
Understanding which pipelines are service-specific and which are cross-cutting is essential for designing a tool chain that scales with your architecture.
Integrating the Continuous Delivery Tool Chain
To build the most effective continuous delivery tool chain, you must ensure that all of your smaller pipelines feed into the overall delivery chain in the right order and at the right time. Your various DevOps pipelines are interdependent: if work from one pipeline flows downstream before another pipeline is ready for it — or if it arrives too late — your workflows will fall out of sync and delivery quality will suffer.
The following steps can help you integrate your pipelines in an optimized way:
1. Identify Independent Pipelines
Ask yourself which workflows can begin without depending on the completion of any other workflow. Software architecting, for example, is often the first step in the delivery process and can start independently. Identifying these free-standing pipelines lets you kick them off immediately without waiting for upstream work.
2. Identify Dependent Pipelines and Order Them
Determine which pipelines depend on others and the sequence in which they must be arranged. Architecture needs to be largely complete before coding begins. Coding needs to reach a stable state before quality assurance and usability testing can start. Map out these dependencies explicitly and arrange your delivery chain accordingly.
3. Identify Pipelines That Should Run in Parallel
Some processes — such as different categories of testing — can and should run concurrently. If parallel-capable pipelines are forced to run sequentially, you waste time and slow the overall delivery cadence. Identify where parallelism is safe and design your tool chain to exploit it.
4. Adjust Volume and Scale
Some pipelines involve larger teams and greater throughput than others. To keep the delivery chain operating smoothly, the capacity of each pipeline must be balanced with the demands placed on it by upstream pipelines. For example, if your coding pipeline produces a high volume of new code quickly, the downstream testing pipelines must be scaled to handle that volume — otherwise a bottleneck forms and slows delivery.
5. Keep Pipelines Flexible
Agility is a core part of the DevOps mindset. When designing your pipeline structure, build in flexibility. Maintain the ability to reassign team members between pipelines when scale needs to shift. Be prepared to add or remove pipelines as organizational priorities evolve. A rigid tool chain that cannot adapt will become an obstacle rather than an accelerator.
Conclusion
Building an effective continuous delivery tool chain requires thinking beyond the concept of a single pipeline. In any realistic environment — and especially in a microservices architecture — you must account for all of the individual, disparate workflows that collectively make up your software delivery process. Each of those workflows represents a distinct pipeline, and the health of your overall delivery chain depends on how well those pipelines are identified, ordered, parallelized, balanced, and kept flexible.
Start by mapping your existing pipelines honestly. Then work to understand their interdependencies, their throughput, and where they can run concurrently. That analysis is the foundation of a continuous delivery tool chain that can scale with your organization.
Originally sourced from theserverside.com.