Harness today announced it has updated the community edition of its continuous integration (CI) platform with, among other updates, a revamped user interface.

Formerly known as Drone.io, the latest release also adds pipeline execution visualizations, governance and security controls and live CI pipeline debugging capabilities. Harness last year acquired Drone.io, an open source CI project, to extend its existing continuous delivery (CD) platform into the realm of CI. Harness claims there are now more than 50,000 active monthly users of the community edition of Harness CI.

The update to the community edition of Harness CI comes at a time when the debate over how tightly integrated CI and CD platforms need to be is heating up. Harness launched a namesake CD platform before it acquired Drone.io to add a CI capability. While most DevOps teams are proficient with CI, the CD side of the DevOps equation has proven more elusive. There are now GitOps advocates that contend the rise of Kubernetes makes it more advantageous to enable a separate CD platform to pull updates from a Git repository, rather than having them pushed out by a CI/CD platform. The challenge with CD has always been that each platform on which code is deployed is unique. Kubernetes, however, now provides a de facto standard layer of abstraction that runs atop multiple platforms.

Rydzewski said the integrated CI/CD approach will ultimately prevail, because it enables DevOps teams to more easily integrate a range of processes, including testing, that are required to consistently build and deploy applications at scale. Harness is, essentially, making a case for solving the CD challenge using a platform that encourages – but doesn’t require – a DevOps team to replace their existing CI platform.

Regardless of how DevOps evolves, the CD issue needs to be addressed. As the enterprise becomes more extended with the rise of both cloud and edge computing, the number of platforms a DevOps team needs to support continuously expands.

Despite the fact that DevOps teams employ CI/CD platforms, it is clear most of those organizations are not overly committed to the way they currently manage CD. It may be years before most organizations finally implement a consistent set of processes for automating application deployment.

In the meantime, the number of organizations just starting out on their DevOps journey continues to increase. There are still plenty of instances where organizations have yet to deploy any type of CI/CD platform. It’s also worth noting that, in many cases, different DevOps teams within the same enterprise have decided to deploy different CI/CD platforms.

Regardless of how DevOps evolves the pressure to automate is rising. The challenge many DevOps teams face today is the current platform they are relying on to automate the building and deployment of applications requires a small army of specialists relying on all too often manual processes to maintain.

Continuous Integration and Continuous Delivery (CI/CD) can bring a seamless integration from end-to-end for the software development and deployment process. By doing this, CI/CD allows developers to dedicate more of their time developing code to improve software features instead of worrying about the deployment.

Yet, they are still faced with many security challenges. CI/CD might speed up the process but not the security. With DevSecOps – Development, Security, and Operations -, however, there is a possibility to accelerate the delivery of security within the software. DevSecOps engineers try to operate most security controls as part of the software by introducing it as a design constraint and then getting it checked by the CI/CD pipelines without damaging the integrity of controls.

Why DevSecOps?

As digital transformations increase, there is a vital need for safe and secure software, otherwise, everything, from the building to the delivery can be at risk. Security breaches are now one of the biggest threats to companies and products.

DevSecOps promotes collaboration between the development and security teams, hence, avoiding late handoffs to security professionals. By introducing security at the beginning of the process, the value and quality of the product can only be reinforced. Indeed, without DevSecOps, the software might be considered unsafe at the last minute, thus causing multiple costly iterations. With DevSecOps, security standards are implemented directly into the pipelines making the products more secure from the beginning.

Overall, DevSecOps ensures credibility and agility in the market, as well as trust with consumers.

DevSecOps in CI/CD

There are many security vulnerabilities that can exist in open-source software. Hence, implementing DevSecOps practices within CI/CD will bring continuity to securing software deliveries.

Integrating automated security checks within the pipeline will allow developers to have early warnings of vulnerabilities and monitor any security defects or else. With an integrated continuous security approach, companies can expand while upgrading their security and development process as it goes.

Moreover, unit tests and static code analysis operate close to source code as well as run checks without executing the code. Hence, investing in security unit tests and static analyzers will only be beneficial as it can speed up the lifecycle while detecting quickly any vulnerabilities.

The future of DevSecOps and CI/CD pipelines

With the many challenges our world bring today, security is crucial in order to remain on top of the market. With DevSecOps, companies are able to speed up their CI/CD pipelines all the while keeping it secure from any vulnerabilities. Collaboration and communication are then vital between development and security teams and shouldn’t be overlooked.

With the rise of DevSecOps, security has become an important part of the continuous delivery pipeline. Having continuity and security ensures the best software delivery.

Continuous Integration and Continuous Delivery (CI/CD) can bring a seamless integration from end-to-end for the software development and deployment process. By doing this, CI/CD allows developers to dedicate more of their time developing code to improve software features instead of worrying about the deployment.

Yet, they are still faced with many security challenges. CI/CD might speed up the process but not the security. With DevSecOps – Development, Security, and Operations -, however, there is a possibility to accelerate the delivery of security within the software. DevSecOps engineers try to operate most security controls as part of the software by introducing it as a design constraint and then getting it checked by the CI/CD pipelines without damaging the integrity of controls.

Why DevSecOps?

As digital transformations increase, there is a vital need for safe and secure software, otherwise, everything, from the building to the delivery can be at risk. Security breaches are now one of the biggest threats to companies and products.

DevSecOps promotes collaboration between the development and security teams, hence, avoiding late handoffs to security professionals. By introducing security at the beginning of the process, the value and quality of the product can only be reinforced. Indeed, without DevSecOps, the software might be considered unsafe at the last minute, thus causing multiple costly iterations. With DevSecOps, security standards are implemented directly into the pipelines making the products more secure from the beginning.

Overall, DevSecOps ensures credibility and agility in the market, as well as trust with consumers.

DevSecOps in CI/CD

There are many security vulnerabilities that can exist in open-source software. Hence, implementing DevSecOps practices within CI/CD will bring continuity to securing software deliveries.

Integrating automated security checks within the pipeline will allow developers to have early warnings of vulnerabilities and monitor any security defects or else. With an integrated continuous security approach, companies can expand while upgrading their security and development process as it goes.

Moreover, unit tests and static code analysis operate close to source code as well as run checks without executing the code. Hence, investing in security unit tests and static analyzers will only be beneficial as it can speed up the lifecycle while detecting quickly any vulnerabilities.

The future of DevSecOps and CI/CD pipelines

With the many challenges our world bring today, security is crucial in order to remain on top of the market. With DevSecOps, companies are able to speed up their CI/CD pipelines all the while keeping it secure from any vulnerabilities. Collaboration and communication are then vital between development and security teams and shouldn’t be overlooked.

With the rise of DevSecOps, security has become an important part of the continuous delivery pipeline. Having continuity and security ensures the best software delivery.

The Global Continuous Integration Tools (CI) Market is anticipated to rise at a considerable rate over the estimated period between 2020 and 2027. The Global Continuous Integration Tools (CI) Market Industry Research Report is an exhaustive study and a detailed examination of the recent scenario of the Global Continuous Integration Tools (CI) industry.

Continuous integration is a software development practice in which developers regularly unite their codes changed into a central fount, automated builds and tests are being processed. The main objective of continuous integration (CI) is to discover quick address bugs.

Growing demand for continuous integration (CI) tools in all industry verticals, high demand of automation in the development process of software, and rising productivity with the help of continuous integration (CI) tools are the factors driving the growth of the continuous integration (CI) tools market. Traditional integration methods and customers attraction towards the software updates are the factors restraining the growth of continuous integration (CI) tools market.

Major Key Players Covered In This Report: Atlassian, AWS, Cloudbees, IBM Corporation, Jetbrains, Micro Focus, Microsoft Corporation, Oracle, Puppet, Red Hat

The study conducts SWOT analysis to evaluate strengths and weaknesses of the key players in the Continuous Integration Tools (CI) market. Further, the report conducts an intricate examination of drivers and restraints operating in the market. The report also evaluates the trends observed in the parent market, along with the macro-economic indicators, prevailing factors, and market appeal with regard to different segments. The report predicts the influence of different industry aspects on the Continuous Integration Tools (CI) market segments and regions.

The global Continuous integration tools (CI) market is segmented on the basis of, deployment model, and size. On the basis of deployment model, market is segmented as on premises, on cloud. On the basis of size, market is segmented as SMEs, large enterprise.

The research on the Continuous Integration Tools (CI) market focuses on mining out valuable data on investment pockets, growth opportunities, and major market vendors to help clients understand their competitor’s methodologies. The research also segments the Continuous Integration Tools (CI) market on the basis of end user, product type, application, and demography for the forecast period 2020–2027. Comprehensive analysis of critical aspects such as impacting factors and competitive landscape are showcased with the help of vital resources, such as charts, tables, and infographics.

This report strategically examines the micro-markets and sheds light on the impact of technology upgrades on the performance of the Continuous Integration Tools (CI) market.

Continuous Integration Tools (CI) Market Segmented by Region/Country: North America, Europe, Asia Pacific, Middle East & Africa, and Central & South America

Highlights of the report:

A complete backdrop analysis, which includes an assessment of the parent market
Important changes in market dynamics
Market segmentation up to the second or third level
Historical, current, and projected size of the market from the standpoint of both value and volume
Reporting and evaluation of recent industry developments
Market shares and strategies of key players
Emerging niche segments and regional markets
An objective assessment of the trajectory of the market
Recommendations to companies for strengthening their foothold in the market

The Insight Partners is a one stop industry research provider of actionable intelligence. We help our clients in getting solutions to their research requirements through our syndicated and consulting research services. We are a specialist in Technology, Healthcare, Manufacturing, Automotive and Defence.

Are you an AWS DevOps Engineer who is keen to lead and define the DevOps practices of an organization going through a major transformation? Reporting to the CTO you will have the opportunity to work closely with the architects and delivery team and define how operations work. You will be responsible for delivery DevOps best practice and will help define how the operations will work.

Key Benefits of this role

Opportunity to be a lead in a transformation programme
Very competitive salary and benefits
Challenging role with ability to make a real difference to the delivery process
The Role

You will be required to

1 Provide technical leadership in effectively managing the AWS environment and DevOps toolchain
2 Automate deployment of container applications/services in Kubernetes, system configurations and security settings
3 Provide guidance and support during deployment
4 Create and manage CI/CD pipelines in Jenkins
5 Write and maintain Terraform scripts
6 Build DevOps data flow architecture map
7 Help define development standards, procedures and guidelines to improve the overall  development process
The Person:

1 You will have experience as a technically hands on DevOps Engineer (with leadership experience)
2 Demonstrable experience of driving DevOps best practice and creating and managing CI/CD pipelines and ideally have experience with Jenkins
Experience with Kubernetes
3 Strong experience in AWS administration and development
4 Experience of configuration management tools such as Chef, Puppet, Ansible
5 Excellent communication skills and ability to lead and get buy in for new project roll outs
Strong people management and leadership skills

San Francisco conference to offer 135 sessions presented by experts on current digital transformation and software delivery topics

CloudBees, the lead organizing sponsor, the Jenkins community and the Jenkins X community, announced today availability of the agenda preview for DevOps World | Jenkins World 2019 San Francisco. This year’s theme is “Deliver” and the conference will educate attendees on the value that digital transformation has in delivering software more efficiently, more quickly and with higher quality.

The conference is taking place in two locations this year:

• August 12 – 15, Moscone West, San Francisco
• December 2 – 5, Lisbon Congress Center, Lisbon, Portugal
  (Agenda preview available in August)

The conferences will be the largest gathering of IT executives, DevOps practitioners and Jenkins users in the world, with DevOps and continuous delivery experts presenting sessions. There are many opportunities for attendees to learn, network and explore DevOps and many software delivery topics.

This year’s conference in San Francisco includes a total of 135 sessions covering diverse DevOps, software delivery and Jenkins topics. In San Francisco, sessions will be conducted by practitioners from organizations such as: Adobe, Capital One, CloudBees, CVS, Eli Lilly, Fugue, Google, IBM, Microsoft, SAP Labs and Verizon. Sessions will cover topics ranging from cloud native and Kubernetes, to pipeline development and management, CI/CD topics, release management, data and reporting, DevOps and security/compliance.

• Conference sessions
  A sampling of sessions from the DevOps World | Jenkins World San Francisco agenda includes:
  • From Zero to DevOps Superhero: The Container Edition
    Jessica Dean, Microsoft
  • Creating a CI/CD Pipeline for Your Shared Libraries
    Roderick Randolph, Capital One
  • Cloud Native CI/CD with Jenkins X and Tekton
    James Rawlings, CloudBees
  • Using Chaos Engineering for Resilient Cloud Infrastructure Security
    Josh Stella, Fugue
  • Jenkins X Optimized for Google Cloud Platform
    Craig Barber, Google
  • Continuous Delivery Directly to the Car
    Rafal Kowalski, Grape Up
  • 10 Attributes of the DevSecOps Elite
    Derek Weeks, Sonatype and Hasan Yasar, Carnegie Mellon Software Engineering Institute
  • Can Jenkins be the Engine of Mobile DevOps?
    Shashikant Jagtop, XCTEQ Limited

“DevOps World | Jenkins World provides a broad range of software delivery topics. It is the most comprehensive event for IT practitioners around the globe to learn how to accelerate software delivery and decrease time to value,” said Sacha Labourey, CEO and co-founder, CloudBees.

US Sponsors

Sonatype, Delphix, IBM, KPMG, MacStadium, Sauce Labs, GitHub, Go2Group, Harness, LogDNA, Micro Focus, NowSecure, Perfecto, Plutora, SmartBear, SonarSource, WhiteSource Software, Applitools, CheckMarx, CircleCI, Codefresh, Column Technologies, Compuware, Datical, GitLab, iTMethods, LaunchDarkly, Neotys, Perforce, Synopsys, Sysdig, Twistlock, XebiaLabs, XpandIT (XRAY), GitKraken by Axosoft, Gradle, Honeycomb, IncrediBuild, Instana, OverOps, Scalyr, Grape Up and Verifa.

About CloudBees

CloudBees is powering the continuous economy by offering the world’s first end-to-end continuous software delivery management system (SDM). For millions of developers and product teams driving innovation for businesses large or small, SDM builds on continuous integration (CI) and continuous delivery (CD) to enable all functions and teams within and around the software delivery organization to best work together to amplify value creation.

CloudBees is the CI, CD and application release orchestration (ARO) powerhouse, built on the commercial success of its products as well as its open source leadership. CloudBees is the largest contributor to Jenkins and Jenkins X, and a founding member of the Continuous Delivery Foundation (CDF). From startups with full-stack developers practicing NoOps to large Fortune 100 companies, CloudBees enables all software-driven organizations to intelligently deploy the right capabilities at the right time.

Over 3,500 of the world’s best-known brands and over 50% of the Fortune 500, invest in CloudBees because of its ability to work across any cloud, in any development environment and to balance corporate governance and control with developer flexibility and freedom. CloudBees is home to the world’s leading DevOps experts, helping thousands of companies harness the power of “continuous everything” and putting them on the fastest path from great idea, to great software, to great business value.

Backed by Matrix Partners, Lightspeed Venture Partners, Verizon Ventures, Delta-v Capital, Golub Capital and Unusual Ventures, CloudBees was founded in 2010 by former JBoss CTO Sacha Labourey and an elite team of continuous integration, continuous delivery and DevOps professionals.

The Real Problem: Culture, Not Tools

When organizations ask how to start with DevOps, the instinct is often to reach for tools — adopt containers, migrate to the cloud, stand up a CI/CD pipeline. But according to Eric Schoonover and Robert Duffy, co-founders of the consultancy Sodo (Ship On Day One), that instinct misses the deeper issue.

Duffy, who brings experience from Amazon and Time magazine, and Schoonover, who has worked at Microsoft, Netflix, and the U.S. Department of Defense’s Digital Service, argue that the cultural transformation is the hard part — and the part most organizations underestimate.

“A lot of companies are still doing engineering delivery like 1800s manufacturing was done. It’s a craft-based system, parsing work out to different teams. We want to help non-tech companies think about the cultural transformation. It’s not just about moving to the cloud and installing Docker.”

— Robert Duffy, co-founder, Sodo

Notably, Duffy points out that neither he nor Schoonover used the term “DevOps” during their time at Amazon and Netflix. The practices were simply how software engineering was done. That framing is useful: DevOps is not a product you install, but a set of behaviors and capabilities that high-performing teams exhibit naturally.

Small Plans Work Best: The 15-Capability Framework

To create a repeatable starting point, Duffy and Schoonover interviewed approximately 150 high-performing software development teams, documenting observations and patterns. From that research, they distilled their findings into 15 capabilities, each broken into four stages of maturity.

The goal of this structure is deliberate: avoid overwhelming the organization. Instead of presenting a sweeping transformation roadmap, the framework identifies the highest-priority items and sequences them so that early wins build credibility with business stakeholders.

“We try to get to a small set of very actionable items that we helped select that justify the ROI calculations and case studies. We help companies recognize the size of the problem and attack it in small identified chunks to gain credibility with the business.”

— Robert Duffy, co-founder, Sodo

This approach mirrors advice found throughout DevOps literature: start with a value stream mapping exercise, identify bottlenecks, pick one constraint to improve, and demonstrate measurable progress before expanding scope. The specifics of Sodo’s 15 capabilities were not publicly detailed in the original source, but the principle — narrow focus, visible momentum — is well-established in DevOps practice.

Key Principles for Starting a DevOps Transformation

Whether you engage external consultants or pursue transformation internally, the following principles reflect what high-performing teams consistently do:

• Start with culture, then process, then tools. Technology is the easiest part to change. Organizational habits and incentive structures are the hard part.
• Identify your value stream. Understand how work flows from idea to production, and where it gets stuck. This surfaces the highest-leverage improvement opportunities.
• Pick small, visible wins first. Early successes build trust with leadership and reduce resistance from skeptical engineers. Don’t attempt to transform everything at once.
• Make CI/CD a baseline expectation. Continuous integration and continuous delivery pipelines are not optional extras — they are the foundation that enables everything else, including fast feedback and safe deployments.
• Treat legacy systems as a continuum. Not every system is equally amenable to DevOps practices. Some will adopt CI/CD quickly; others, particularly those involving custom hardware or highly regulated environments, require more time and creative approaches. Acknowledge this spectrum explicitly rather than pretending all workloads are equal.
• Ship something. The discipline of getting working software out the door — even incrementally — creates real feedback loops and real learning. “Something shipped out the door is really what we preach,” Duffy said.

DevOps in Non-Traditional Environments: The Legacy and Defense Context

One of the more instructive aspects of the Sodo case study involves a defense contractor — an environment about as far from a Silicon Valley startup as possible. A senior engineering manager at that organization described the challenge plainly: the organization wanted a consistent software delivery process across all systems, but the range of environments was enormous, from straightforward web applications to custom hardware with deeply embedded software.

“We want to look at software development as a continuum. But at one spectrum, we have the low-hanging fruit that it’s easy to see how to make changes to, and on the other spectrum we have custom hardware and software that is not low-hanging fruit. We want to have the same kind of systems in place for both so that we can have CI and CD running. That’s the area we’re a little concerned about.”

— Senior engineering manager, defense contractor (anonymous)

This is a real and common challenge. Organizations in aerospace, defense, financial services, healthcare, and manufacturing all face legacy systems that resist easy automation. The practical answer is not to exclude those systems from DevOps thinking, but to apply the same principles at whatever pace and with whatever constraints the system imposes — while capturing quick wins on the more flexible parts of the portfolio.

The same manager noted that the Sodo team’s willingness to work alongside engineers — rather than simply diagnosing problems and leaving — was a key factor in acceptance. Engineers, who are often skeptical of outside consultants, responded positively precisely because the consultants did not arrive claiming to have all the answers.

The Role of External Help: When Consultants Make Sense

Bringing in external consultants to help start a DevOps transformation is not the only option, but it is a legitimate one — particularly for organizations where internal advocates lack the organizational authority or cross-functional visibility to drive change on their own.

The value a good DevOps consultancy brings is not primarily technical knowledge (your engineers likely already know what CI/CD pipelines and infrastructure-as-code are). The value is in:

• Providing an outside perspective that internal politics cannot easily dismiss
• Translating engineering improvements into business ROI language that resonates with executives
• Offering a structured, repeatable approach that prevents the team from getting lost in tool debates
• Creating accountability and momentum through regular check-ins and visible milestones

The Sodo team explicitly declined to simply deliver projects for clients. Their stated focus was on teaching organizations how to sustain DevOps practices independently — which is the right model. A transformation that depends indefinitely on external consultants is not a transformation.

Patience and Momentum Are Both Required

Duffy’s framing of “patience is important, but so is momentum” captures a genuine tension in DevOps transformations. Moving too fast burns out teams and produces fragile systems. Moving too slowly allows organizational inertia to kill the initiative before it gains traction.

The practical resolution is to run a short, focused pilot — a single team, a single service, a single pipeline — and make that work well before expanding. Document what changed, measure the outcome, and use that story to justify the next step. This is the pattern that works in organizations of all sizes and industries.

DevOps is not a destination you reach; it is a direction you move in. The question of how to start matters less than the decision to start — and the discipline to keep going.

Original reporting by TechTarget. This article summarizes and expands on the source material for a DevOps engineering audience.

Understanding Git Hooks

Git hooks are Git’s native mechanism for triggering custom scripts before or after core repository operations such as commit, merge, push, and checkout. Think of them as Git’s plugin system — lightweight, portable, and entirely under your control.

Every Git repository contains a .git/hooks directory populated with example hook scripts. To activate a hook, rename the relevant example file (removing its .sample extension), make it executable, and add your logic. The Git documentation covers installation in detail.

Client-Side vs. Server-Side Hooks

Hooks fall into two broad categories:

• Client-side hooks run on a developer’s local workstation and affect local operations like committing and checking out branches.
• Server-side hooks run on the remote Git host (e.g., Bitbucket Server, GitLab, or GitHub Enterprise) and can gate pushes and merges centrally across the whole team.

Pre-Receive vs. Post-Receive Hooks

Hooks are also classified by when they fire relative to the triggering operation:

• Pre-receive (pre-) hooks run before an operation completes. Crucially, they can exit with a non-zero status to cancel the operation entirely — making them ideal for enforcing quality gates.
• Post-receive (post-) hooks run after an operation has already completed and cannot cancel it. They are well-suited for automation tasks like sending notifications or triggering deployments.

Common tasks automated with Git hooks include:

• Validating that a commit message references the correct issue tracker key (e.g., a Jira ticket)
• Enforcing merge preconditions such as passing tests or minimum code coverage
• Posting build or merge notifications to a team chat channel
• Configuring the workspace automatically when switching branches

Hook 1: Enforce Clean Builds Before Merging to Main

Developers are generally diligent, but it is easy to forget to check the CI status on a branch — especially when multiple contributors are pushing changes concurrently. A server-side pre-receive hook closes this gap by making the check automatic and mandatory.

The hook inspects incoming push operations targeting the main (or master) branch. When it detects a merge, it queries the CI server for the latest build result on the source branch. If any tests are failing, the hook exits with a non-zero code, rejecting the push and printing a message explaining why.

A reference implementation by Atlassian’s Tim Petterson, designed to work with Bamboo CI, is available on Bitbucket. The script is intentionally straightforward to customise for other CI servers such as Jenkins or CircleCI — the key integration point is an API call to retrieve the latest build status for a given branch and revision.

How It Works in Practice

1. A developer runs git push origin feature/my-branch and opens a pull request to merge into main.
2. On push (or on merge attempt, depending on your server configuration), the pre-receive hook fires.
3. The hook script calls the CI server’s API to retrieve the build status for the branch’s HEAD revision.
4. If the build is red (failing tests), the push is rejected with a descriptive error message.
5. If the build is green, the push proceeds normally.

Hook 2: Protect Hard-Earned Code Coverage

Teams that have invested significant effort in retroactively adding test coverage to a legacy codebase know how painful it is to watch that coverage percentage erode as new features ship without accompanying tests. A pre-receive server-side hook can enforce a coverage floor on the main branch.

This hook also intercepts incoming merges to main. It then performs two lookups against the CI server:

1. Current coverage on main — the baseline the incoming branch must meet or exceed.
2. Coverage on the feature branch — pulled from the build associated with the commit being merged.

If the branch’s coverage is lower than main’s, the merge is rejected.

Prerequisites and Configuration

Because most CI servers do not expose code coverage metrics through a standard remote API, the script fetches the coverage report directly as a build artifact. This requires that your build pipeline is configured to publish the coverage report as a shared artifact on every build — for both the main branch and feature branches.

The reference implementation targets Bamboo and Clover (Atlassian’s code coverage tool for Java and Groovy). It can be adapted for other tool combinations — the essential requirement is that your CI server makes coverage report artifacts accessible via a predictable URL or API endpoint.

Important: This hook does not generate code coverage. It only reads coverage data that your existing build process already produces. If your builds do not measure coverage today, you will need to add that to your pipeline first.

Hook 3: Surface Branch Build Status at Checkout

Because friends don’t let friends check out broken branches.

This is a client-side post-checkout hook. Unlike the server-side hooks above, it runs on each developer’s local workstation and fires automatically whenever they switch branches with git checkout (or git switch).

What the Hook Does

1. After git checkout completes, the script reads the HEAD revision of the newly checked-out branch from the local repository.
2. It queries the CI server to determine whether that revision has been built — and if so, whether the build passed or failed.
3. The result is printed directly in the terminal alongside the normal checkout output.

Two Practical Scenarios

Creating a new feature branch from main: The hook tells you immediately whether the HEAD of main built successfully, confirming it is a safe and stable starting point for new work.

Detecting a stale local copy: If the hook reports a failed build for a revision but the team’s CI dashboard shows a green build for that branch, your local copy is behind. You can then decide whether to git pull before continuing.

This hook is the lowest-friction option to introduce to a team. It requires no server-side configuration, no repository admin access, and no changes to the shared workflow — each developer installs it independently in their local .git/hooks directory.

Tooling and Portability

The three hooks described above were originally written to target Bamboo (build server), Clover (code coverage), and Bitbucket (Git hosting). However, Git hooks themselves are vendor-neutral — they are simply executable scripts in a well-known directory. The CI-specific logic in any hook script is limited to a handful of HTTP API calls or artifact URL patterns, which can be swapped out for equivalent calls to Jenkins, GitHub Actions, GitLab CI, CircleCI, or any other system that exposes build and coverage status via HTTP.

If you work in a polyglot environment or want to manage hooks consistently across a team, tools such as pre-commit or Husky can help version-control and distribute client-side hooks alongside your project code.

Summary

Git hooks are one of the most underutilised features in a CI/CD toolkit. Introducing even one of these hooks can measurably reduce the number of broken builds that reach your main branch — and the post-checkout hook in particular pays dividends immediately with zero team coordination required.

The CI Server Is Not Enough

Installing a CI server such as Jenkins or Concourse is now standard practice for any team that claims to be doing DevOps. But having a server that runs a job every hour or two is not the same as practising continuous integration in any meaningful sense.

“I’d say that continuous integration is nowadays a commodity, up to the point where people actually have forgotten what it actually means. People are using integration servers just to do a regular compile,” said Eberhard Wolff, a software architect and Fellow at innoQ, in an interview with TheServerSide.

Wolff points to a pattern he sees repeatedly: a team creates a long-lived feature branch in their Git repository, works on that branch independently for weeks or months, and uses the CI server only to compile that branch. The CI server is doing work, but the team is not integrating continuously — they are integrating rarely, in large batches, which is the opposite of what CI is designed to achieve.

“They’re missing the original point about continuous integration. That’s the state we are in. People have forgotten about the original ideas behind CI.”

— Eberhard Wolff, innoQ

True continuous integration requires that every developer integrates their work into a shared mainline frequently — ideally at least once per day — so that integration problems are caught early and in small, manageable increments. A CI server is a tool that supports this workflow; it cannot create the workflow by itself.

Continuous Integration and Effective DevOps

The relationship between CI and DevOps is more than superficial. DevOps as a practice aims to shorten the feedback loop between writing code and running it in production. CI is the first stage of that loop — catching defects before they accumulate. Without genuine CI (frequent integration into a shared branch, automated build, automated tests), the later stages of the pipeline — continuous delivery and continuous deployment — become unreliable.

Wolff argues that the CI server is only one small piece of the puzzle. The larger pieces are comprehensive automated testing and the automation of manual processes that would otherwise interrupt the pipeline.

The “Continuously Interrupted” Anti-Pattern

One of the most instructive concepts in Wolff’s thinking is what happens when a pipeline is technically automated but still requires a human to approve or perform steps along the way. He describes this as turning “continuous delivery” into “interrupted delivery.”

“Just having Jenkins and just having a deployment pipeline is not the key challenge. The key challenge is going into production without sign-off,” Wolff said.

When Wolff works with new customers, he starts by asking what the biggest impediments to running their continuous delivery pipeline are. The answer is almost always the same: a manual process — frequently in the user acceptance testing (UAT) phase — that requires a human to review and approve before the pipeline can proceed. That single manual gate can negate much of the speed and reliability benefit that automation is supposed to provide.

This does not mean that all governance and oversight should be eliminated. It means that wherever possible, checks that currently require human intervention should be replaced with automated tests that can make the same determination faster and more consistently. The goal is to shift quality assurance left — catching problems earlier in the pipeline through automation rather than relying on a human reviewer at the end.

Automation, Testing, and Confidence in Deployment

There is an important dependency that is easy to overlook: you cannot safely automate deployment unless you have a comprehensive automated test suite. Without tests that reliably catch regressions, removing human sign-off from a deployment pipeline is genuinely risky. This is why Wolff treats testing and automation as inseparable topics.

A robust test strategy for a continuous delivery pipeline typically includes:

• Unit tests — fast, isolated tests that verify individual functions or classes and run on every commit.
• Integration tests — tests that verify how components interact, including databases, message queues, and external services.
• Acceptance tests — automated tests written in terms of business behaviour (often using tools such as Cucumber or Gherkin) that replace or supplement manual UAT.
• Performance and security tests — automated checks that can be run on a schedule or as a pipeline stage to catch non-functional regressions.

Only when this test pyramid is in place can a team confidently remove the manual sign-off gate and allow the pipeline to deploy to production automatically.

A Practical Approach: Tools, Examples, and Hands-On Guidance

Wolff’s book, A Practical Guide to Continuous Delivery, is designed to be less theoretical than much of the existing literature on the subject. Rather than explaining concepts in the abstract, it covers specific tools — including Chef, Docker, Vagrant, Jenkins, and SonarQube — and provides concrete examples of how to use them to build a real continuous delivery pipeline.

“I talk about specific tools, I provide advice about how to use them, I have examples of how to use the tools,” Wolff said. The goal is to give readers enough practical experience with individual tools that they can assemble a working pipeline themselves, rather than understanding only the theory behind why such a pipeline should exist.

This practical orientation matters because one of the common failure modes when adopting DevOps practices is that teams understand the goals — faster delivery, fewer defects, higher confidence — but struggle to translate those goals into a working system of pipelines, scripts, and configuration. A tool-focused, example-driven resource closes that gap.

Key Takeaways for DevOps Engineers

• A CI server is a prerequisite, not a solution. Running builds on Jenkins or any other CI platform does not make your process continuous integration unless developers are integrating into a shared branch frequently.
• Long-lived feature branches undermine CI. The longer a branch lives in isolation, the larger and riskier the eventual merge becomes. Trunk-based development or very short-lived branches are the approaches most compatible with genuine CI.
• Manual gates break the pipeline. Any step that requires a human to intervene — approving a deployment, executing a test case by hand — introduces delay and variability. Identify these gates and replace them with automated equivalents.
• Automation requires test coverage. Before removing human oversight from a deployment pipeline, invest in a comprehensive automated test suite. The tests are what make automation safe.
• DevOps is defined by outcomes, not tools. The goal is to reduce the cycle time from code commit to production deployment while maintaining quality and reliability. Tools support that goal; they do not achieve it automatically.

What Is Continuous Integration?

Continuous Integration is a software development practice in which every team member integrates their work into a shared repository frequently — ideally at least once per day. Each integration is verified by an automated build that runs a regression test suite, allowing the team to detect errors as quickly as possible.

Martin Fowler, whose writing on CI remains widely referenced, describes the practice as the foundation that keeps a team’s codebase in a consistently working state. The key rule is simple but strict: the build must never be left in a broken state overnight. If a developer’s commit breaks the build, it is their responsibility to fix it before the end of the working day.

CI did not appear in isolation. It was popularised as a core practice of Extreme Programming (XP) and sits alongside other engineering disciplines such as configuration management, automated compilation, build scripting, automated deployment, and comprehensive testing — all combined into a single, repeatable automated pipeline.

Why CI Is Inseparable from Agile

Agile methodology is built on iterative, incremental delivery and rapid feedback loops. Without CI, those principles break down quickly. Consider the alternative: a team that relies primarily on manual testing may not receive feedback on a code change for several hours or even days. During that window, other changes accumulate, making it extremely difficult to isolate the root cause of any failure.

CI closes this feedback loop. At the end of each working day, every team member knows that their code compiles, passes the test suite, and integrates correctly with the rest of the codebase. That confidence is what allows Agile teams to move fast without accumulating hidden technical debt.

CI and Test-Driven Development

CI provides the delivery mechanism; Test-Driven Development (TDD) provides the quality gate. In TDD, a developer writes a failing unit test before writing the implementation code, then writes just enough code to make the test pass, and refactors. As the codebase grows, the test suite grows with it.

When CI runs the build on every commit, it executes the full test suite automatically. New code that breaks existing behaviour is caught immediately and attributed to a specific commit — not discovered weeks later during an integration phase. This combination means:

• Developers get rapid, precise feedback on every change.
• Regressions are caught at the earliest possible moment.
• The team can refactor with confidence, knowing the test suite will flag unintended side effects.
• An always-green build pipeline becomes a reliable signal of software quality.

Together, CI and TDD form a safety net that makes it realistic — not reckless — to ship frequently and respond to changing requirements inside a sprint.

Key Benefits of Continuous Integration

1. Reduced Integration Risk

The longer a feature branch lives without being merged, the greater the divergence from the mainline. Frequent integration keeps the gap small. There is less room for conflicting assumptions between developers, fewer “works on my machine” surprises, and a dramatically lower risk of expensive rework caused by late-discovered incompatibilities.

2. Fast Issue Detection

Automated builds and tests run on every commit. Problems surface within minutes rather than days. Think of it as a smoke detector for your codebase: it does not prevent fires, but it ensures that when something goes wrong, the team knows about it immediately — while the change is still fresh in the committing developer’s mind.

3. Process Automation

Agile is fundamentally about efficiency. CI eliminates slow, error-prone manual steps — compiling code, running tests, packaging artefacts — by making them automatic and repeatable. Engineers spend time writing code, not running manual build procedures.

4. Deployable Software at Any Time

A well-configured CI pipeline produces a tested, deployable artefact on every successful build. This means a release decision becomes a business decision, not a technical crisis. Teams that lack CI frequently experience delayed releases caused by unforeseen integration problems that only surface when all the moving parts are finally assembled.

5. Confidence in Code Quality

With every green build, the team has evidence — not just hope — that the software behaves as specified. Automated checks can also enforce coding standards, run static analysis, and measure test coverage, making quality a continuous measurement rather than a checkpoint at the end of a release cycle.

6. Team Transparency

Build status should be visible to everyone. A passing build increases collective confidence. A failing build is an immediate, unambiguous signal that prompts collaboration: the developer who broke the build can ask colleagues to help diagnose the problem while the context is still current. Transparency in the pipeline removes the “who broke it?” culture and replaces it with shared ownership of quality.

Business Impact

For engineering managers and product owners, the case for CI is equally compelling from a business perspective:

• Faster time to market: Smaller, frequently integrated changes can be released as soon as they are validated, rather than waiting for a large batch release.
• Better response to changing requirements: When integration is cheap and continuous, teams can pivot quickly without accumulating a mountain of deferred integration work.
• Lower rework costs: Bugs caught at commit time cost a fraction of what they cost to fix after they have propagated through the system.
• Predictable delivery: Stakeholders can rely on a steady stream of working software increments rather than infrequent, high-risk releases.

Getting Started with CI

If your team is new to CI, the following practices form a practical starting point:

1. Maintain a single source repository and ensure every developer commits to the mainline (or short-lived feature branches that merge daily).
2. Automate the build so that a single command compiles and packages the entire application.
3. Write automated tests that cover critical paths, and add to the suite with every new feature or bug fix.
4. Run the build on every commit using a CI server (Jenkins, GitHub Actions, GitLab CI, CircleCI, and others are common choices).
5. Fix broken builds immediately. A broken build is the team’s top priority — nothing else is merged until the build is green again.
6. Make build results visible to the whole team via dashboards, notifications, or status badges.

Conclusion

Agile transformation is not complete with stand-ups and Kanban boards alone. The engineering practices that underpin genuine agility — Continuous Integration and Test-Driven Development — are what give teams the technical foundation to deliver quickly, safely, and repeatedly. CI is not an optional add-on; it is the mechanism that makes iterative delivery sustainable at scale.

Teams that invest in CI consistently report fewer integration failures, faster feedback cycles, and greater confidence in their releases. Once adopted, CI is rarely abandoned — because the alternative, going back to infrequent manual integration, is simply too costly.

Why Jenkins Pipelines and Blue Ocean?

Mature engineering teams tend to iterate constantly on their CI/CD workflows. Jenkins has long been a staple of that process, but its classic UI and job-configuration model can become unwieldy as project count grows. Cloud-native alternatives like Travis CI and Drone gained popularity in part because they store build configuration alongside source code in Git — something Jenkins historically made difficult.

Two Jenkins features close that gap significantly:

• Jenkins Pipelines — lets you define your entire build, test, and deployment process as code in a Jenkinsfile committed to your repository.
• Blue Ocean — a plugin that rethinks the Jenkins user experience, surfacing pipeline status, stage results, and test data with minimal navigation.

“Blue Ocean is a project that rethinks the user experience of Jenkins, modelling and presenting the process of software delivery by surfacing information that’s important to development teams with as few clicks as possible.”

The sections below walk through spinning up Jenkins Blue Ocean with Docker Compose, writing a declarative Jenkinsfile, and using key Pipeline features such as Docker agents, conditional stages, and post actions.

Running Jenkins Blue Ocean with Docker Compose

Containerising Jenkins keeps your host clean and makes the setup reproducible. You need three files: a Dockerfile, a docker-compose.yaml, and a docker.conf that exposes the Docker socket so Jenkins can spin up agent containers.

Dockerfile

Start from the official Blue Ocean image and copy in the Docker daemon configuration:

FROM jenkinsci/blueocean:latest
MAINTAINER R. Moorman <rob@moori.nl>
USER root
COPY docker.conf /etc/init.d/docker

docker-compose.yaml

Mount the Docker socket so Jenkins can create sibling containers on the host, and persist Jenkins home to a local directory:

version: '2'
services:
  jenkins:
    build: .
    ports:
      - 8080:8080
    volumes:
      - ./jenkins_home:/var/jenkins_home
      - /var/run/docker.sock:/var/run/docker.sock

docker.conf

Expose the Docker daemon on TCP as well as the Unix socket so the Jenkins Docker plugin can reach it:

DOCKER_OPTS="-H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock --tls=false"

With all three files in the same directory, start the stack:

docker-compose up

Jenkins will be available at http://localhost:8080/. Complete the initial setup wizard, then navigate to the Blue Ocean view via the sidebar.

Writing Your First Declarative Jenkinsfile

After switching to the Blue Ocean view you can create a new Pipeline and point it at a source repository. Blue Ocean will look for a Jenkinsfile at the repository root. You can also use the built-in visual Pipeline editor for quick experiments, but writing the Jenkinsfile by hand gives you full version-control history and the most flexibility.

The following example is based on a Python project. It checks out source code, runs the flake8 linter, converts results to JUnit XML format, and publishes them so Blue Ocean can display them as test results. The full example repository is available at https://github.com/robmoorman/jenkins-blueocean.

pipeline {
  agent {
    docker 'python:3.6.1'
  }
  stages {
    stage('Install dependencies') {
      steps {
        sh 'pip install -r requirements.txt'
      }
    }
    stage('Test') {
      steps {
        sh 'flake8 src/ --exit-zero --output-file flake8-output.txt'
        sh 'flake8_junit flake8-output.txt flake8-output.xml'
      }
    }
  }
  post {
    always {
      junit 'flake8-output.xml'
    }
    failure {
      echo 'Failed!'
    }
    success {
      echo 'Done!'
    }
  }
}

Key Pipeline Concepts Explained

Agents

The agent directive tells Jenkins where and how to run a pipeline or an individual stage. Specifying a Docker image as the agent is especially clean: Jenkins pulls the image, runs your steps inside the container, then discards it. This mirrors the approach used by Travis CI and Drone, so teams familiar with those platforms adapt quickly.

You can define different agents per stage, which makes it straightforward to compile in one environment and deploy from another — all within the same Jenkinsfile.

Stages and Steps

Splitting your build into named stages and steps serves two purposes:

• Visibility — Blue Ocean renders each stage as a column in its pipeline view, making it immediately obvious where a build failed.
• Control flow — when a step fails, the pipeline halts and subsequent steps are skipped. You can also attach post blocks to individual stages (for example, always publishing coverage reports after the test stage, regardless of pass/fail).

Conditional Steps with when

The when directive lets you gate a stage on branch name, environment variable, or other conditions. This is particularly useful for deployment stages that should only run on specific branches:

stage('Deploy') {
  when {
    branch 'develop'
  }
  steps {
    sh 'sh ./etc/deploy_test.sh'
  }
}

Combining branch conditions with pull-request detection keeps a single Jenkinsfile as the authoritative source of truth for the entire delivery process — from feature branch lint checks through to production deployment.

Post Actions and Notifications

The post block supports conditional sections including always, success, failure, unstable, and changed. Common uses include:

• Publishing JUnit test results (junit step)
• Sending Slack notifications on build failure
• Archiving build artefacts
• Triggering downstream pipelines

Note that at the time this was written (early 2017, around the first public Blue Ocean release), not all Jenkins plugins had been updated to render correctly within the Blue Ocean UI. The classic Jenkins view remained fully compatible, so plugin incompatibility was an inconvenience rather than a blocker. Plugin compatibility has continued to improve since then — check the Blue Ocean compatibility matrix for current status.

Conclusion

Jenkins Pipelines, with declarative Jenkinsfiles and Docker agents, offer a flexible and version-controlled approach to CI/CD that competes directly with cloud-native platforms. Storing pipeline logic in the repository itself — rather than in Jenkins job configuration — addresses one of the longest-standing pain points of traditional Jenkins setups.

Blue Ocean addresses the usability gap with a modern, stage-oriented interface that makes pipeline status easy to understand at a glance. Plugin ecosystem maturity was a real limitation at the time of Blue Ocean’s initial release, but it should not be treated as a dealbreaker given the pace of development in the Jenkins ecosystem.

If you are evaluating whether to migrate from a cloud CI platform back to self-hosted Jenkins, or simply looking to modernise an existing Jenkins setup, starting with a Dockerised Blue Ocean instance and declarative Pipelines is a low-risk way to assess the workflow before committing fully.

Next steps: Explore shared libraries to make pipeline steps reusable across repositories, and look at the parallel directive for running independent stages concurrently to cut build times.

1. Maintain a Single Source-of-Truth Repository

Every artifact required to build, test, and deploy your application should live in version control. This includes application source code, infrastructure-as-code scripts, database migration scripts, build configuration files, and deployment manifests. Using a system such as Git ensures that any developer or CI server can check out a known revision and produce a deterministic build from scratch, without relying on files that exist only on a local machine or a shared network drive.

Practical tip: Treat your repository as the single source of truth. If it is not in version control, it does not officially exist.

2. Automate the Build

The build process must be fully automated and triggerable by a single command or a CI server webhook. Manual build steps introduce human error and make reproducibility impossible. Whether your toolchain is Make, Gradle, Maven, MSBuild, npm scripts, or a custom shell script, the goal is the same: any engineer — or a headless CI agent — should be able to run the complete build without prior knowledge of local environment quirks.

Practical tip: Store build scripts alongside application code. Avoid build logic that only works inside a specific IDE or on a specific developer’s laptop.

3. Make the Build Self-Testing

A successful compilation is not the same as a working application. Incorporate automated tests — unit tests at a minimum, and integration tests where practical — directly into the build pipeline. The CI server should run these tests on every triggered build and report failures immediately. A build that compiles but ships broken behavior has failed, even if the compiler reported success.

Practical tip: Structure your test suite so that fast unit tests run first. Slower integration or end-to-end tests can run in a subsequent pipeline stage to keep early feedback rapid.

4. Everyone Commits to the Mainline Every Day

Long-lived feature branches are the enemy of CI. The longer a branch lives, the larger the eventual merge conflict and the longer integration problems stay hidden. Encourage every developer to integrate their in-progress work into the mainline branch at least once per working day, using techniques like feature flags or branch-by-abstraction to keep incomplete features invisible to end users while still allowing code to merge continuously.

Practical tip: If a change is too large to commit daily, the task is probably too large. Break it into smaller, independently mergeable increments.

5. Every Commit to the Mainline Triggers a Build

CI servers should be configured to watch the mainline branch and trigger a full build-and-test cycle automatically on every push. If a build fails, the team’s top priority becomes restoring the build to green — either by fixing the defect or reverting the offending commit. A broken mainline blocks every other developer from having confidence in their own integrations.

Practical tip: Define a clear team agreement: whoever breaks the build is responsible for fixing it immediately, not at the end of the sprint.

6. Keep the Build Fast

Feedback latency is the enemy of CI. A build that takes 45 minutes to complete will be ignored; developers will push and move on without waiting for results. Aim to keep the primary build-and-unit-test cycle short — many teams target under 10 minutes as a practical ceiling, though the right number depends on project size. Techniques that help include parallelising test execution, caching dependency downloads, and separating slow integration tests into a later pipeline stage.

Practical tip: Profile your build regularly. Identify the slowest steps and investigate whether they can be parallelised, cached, or moved to a later stage.

7. Make Build Results Visible to Everyone

Build status should not be hidden inside a tool that only the DevOps team monitors. Display the current state of every pipeline — passed, failed, or running — on a shared dashboard visible to the whole team. CI platforms such as Jenkins, GitLab CI/CD, GitHub Actions, CircleCI, and TeamCity all provide build dashboards and notification integrations (email, Slack, Teams) out of the box. Visibility creates shared accountability.

Practical tip: Consider a physical build radiator (a wall-mounted monitor showing CI status) in open-plan offices. The visual signal of a red build creates immediate social pressure to fix it.

Summary

Implementing these practices consistently transforms CI from a tool you have installed into a genuine engineering discipline. The result is a codebase that is always in a releasable state and a team that can detect and fix integration problems in minutes rather than days.

Testing Is No Longer Optional

Not long ago, development teams had to actively argue that software testing was a core part of building software — not a budget-dependent luxury. That battle is largely won. Today it is widely accepted that shipping an application or a feature means delivering code, tests, and documentation together as a single unit.

Open Source projects have operated this way for decades. Platforms like GitHub have generalised those good practices across the broader industry, and hosted CI services such as Travis CI and CircleCI made automated testing more accessible by providing tight version-control integrations that developers could enable with minimal configuration.

What Continuous Integration Actually Means

Continuous Integration (CI) is the practice of using a build server to automatically:

1. Build the software from its source code.
2. Run the full test suite.
3. Report the results back to the development team.

By connecting a CI service to a repository hosted on GitHub or GitLab (which includes built-in CI), developers get near-instant feedback on the health of the codebase. This feedback loop is primarily aimed at developers: it tells them whether their changes can be safely integrated with the current stable version of the software, typically indicated by a green (passing) or red (failing) build status.

In its strictest definition, CI requires developers to push changes to the mainline branch at least once per day, minimising long-running divergent work. In practice, most teams — including the team behind this article — use feature branches and merge them when the build is green rather than on a strict daily schedule.

Feature Branches and the Pull Request Workflow

Feature branches became standard with the rise of Distributed Version Control Systems (DVCS) like Git, and both GitHub and GitLab are built around them. A feature branch is a short-lived branch that encapsulates a single change — a bug fix, a new feature, or a refactor — keeping the mainline stable.

The typical workflow looks like this:

1. Create a new branch from the latest state of the mainline.
2. Develop and commit incrementally.
3. Push the branch to the shared remote repository.
4. Open a Pull Request (GitHub) or Merge Request (GitLab).
5. The CI server runs the build and test suite, updating the branch status.
6. Once the build is green and conflicts are resolved, merge into the mainline.

GitHub and GitLab both surface conflict detection and build status directly in the Pull Request / Merge Request UI, making it easy for the whole team to assess readiness at a glance. Many teams use a simple Git branching model that treats master (or main) as the mainline and uses rebase to keep history clean and avoid unnecessary merge commits.

Organisational Benefits: Ownership and Code Review

Adopting CI with a feature-branch workflow naturally introduces useful team conventions:

• Branch ownership: The developer who opens a feature branch is responsible for it until it is merged. This prevents orphaned branches and unclear accountability.
• Code review: Pull Requests and Merge Requests make peer review a natural part of the workflow. Both GitHub and GitLab provide inline commenting tools that make code review efficient.
• Build health accountability: The developer who breaks the build is responsible for fixing it. This simple rule removes ambiguity and encourages everyone to keep the mainline green.

The result is a mainline branch that can reasonably be treated as the current stable version of the software — buildable and deployable at any time.

From CI to Continuous Delivery and Continuous Deployment

A stable, always-deployable mainline is the prerequisite for the next step: Continuous Delivery (CD).

Continuous Delivery means having the methods, tooling, and processes in place to release software to production at any time — safely, repeatably, and with confidence. It does not mean every commit goes to production automatically; it means every commit could.

Continuous Deployment goes one step further: every change that passes the pipeline is deployed to production automatically, without manual intervention. It is an extension of Continuous Delivery, not a replacement for it.

The key relationship: you cannot achieve reliable Continuous Delivery without Continuous Integration. CI is the foundation on which CD is built.

The Continuous Delivery Feedback Loop

While the CI feedback loop is primarily developer-facing, Continuous Delivery extends the feedback loop to multiple stakeholders:

• Developers learn whether a change behaves as expected in a production-like environment.
• Operations teams get confirmation that the software deploys correctly and that infrastructure is not overloaded.
• Business stakeholders can ship features to users more frequently, shortening the distance between idea and value.
• End users benefit from more frequent, smaller, lower-risk updates.

This multi-stakeholder feedback loop is closely connected to the broader DevOps cultural movement, which emphasises collaboration between development, operations, and quality assurance functions.

Practical CI/CD Setup: Environments and Deployment Scripts

In practice, teams implementing CD typically maintain multiple deployment environments:

• Development (dev): Updated automatically on every merge to the mainline. Used for integration testing.
• Staging: A production-like environment used for pre-release verification with customers or stakeholders.
• Production: The live environment, deployed manually (Continuous Delivery) or automatically (Continuous Deployment) depending on team policy.

Deployment scripts should be kept as simple as possible. A shell script that describes exactly the commands needed to build and deploy the software becomes the single source of truth for the deployment process. That process should also be documented in a README so any team member can understand it without tribal knowledge.

GitLab CI/CD pipelines can expose manual deploy buttons directly in the GitLab UI, allowing any authorised team member to trigger a deployment to a specific environment without SSH access or specialist knowledge. GitHub offers similar capabilities through its Deployments API and integrations with chat-ops tools.

Four Concrete Benefits of This Approach

1. Automated, reproducible, documented deployments: The deployment script is the definitive record of how the software is built and shipped. There is no undocumented manual step hiding in someone’s head.
2. Anyone can deploy at any time: When deployments are automated and exposed through a UI or chat-ops command, deploying is no longer a specialist task. This reduces bottlenecks and single points of failure.
3. Smaller, more frequent releases: Shipping smaller chunks of work more often reduces risk. Each release is easier to reason about, easier to test, and easier to roll back if needed. As Eric S. Raymond put it: release early, release often.
4. Faster bug resolution: Automated tests catch many issues, but real production data surfaces edge cases that tests miss. Frequent deployments mean bugs introduced in production are identified and fixed quickly, because the delta between releases is small. As Edsger Dijkstra observed, testing reveals the presence of bugs, not their absence — so fast feedback from production is invaluable.

Why Start Every Project With CI/CD?

Setting up CI/CD at the start of a project — rather than retrofitting it later — has compounding benefits:

• Quality controls are automated from day one. The team can focus on building value rather than manually verifying that nothing is broken.
• Standards are established early. Feature branches, code reviews, and green-build policies become habits before bad practices take root.
• The project remains accessible to new contributors. Documented, automated pipelines mean a new developer can get productive quickly without needing a lengthy onboarding session from a senior engineer.
• Handoffs are cleaner. When a consulting team or contractor finishes an engagement, a project with working CI/CD and clear deployment scripts is far easier for the client team to take ownership of than one with undocumented manual processes.

The upfront investment — typically a few days to configure pipelines, write deployment scripts, and set up environments — pays dividends throughout the life of the project. It is one of the highest-return early decisions a development team can make.

Summary

Continuous Integration ensures every code change is automatically built and tested, giving developers fast feedback and keeping the mainline stable. Continuous Delivery extends that stability into a reliable, repeatable release process that every stakeholder benefits from. Together, they form the backbone of a healthy engineering culture — reducing risk, improving collaboration, and allowing teams to focus on delivering value rather than managing chaos. Starting with CI/CD is not perfectionism; it is pragmatism.

Further reading: Continuous Integration training resources

Why Integrate Docker with Your CI Tool?

DevOps is fundamentally about moving toward continuous integration (CI) and continuous delivery (CD). Every step in the software lifecycle should be programmatic, repeatable, and automated. That goal is difficult to achieve when development, QA, and operations teams work in silos, each using different environments and toolchains.

Two complementary technologies help break down those silos:

• CI tools (such as Jenkins, CircleCI, or GitLab CI) automate the build, test, and release workflow.
• Docker provides a consistent, portable packaging format so the same container image runs identically across development, test, and production environments.

Combining the two eliminates the classic “works on my machine” problem and lays the groundwork for a reliable CD pipeline. If you plan to graduate from CI to full CD, integrating Docker with your CI tool is a prerequisite, not an optional enhancement.

What Happens When a CI Tool and Docker Work Together?

When your CI tool and Docker are integrated, a typical pipeline executes the following steps automatically whenever a developer pushes new code:

1. Build – The CI tool builds a Docker image from the Dockerfile in your repository.
2. Test – The freshly built image is spun up as a container and your automated test suite runs inside it, ensuring tests execute against the exact same environment that will reach production.
3. Push – If tests pass, the CI tool pushes the verified image to Docker Hub or a private container registry.
4. Deploy – The CI/CD tool pulls the image from the registry and deploys it to the target environment (staging or production), often on a cloud host.

This four-step loop gives every team a single source of truth: the Docker image. Developers, testers, and operations engineers all work with the same artifact, removing environment-specific surprises.

How to Integrate Docker with Popular CI Tools

Jenkins

Jenkins extends its Docker capabilities through a set of plugins. The commonly used ones include:

• Jenkins Cloud Plugin for Docker – Allows Jenkins to spin up Docker containers as dynamic build agents.
• CloudBees Docker Traceability – Tracks which Docker images were used in which builds for audit and compliance purposes.
• CloudBees Docker Hub Notification – Triggers Jenkins jobs when a new image is pushed to Docker Hub.
• CloudBees Docker Build and Publish – Automates building a Docker image from a Dockerfile and publishing it to a registry as part of a Jenkins job.
• CloudBees Docker Pipeline – Provides Docker-aware steps for use inside a Jenkins Pipeline (Jenkinsfile), enabling fine-grained control over container lifecycle within a declarative or scripted pipeline.

Each plugin addresses a distinct stage of the pipeline. A typical Jenkins setup uses several of them in combination to cover build, publish, and deploy phases.

CircleCI

CircleCI takes a configuration-file approach rather than a plugin model. To enable Docker support, you declare it as a required service in your circle.yml configuration file:

machine:
  services:
    - docker

With Docker enabled as a service, you can use standard Docker CLI commands anywhere in your circle.yml build steps, for example:

test:
  override:
    - docker build -t myapp .
    - docker run myapp ./run-tests.sh

deploy:
  override:
    - docker tag myapp myorg/myapp:$CIRCLE_SHA1
    - docker push myorg/myapp:$CIRCLE_SHA1

CircleCI triggers these steps automatically based on events it watches for, such as a push to a specific branch or the opening of a pull request.

Note: CircleCI has since evolved beyond circle.yml to a .circleci/config.yml format with version 2.x and later syntax. The example above reflects the older v1 configuration style described in the original source. Check the current CircleCI documentation for up-to-date syntax.

GitLab CI and Other Tools

GitLab CI integrates with Docker through its .gitlab-ci.yml file and the Docker-in-Docker (dind) service or by running jobs inside Docker containers directly. Most modern CI platforms provide first-class Docker support through similar configuration-file mechanisms, making the integration pattern consistent regardless of the tool you choose.

Key Benefits at a Glance

• Environment consistency – The same Docker image runs in dev, test, and production, eliminating environment drift.
• Faster feedback loops – Automated builds and tests trigger on every commit, catching regressions immediately.
• Repeatable deployments – Immutable container images mean deployments are predictable and rollbacks are straightforward.
• Cross-team collaboration – A shared image registry gives dev, QA, and ops a common artifact to work from.
• Scalable build infrastructure – CI tools can spin Docker containers up and down as ephemeral build agents, optimizing resource usage.

Getting Started

If you are new to this integration, a practical starting point is:

1. Write a Dockerfile for your application if you have not already done so.
2. Choose a CI tool and create a pipeline configuration file (Jenkinsfile, .circleci/config.yml, .gitlab-ci.yml, etc.).
3. Add a build step that runs docker build.
4. Add a test step that runs your test suite inside the built container.
5. Add a publish step that pushes a tagged image to your registry when tests pass.
6. Optionally, add a deploy step that pulls the image to your target environment.

Integrating Docker with your CI tool is one of the highest-leverage steps you can take on the journey to a mature DevOps practice. It brings consistency, automation, and traceability to every stage of your delivery pipeline.

Tier your tests

Not all tests are equal, and you don’t want to treat them as such. I have found 3 tiers to be extremely effective, assuming you’re working on a system that can have some deep complexity to review.

The first tier: “smoke test”. Lots of projects have this concept and it is a good one to optimize for. This is where the 10 minute rule that SemaphoreCI is promoting is a fine rule of thumb. You want to verify as much of the most common paths of functionality that you can, within a time-boxed boundary – you pick the time. This is the same tier that I generally encourage for pull request feedback testing, and I try to include all unit testing into this tier, as well as some functional testing and even integration tests. If these tests don’t pass, I’ve generally assumed that a pull request isn’t acceptable for merge – which makes this tier a quality gate.

I also recommend you have a consistent and clearly documented means of letting a developer run all of these tests themselves, without having to resort to your CI system. The CI system provides a needed “source of truth”, but it behooves you to expose all the detail of what this tier does, and how it does it, so that you don’t run into blocks where a developer can’t reproduce the issue locally to debug and resolve it.

The second tier: “regression tests”. Once you acknowledge that timely feedback is critical and pick a marker, you may start to find some testing scenarios (especially in larger systems) that take longer to validate than the time you’ve allowed. Some you’ll include in the first tier, where you can fit things to the time box you’ve set – but the rest should live somewhere and get run at some point. These are often the corner cases, the regression tests, integration tests, upgrade tests, and so forth. In my experience, running these consistently (or even continuously) is valuable – so this is often the “nightly build & test” sequence. This is the tier that starts “after the fact” feedback, and as you’re building a CI system, you should consider how you want to handle it when something doesn’t pass these tests.

If you’re doing continuous deployment to a web service then I recommend you have this entire set “pass” prior to rolling out software from a staging environment to production. You can batch up commits that have been validated from the first tier, pull them all together, and then only promote to your live service once these have passed.

If you’re developing a service or library that someone else will install and use, then I recommend running these tests continuously on your master or release branch, and if any fail then consider what your process needs to accommodate: Do you want to freeze any additional commits until the tests are fixed? Do you want to revert the offending commits? Or do you open a bug that you consider a “release blocker” that has to be resolved before your next release?

An aside here on “flakes“. The reason I recommend running the second tier of tests on a continuous basis is to keep a close eye on an unfortunate reality of complex systems and testing: Flakey Tests. Flakes an invaluable for feedback, and often a complete pain to track down. These are the tests that “mostly pass”, but don’t always return consistently. As you build into more asynchronous systems, these become more prevalent – from insufficient resources (such as CPU, Disk IO or Network IO on your test systems) to race conditions that only appear periodically. I recommend you take the feedback from this tier seriously, and collect information that allows you to identify flakey tests over time. Flakes can happen at any tier, and are the worst in first tier. When I find a flakey test in the first tier, I evaluate if it should “stop the whole train” – freeze the commits until it’s resolved, or if I should move it into a second tier and open a bug. It’s up to you and your development process, but think about how you want to handle it and have a plan.

The third tier: “deep regression, matrix and performance tests”. You may not always have this tier, but if you have acceptance or release validation that takes an extended amount of time (such as over say a few hours) to complete, then consider shifting it back into another tier. This is also the tier where I tend to handle the (time consuming) and complex matrixes when they apply. In my experience, if you’re testing across some matrix or configurations (be that software or hardware), the resources are generally constrained and the testing scenarios head towards asymptotic in terms of time. As a rule of thumb, I don’t include “release blockers” in this tier – it’s more about thoroughly describing your code. Benchmarks, matrix validations that wouldn’t block a release, and performance characterization all fall into this tier. I recommend if you have this tier that you run it prior to ever major release, and if resources allow on a recurring periodic basis to enable you to build trends of system characterizations.

There’s a good argument for “duration testing” that sort of fits between the second and third tiers. If you have tests where you want to validate a system operating over an extended period of time, where you validate availability, recovery, and system resource consumption – like looking for memory leaks – then you might want to consider failing some of these tests as a release blocker. I’ve generally found that I can find memory leaks within a couple of hours, but if you’re working with a system that will deployed where intervention is prohibitively expensive, then you might want to consider duration tests to validate operation and even chaos-monkey style recover of services over longer periods of time. Slow memory leaks, pernicious deadlocks, and distributed system recovery failures are all types of tests that are immensely valuable, but take a long “wall clock” time to run.

Reporting on your tiers

As you build our your continuous integration environment, take the time and plan and implement reporting for your tiers as well. The first tier is generally easiest – it’s what most CI systems do with annotating in pull requests. The second and third third take more time and resources. You want to watch for flakey tests, collecting and analyzing failures. More complex open source systems look towards their release process to help coral this information – OpenStack uses Zuul (unfortunately rather OpenStack specific, but they’re apparently working on that), Kubernetes has Gubernator and TestGrid. When I was at Nebula, we invested in a service that collected and collated test results across all our tiers and reported not just the success/failure of the latest run but also a history of success failure to help us spot those flakes I mentioned above.

Overview

Continuous Integration (CI) for Salesforce metadata differs from traditional software CI because deployments target a cloud platform rather than a compiled binary. The Salesforce Metadata API, exposed through the ANT Migration Toolkit, is the standard mechanism for scripted deployments. By combining TFS (or its cloud-hosted equivalent, Visual Studio Team Services) with Apache ANT, teams can trigger automated retrieve-and-deploy cycles on every code commit or on a schedule.

This guide uses the cloud-hosted version of TFS (Visual Studio Team Services) to avoid the overhead of maintaining a local TFS installation, but the same build definition steps apply to an on-premises TFS server.

Prerequisites

• A Microsoft account (used to sign in to Visual Studio Team Services)
• Apache ANT installed on your local machine for local testing
• The Salesforce ANT Migration Toolkit (ant-salesforce.jar), downloaded from your Salesforce org’s Setup page
• A Git client (for example, TortoiseGit) for pushing code to TFS
• Credentials for the source and target Salesforce orgs

Step 1: Create a Developer Account on Visual Studio Team Services

Navigate to https://www.visualstudio.com/ and choose Get Started for free under the Visual Studio Team Services section. If you do not already have a Microsoft developer account, you will be prompted to create one during sign-up. Once registered, you will land in the VSTS dashboard where you can create your first Team Project.

Step 2: Create a TFS Git Repository for Your Salesforce Metadata

One advantage TFS has over Jenkins-based CI setups is that source control is built in — you do not need a separate GitHub or Bitbucket account. After logging in, use the New Project wizard to create a project. The wizard is straightforward: provide a project name, choose a version control type (select Git), and choose a process template.

Once the project exists, clone the repository to your local machine using any Git client. For authentication you can use your VSTS username and password, or generate an SSH key pair and register the public key in your VSTS profile under Security > SSH public keys. SSH key authentication is recommended for CI agents because it avoids storing plain-text passwords.

Step 3: Create and Test the ANT Deployment Script Locally

Before wiring ANT into TFS, verify that the scripts work correctly on your local workstation. You need two files in the root of your repository: build.properties (credentials and environment URLs) and build.xml (the ANT targets).

build.properties

This file stores environment-specific values. Keep it in version control for non-sensitive settings, but consider externalising passwords as TFS build variables or secrets so that plain-text credentials are never committed to source control.

# build.properties

# Server URLs
sfSandbox.serverurl = https://test.salesforce.com
sfPRO.serverurl    = https://login.salesforce.com

# Source org (retrieve)
sfdc.cb.df14.username        = your-source-org@example.com
sfdc.cb.df14.password        = yourPasswordPlusSecurityToken
sfdc.cb.df14.retrieveTarget  = src
sfdc.cb.df14.unpackaged      = src/package.xml
sfdc.cb.df14.retrieveMessage = Retrieve metadata from source Salesforce org

# Target org (deploy)
sfdc.cb.df15.username        = your-target-org@example.com
sfdc.cb.df15.password        = yourPasswordPlusSecurityToken
sfdc.cb.df15.retrieveTarget  = src
sfdc.cb.df15.unpackaged      = src/package.xml
sfdc.cb.df15.retrieveMessage = Deploy metadata to target Salesforce org

# Salesforce ANT JAR — path relative to repository root
sfdcJarFile = ant-salesforce.jar

build.xml

The build.xml file defines two ANT targets: df14 retrieves metadata from the source org, and deploy pushes that metadata to the target org. Notice that classpath points to ${sfdcJarFile}, which resolves to a path relative to the repository root. This is essential for cloud CI agents that have no pre-installed copy of the Salesforce JAR, so the JAR must be committed to the repository alongside the build scripts.

<project name="Deployment Scripts" default="deploy" basedir="." xmlns:sf="antlib:com.salesforce">

    <property file="build.properties"/>

    <!-- Retrieve metadata from source org -->
    <target name="df14">
        <taskdef resource="com/salesforce/antlib.xml"
                 uri="antlib:com.salesforce"
                 classpath="${sfdcJarFile}"/>

        <echo message="Retrieving as: ${sfdc.cb.df14.username}"/>

        <sf:retrieve
            username="${sfdc.cb.df14.username}"
            password="${sfdc.cb.df14.password}"
            serverurl="${sfPRO.serverurl}"
            retrieveTarget="${sfdc.cb.df14.retrieveTarget}"
            unpackaged="${sfdc.cb.df14.unpackaged}"/>
    </target>

    <!-- Deploy metadata to target org -->
    <target name="deploy">
        <taskdef resource="com/salesforce/antlib.xml"
                 uri="antlib:com.salesforce"
                 classpath="${sfdcJarFile}"/>

        <sf:deploy
            username="${sfdc.cb.df15.username}"
            password="${sfdc.cb.df15.password}"
            serverurl="${sfPRO.serverurl}"
            deployRoot="src"
            maxPoll="1000"
            testLevel="NoTestRun"
            checkOnly="false"
            pollWaitMillis="10000"
            rollbackOnError="true"/>
    </target>

</project>

To test locally, run the following command from the repository root:

ant df14 deploy

The df14 target retrieves metadata from the source Salesforce instance; the deploy target then pushes that retrieved metadata to the target instance. Confirm that both targets complete without errors before moving to the TFS build definition.

Files to commit to the repository

• build.xml
• build.properties (with passwords replaced by TFS build variables in production)
• ant-salesforce.jar — the Salesforce Migration Toolkit JAR, committed so the TFS cloud agent can find it
• src/package.xml — your metadata manifest
• src/ — the retrieved metadata directory

Step 4: Create a New Build Definition in Team Foundation Server

With the scripts in the repository, you can now tell TFS how to run them automatically.

In the VSTS portal, navigate to Build & Release → Builds → New definition and select the ANT template.

Select the repository and branch

On the repository selection screen, choose your TFS-hosted Git repository. You can also connect to GitHub, a remote Git server, or SVN if your code lives elsewhere. Select the branch you want this build definition to monitor. TFS supports multiple build definitions per project, so you can configure, for example, a nightly build on main and a triggered build on a develop branch independently.

Configure the ANT build step

In the build steps editor, configure the ANT task:

• ANT build file: build.xml
• Options: specify the targets to run, for example df14 deploy, or leave blank to use the default target defined in build.xml
• JDK version: ensure a compatible JDK is selected on the build agent

Save the build definition. You can trigger it manually the first time using the Queue build button to confirm everything works end to end before enabling a continuous integration trigger.

Security Considerations

Storing Salesforce credentials in build.properties and committing that file to source control is convenient for demonstration purposes but is a security risk in production. Instead:

• Define Salesforce usernames and passwords as TFS/VSTS build variables marked as secret. Reference them in build.xml using ANT property overrides passed via the build options field (e.g., -Dsfdc.cb.df15.password=$(SF_PASSWORD)).
• Use a connected app with OAuth and a session ID instead of username/password where your tooling supports it.
• Consider whether committing ant-salesforce.jar to Git is appropriate for your organisation; an alternative is to download it as an artifact during the build using a script step.

Summary

Setting up Salesforce CI with TFS involves four main steps: creating a VSTS account and project, pushing your Salesforce metadata and ANT scripts (including the Salesforce JAR) to the built-in Git repository, validating the ANT script locally, and then creating an ANT build definition in TFS that executes the same script on every commit or on a schedule. The integrated source control in TFS/VSTS reduces the number of third-party tools required compared to a Jenkins-based approach, making it a practical choice for teams already in the Microsoft ecosystem.

Original source: jitendrazaa.com

One of the famous tools used for CI testing is Jenkins.

“Jenkins is an open source automation tool, which can be used to program all categories of tasks such as building, testing, and deploying software. Jenkins can be installed through native system packages, Docker, or even run standalone by any machine with the Java Run-time Environment installed.”

It helps to automate the non-human part of the software development life-cycle.

Jenkin pipeline is a package of plugins used for implementing the CI process in the delivery pipeline. Extensive set of tools are available with pipeline for modelling simple to complex delivery pipelines. It is a server based tool, which runs on Java servlet such as Apache Tomcat. It supports source control management such as AccuRev, CVS, Git, Mercurial, Subversion, RTC, Perforce and Clearcase.

There are a lot of plugins released for Jenkins to extend its use to projects, which are written in various languages besides Java.

Below is the flow diagram to explain how continuous integration testing can be done with Jenkins.

Now let’s take a look how to configure and manage Jenkins

Step 1

Download Jenkins from the website https://jenkins.io

Step 2

Login to the Jenkins

E-Mail Configuration:

Step 3

Click on ‘Manage Jenkins’ link

Step 4

Click on ‘Configure System’ link

Step 5

Configure the fields given as screenshot

Step 6

Enter ‘System Admin’ Email Id

Step 7

Enter the following fields on the ‘Extended Email Notification section –Ref’ screenshot

SMTP Server: smtp.gmail.com

Default user E-mail suffix: @gmail.com

Default Recipients: your123@gmail.com

Reply To List: noreply@gmail.com

Default Content Type: $PROJECT_NAME – Build # $BUILD_NUMBER – $BUILD_STATUS!

Step 8

Select the checkbox option given on the screen shot to Trigger the Email

Step 9

Enter the fields on the ‘E-mail Notification’ section as given on the screenshot

Select ‘Test Configuration’ checkbox then click on ‘Test Configuration’ button. You will get success message like – “Email was successfully sent” [if you get this message means you have configured successfully] then Click on ‘Save’ button.

Steps to create a new Project in Jenkins and Project Configuration

Step 1

Go to Jenkins -> New Item

Step 2

Enter project name and select ‘Freestyle Project’ then click on ‘OK’ button

Step 3

Click on tab ‘General’

Step 4

Set the time on ‘Build Trigger’ Section

Example Time:

# every fifteen minutes (perhaps at :07, :22, :37, :52)

H/15 * * * *

# every ten minutes in the first half of every hour (three times, perhaps at :04, :14, :24)

H(0-29)/10 * * * *

# once every two hours at 45 minutes past the hour starting at 9:45 AM and finishing at 3:45 PM every weekday.

45 9-16/2 * * 1-5

# once in every two hours slot between 9 AM and 5 PM every weekday (perhaps at 10:38 AM, 12:38 PM, 2:38 PM, 4:38 PM)

H H(9-16)/2 * * 1-5

# once a day on the 1st and 15th of every month except December

H H 1,15 1-11 *

Step 5

Enter the name of your .bat file on the ‘Build’ section

Step 6

Fill the field on the ‘Post-build Actions’ section given as screenshot

Step 7

Click on ‘Save’ button

Step 8

Click on your Project then click -> Build Now; you will get the ‘Build Executor Status’ as following screenshot.

Step 9

Once if the build get completed click on the build number then view the console output

Step 10

The console output will be following screenshot

Why Migrate from Jenkins to GitLab CI?

Jenkins has long been a staple of continuous integration pipelines, but as teams grow and infrastructure scales, its centralized architecture can become a bottleneck. Team Alfonzó, working alongside their Build Team, decided to explore a more distributed build infrastructure. Their Implementation Wizard project became the test bed for adopting GitLab CI and the GitLab Container Registry — replacing both Jenkins and DockerHub in their workflow.

GitLab CI vs. Jenkins: Pipeline Configuration Compared

If you have written a Jenkinsfile using Groovy-based DSL, you will likely find GitLab CI’s approach cleaner and more immediately readable. Instead of writing pipeline logic in Groovy, GitLab CI uses a .gitlab-ci.yml file checked into the root of your repository. In this YAML file you declare your build stages and the shell scripts that run at each stage.

Example .gitlab-ci.yml Structure

stages:
  - build
  - test
  - deploy

build-job:
  stage: build
  script:
    - docker build -t my-app:$CI_COMMIT_SHA .

test-job:
  stage: test
  script:
    - docker run my-app:$CI_COMMIT_SHA ./run-tests.sh

deploy-job:
  stage: deploy
  script:
    - kubectl apply -f deployment.yaml
  only:
    - main

The YAML syntax enforces a sensible structure while still giving you enough flexibility to configure each stage independently. The tradeoff is that there is no built-in way to test or lint your pipeline scripts locally without committing and pushing changes to the repository. Teams should be aware of this limitation early on to avoid cluttering the commit history with purely experimental CI changes. (Third-party tools such as gitlab-ci-local have since emerged to help with local pipeline testing, though they were not part of this original migration.)

Setting Up a GitLab Runner

A .gitlab-ci.yml file alone does nothing without a GitLab Runner — the agent that picks up jobs and executes them. Team Alfonzó registered a dedicated runner running inside a Docker container. This gave the team full control over the runner environment, though it comes with a resource cost compared to using shared runners provided by GitLab.

If multiple teams share a codebase or similar build requirements, shared runners are worth considering as a more resource-efficient option. Dedicated runners make the most sense when you need a specific environment, custom tooling, or tighter security boundaries.

Bypassing Docker-in-Docker Difficulties

A common challenge when running Docker-based CI pipelines is the need to build Docker images from within a Docker container — a pattern known as Docker-in-Docker (DinD). GitLab officially supports DinD using the docker:dind service image, but in Team Alfonzó’s environment this approach did not work reliably.

The Unix Socket Workaround

To work around the DinD limitation, the team configured the Docker API to share the same Unix socket between the runner container and the stage container (the container that actually executes build steps). This is done by mounting the host Docker socket into the runner container:

# When registering the GitLab Runner, pass the socket mount:
docker run -d \
  --name gitlab-runner \
  -v /var/run/docker.sock:/var/run/docker.sock \
  -v /srv/gitlab-runner/config:/etc/gitlab-runner \
  gitlab/gitlab-runner:latest

By mounting /var/run/docker.sock from the host into the runner container, and then propagating that same socket mount into stage containers, both containers communicate with the same Docker daemon. This sidesteps the need to run a nested Docker daemon entirely.

Security note: Mounting the Docker socket grants the container root-equivalent access to the host. This approach is acceptable in controlled, trusted CI environments but should be carefully evaluated before use in multi-tenant or public-facing setups.

Data Volumes and Kubernetes Deployment

Even with the socket sharing workaround in place, the team found that Docker containers running build stages could not accept data volumes mounted from outside. This created a specific problem during Kubernetes deployments: the deployment manifest file could not be injected via a volume mount into the stage container.

The solution was to use kubectl‘s ability to read configuration from standard input rather than from a file path:

# Pass a Kubernetes manifest via stdin instead of a file
cat deployment.yaml | kubectl apply -f -

This pattern works reliably when volume mounts are unavailable and is a useful technique in any containerized CI environment where file system sharing between containers is restricted.

Hosting Docker Images with the GitLab Container Registry

GitLab includes an integrated Container Registry that can host Docker images built from your projects. For Team Alfonzó, this proved more convenient than DockerHub for several reasons:

• Granular access control: Registry permissions follow the existing GitLab project permissions model, meaning developers already have appropriately scoped access without a separate admin workflow.
• Reduced friction: No need to manage separate DockerHub credentials or request registry creation from a small group of administrators.
• Tighter integration: Images built and pushed during a CI pipeline are immediately visible and accessible within the same GitLab project.

A typical pipeline stage to build and push an image to the GitLab Registry looks like this:

build-and-push:
  stage: build
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

GitLab automatically provides the predefined CI variables $CI_REGISTRY, $CI_REGISTRY_USER, $CI_REGISTRY_PASSWORD, and $CI_REGISTRY_IMAGE, so no manual credential management is needed for standard project pipelines.

Known Limitation: One Registry per Project

At the time of this migration, the team noted that GitLab’s registry did not support hosting multiple distinct images under a single project — a limitation that surfaced particularly for projects with external dependencies. This issue was reported to the GitLab team. GitLab’s registry has evolved significantly since 2017, so teams evaluating it today should consult the current GitLab documentation for the latest capabilities.

Conclusion

GitLab CI offers a compelling alternative to Jenkins for teams already using GitLab for source control. The YAML-based pipeline configuration is readable and maintainable, the integrated Container Registry removes the need for a separate image hosting service, and the GitLab Runner architecture supports both shared and dedicated execution environments.

The main challenges encountered — Docker-in-Docker limitations and restricted data volume access in nested containers — each have workable solutions. The Unix socket mount approach and the use of kubectl stdin input are practical patterns that apply well beyond this specific migration scenario.

For teams experiencing high load on a shared Jenkins instance or looking to consolidate their toolchain, GitLab CI is a mature, well-integrated option worth evaluating.

Testing

Run Unit Tests Against a Reconstituted QA Database

A common approach for application code is to run unit tests at the point of each commit. For databases, a more reliable pattern is to run the full unit test suite sequentially against a fully reconstituted QA database as a dedicated Test step in the CI pipeline — rather than against an individual developer’s local environment.

This introduces a small lag compared to per-commit testing, but frequent CI iterations largely close that gap. The key advantage is that unit tests execute against a known, reproducible database state rather than a developer’s potentially dirty working copy.

Tools such as ApexSQL Unit Test integrate with the tSQLt framework to author and execute SQL Server unit tests from the command line, making them suitable for inclusion in any automated pipeline.

Keep Unit Tests and Test Scripts in Source Control

Test scripts are first-class artifacts and must be version-controlled alongside the database objects they test. Store them in a dedicated branch or folder (e.g., /tests) within your repository to prevent them from being accidentally deployed to production. The Build step should pull the correct revision of both the schema scripts and the test scripts before executing either.

This approach guarantees that every change to a test is tracked, reviewable, and automatically included in the next build without manual intervention.

Never Use Production Data for Testing

Pulling a copy of production data into a QA or CI environment is tempting as a shortcut, but it creates two significant problems:

• Pipeline direction violation. A well-structured CI/CD pipeline flows in one direction — from a clean baseline toward production. Reading from production to seed a lower environment reverses this flow and introduces an uncontrolled dependency on production state.
• Privacy and compliance risk. Real customer or business data exposed to developers, contractors, or CI agents can violate data-protection regulations and internal security policies.

The correct approach is to generate synthetic test data programmatically. A data generation tool or script should be able to populate transactional tables with configurable volumes of randomized, realistic data on demand. Static reference data (lookup tables, enumeration tables) should be seeded from source control as part of the Build step. Tools such as ApexSQL Generate are purpose-built for this Populate step, but a well-crafted T-SQL or PowerShell script can serve the same role.

Unit Tests Must Return a Machine-Readable Pass/Fail Result

A test run that produces a human-readable HTML summary is not useful in an automated pipeline. Every test execution must return a numeric exit/return code that the calling process can evaluate without string parsing. A non-zero return code on failure allows the pipeline orchestrator (Jenkins, Azure DevOps, TeamCity, a PowerShell script, etc.) to abort subsequent steps immediately.

When selecting or writing a test runner, verify that it surfaces OS-level exit codes — not just log files or report files.

Define a Failure Threshold, Not a 100% Pass Requirement

Requiring 100% of all tests to pass before a build is considered green may seem like a high standard, but in practice it forces manual review of results on every single test failure — defeating the purpose of automation. A more practical approach uses configurable thresholds:

• Fail the build if fewer than X% of tests pass (e.g., 80%).
• Optionally, always fail on specific high-priority or smoke tests regardless of the overall pass rate.

This allows minor, known-flaky tests to be tolerated temporarily without blocking the pipeline, while still catching regressions that cross a meaningful threshold.

Processing

Every Pipeline Step Must Return a Clear Success or Failure

This applies to every step in the pipeline — schema build, static analysis, data population, unit tests, integration tests. Each step must emit a machine-consumable result (typically an OS exit code) so the orchestrator can decide whether to proceed or abort. Steps that only log output and always exit with code 0 are pipeline liabilities.

Audit each tool and script in your pipeline against this criterion before integrating it.

Abort the Pipeline on Any Failure — Do Not Continue Past a Broken Step

Database CI pipelines can involve many sequential steps, and if parallel pipelines are spawned (e.g., one for unit tests, one for performance/load tests, one for policy compliance checks), total execution time can be substantial. Allowing a pipeline to continue past a known failure wastes compute resources and produces misleading results downstream.

Configure your orchestrator to fail fast: abort and notify on the first step that returns a failure code. Immediate email or messaging notification to the team should be triggered at this point, including as much diagnostic detail as the failing step can provide.

Implement a Circuit Breaker to Block Subsequent Builds After a Failure

A failed build that is not fixed before the next scheduled run will cause every subsequent run to fail as well, consuming resources and filling notification queues with noise. Implement a circuit-breaker pattern:

1. On failure, set a flag (e.g., a value in a configuration table or a flag file) that prevents the CI job from starting.
2. A manual trigger can bypass the circuit breaker to allow a hotfix run.
3. A successful completion automatically resets the flag.

This is a lightweight but effective safeguard against runaway failures in unattended overnight pipelines.

Send Targeted Notifications at Pipeline Start, Completion, and Failure

Notification strategy matters. Over-notifying desensitizes the team; under-notifying leaves failures unaddressed. A practical model:

• Pipeline start: Notify QA only (so they know the current build may be temporarily unavailable or in flux).
• Successful completion: Notify QA only (so they can begin testing the new build).
• Failure: Notify the full team — developers, QA, and manager — with as much detail about the failing step as possible.

Where the pipeline environment supports it, build QA environments as a new environment rather than overwriting the existing one, so QA is never completely blocked during a build run.

Build Frequency Should Not Exceed Build Duration

Do not trigger a new build more frequently than it takes for the previous build to complete. If a full pipeline run takes 45 minutes, the minimum build interval should be 45 minutes. Commits arriving during a build window should be queued and incorporated into the next build rather than triggering an overlapping run.

This avoids pipeline collisions, prevents resource contention on the CI agent and database server, and eliminates arbitrary check-in blackout windows for developers.

The CI Process Must Support Three Execution Modes

1. Manual trigger — for hotfix validation, ad-hoc testing, or circuit-breaker bypass.
2. Scheduled trigger — time-based (e.g., every hour, nightly) for low-commit-frequency environments.
3. Commit trigger — initiated by a source-control webhook or polling, subject to the minimum-interval governor described above.

Automation

Never Store Plain-Text Passwords in Automation Scripts

Credentials embedded as plain text in PowerShell scripts, batch files, or YAML pipeline definitions are a serious security risk — particularly when scripts are stored in source control. Use one of the following approaches depending on your environment:

1. Windows Authentication / Integrated Security — the preferred option for SQL Server connections running under a domain service account. No password appears anywhere in the script.
2. Encrypted credentials — use ConvertTo-SecureString / ConvertFrom-SecureString in PowerShell, or a secrets manager (Azure Key Vault, HashiCorp Vault, AWS Secrets Manager) to retrieve credentials at runtime.
3. Interactive prompt for attended runs — use Get-Credential in PowerShell or equivalent for scripts that a human operator launches manually.

For unattended overnight jobs, option 1 or option 2 is required. Many CI tool project files (including those produced by ApexSQL tooling) store passwords in encrypted form, but always verify the encryption mechanism meets your organization’s security requirements.

Putting It All Together: A Reference Pipeline Structure

A SQL Server database CI pipeline that follows these practices will typically look like this:

1. Trigger — commit webhook, schedule, or manual invocation.
2. Circuit breaker check — abort if the previous build is still broken.
3. Notify — alert QA that a build is starting.
4. Source pull — check out schema scripts, static data scripts, and test scripts at the current revision.
5. Build — deploy schema and static data to a clean QA database instance.
6. Populate — generate synthetic transactional test data.
7. Test — execute unit tests; abort and notify on failure or threshold breach.
8. Static analysis (optional) — run policy/rules checks; abort on failure.
9. Notify — alert QA on success, or alert full team with diagnostics on failure.
10. Reset circuit breaker — clear the failure flag on success.

Each step emits an exit code. The orchestrator evaluates it and either proceeds or aborts. No step requires human interpretation to continue.

Summary

Implementing CI for SQL Server databases requires deliberate decisions about where tests run, what data they run against, how results are communicated, and how the pipeline behaves under failure. The key principles are: test against a reproducible QA environment, use synthetic data, make every step machine-readable, fail fast, protect subsequent builds with a circuit breaker, and never embed plain-text credentials. These practices apply regardless of the specific tooling your team chooses.

Year after year, we bear witness to the increasingly rapid evolution of the IT industry. It has now been more than two decades since the groundbreaking slogan ”Write once, run anywhere“ set a whole new level of expectation for the software development community. And here we are today, with a resulting, ever-expanding set of tools that have collectively taken Java development in particular, and software development in general, to a whole new universe of possibility.

Methodologies such as Agile, DevOps, and Continuous Integration and Deployment – along with the evolution of microservices – have collectively boosted software development process productivity to a point where it is a pleasure to be developing software more than ever before.

Utilizing automation and setting up the correct set of tools can make the development and delivery of software products surprisingly painless.

This article will take a look at this new universe from the perspective of a Java developer who crosses into DevOps and searches to optimize product development and delivery to its maximum.

Today, terms such as Spring Boot, Docker, Cloud, Amazon Web Services, Continuous Delivery are widely used but less widely understood. This article will take the easiest route possible to present all these technologies and explain these terms, and wrap it up in the form of a tutorial where we will develop a small piece of software and prepare it for production delivery using all the mentioned tools.

Why These Tools?

Simplify Deployments with Docker

”Write once, run anywhere“ was the conceptual breakthrough that yielded technologies like the Java Virtual Machine (JVM) which enabled your code to run anywhere. And now here we are, a couple of decades later, with something called Docker being presented to the IT community. Docker is a containment tool in which you can place your software and run it painlessly, almost anywhere you wish.

However, a Java developer may look at Docker and say “Why would we need that, we already have the JVM, which is well recognized as the master portable solution.” But is it?

”Write once, run anywhere“ sounds nice, and plays out well… at least most of the time. Until you encounter multiple JVM vendors, multiple Java versions, multiple operating systems, and various permutations and combinations of all the above. You then find yourself switching from the elegant ”Write once, run anywhere“ paradigm to the counter-productive ”Write once, debug everywhere“ pitfall.

And that’s where Docker comes in to help save the day.

Docker simplifies development, testing, and shipping of software. If you have the software you want to test, put it in the Docker container, and it will run and be painless to install for all the parties involved.

Speed Up Development with Spring Boot

Less than a decade after the “run anywhere” slogan was introduced, the Spring framework appeared on the scene. Today, the Spring ecosystem continues to flourish, and has produced lots of valuable Spring-based projects, perhaps most notably Spring Boot. As stated on the Spring Boot website:

Spring Boot makes it easy to create stand-alone, production-grade Spring based Applications that you can “just run”.

Spring Boot enables you to bring the application up and running in a matter of minutes. Software developers can focus on software development and then can benefit from a tool that does all the configuration for them.

In this tutorial, we will use Spring Boot to develop our microservice.

Continuous Integration (CI) with Jenkins

DevOps is a rapidly growing movement that is tightly integrating software development and systems administration teams, with the goal of making a software development and delivery lifecycle as painless, seamless, and productive as possible for all parties involved: developers, sysadmins, testers, and ultimately, end users.

Continuous integration (CI), is one of the cornerstones of the DevOps revolution. The idea is that whenever a developer commits code to the code repository, it is automatically tested and packaged for delivery (deployment) to production.

CI goes hand-in-hand with:

• Continuous Delivery – Automatic delivery of the package prepared for end user business testing with manual trigger to production deployment.
• Continuous Deployment – Automatic deployment of the packaged product directly to production.

More than a few tools exist which can be utilized for implementing the CI process. One of the most popular is Jenkins, an open source CI tool. With more than a thousand plugins and a huge community behind it, Jenkins is an easy choice when starting to think about implementing continuous integration, delivery, or deployment.

In our tutorial, Jenkins will be used to deliver our product to the cloud, more specifically, the Amazon (AWS) cloud.

Cloud Computing with AWS

If you have some sysadmin experience, imagine removing some of the worries of system administration from your shoulders. You have a few applications; you have an idea of how much resource they will require, but you don’t know exactly the hardware sizing you will need. You make the estimation, the resources are bought, and the system goes to production. If you are lucky, you will find that you overestimated and have more resources than you need. But given Murphy’s Law, you will more likely find that you underestimated resource requirements and end up scrambling to get a bit more memory or processing power under tremendous time pressure. In contrast, if you’re deploying to the cloud, you simply put your system out there and size it as needed, with the flexibility offered by the cloud providers. With the cloud, you neither need to worry about running out of system resources, nor do you need to worry about having 90 percent of your memory or CPU sitting idle.

Of course, there is the challenge of deciding which provider to choose. Cloud wars are still in progress. Clash of Microsoft, Amazon and Google for the future of computing is an example title you can find lately in tech-world news. For this blog, I have chosen Amazon Web Services (AWS), largely based on its current popularity and market share.

One of the advantages of AWS is that Amazon offers lots of services after you sign up

In this tutorial we will use the following two AWS services: Elastic Compute Cloud EC2 (more specifically, Amazon EC2 Container Registry, or Amazon ECR), and Amazon S3 (Simple Storage Services).

Amazon ECR

We will need to store our Docker images somewhere. Amazon ECR is a managed AWS Docker registry service. As stated on the Amazon ECR web site:

…makes it easy for developers to store, manage, and deploy Docker container images. Amazon ECR is integrated with Amazon EC2 Container Service (ECS), simplifying your development to production workflow. Amazon ECR eliminates the need to operate your own container repositories or worry about scaling the underlying infrastructure.

Amazon S3

As mentioned, the application we develop will be a Spring Boot microservice that will be uploading files to Amazon S3. As stated on the Amazon S3 web site:

…provides developers and IT teams with secure, durable, highly-scalable cloud storage. Amazon S3 is easy to use object storage, with a simple web service interface to store and retrieve any amount of data from anywhere on the web.

A Practical “How To” Tutorial

The goal is to prepare for deployment a Spring Boot microservice which will upload files to Amazon S3. Steps are the following:

• Develop the microservice
• Define the build process in which the service will be dockerized
• Use Bitbucket for hosting the Git code repository
• Integrate Bitbucket with Jenkins to package the application using Gradle
• Push it to a remote Amazon ECR

What follows is a tutorial for setting up all the needed components:

• Spring Boot example application – microservice packaged and dockerized using Gradle
• Jenkins installation on a fresh Ubuntu server
• Bitbucket integration with Jenkins via webhook
• Jenkins job configuration
• Amazon ECR to store the Docker images containing our application

Prerequisites

To be able to use AWS cloud resources, we need to register at Amazon first. By registering, we will get an account with immediate Free Tier usage benefits, for purposes of enabling hands-on experience during the 12 months following registration.

As mentioned, in this tutorial we will use Amazon S3 and Amazon ECR. For both, we will need access keys to connect to the services.

After signing up with AWS, we go to our account Security credentials, where we choose Access keys and click on “Create New Access Key”. After clicking, a key is generated along with its ID. You need to store this somewhere safe, as we will use it later when configuring AWS Jenkins integration and developing our S3 file upload.

The next prerequisite is that we need an Amazon S3 bucket (storage container). Our Spring Boot Service will be uploading and downloading files to and from Amazon S3 storage. Bucket creation is simple enough and just requires few clicks. A full description of how to do it is provided in the Create a Bucket documentation.

We will also be using Bitbucket for hosting our code and triggering requests to Jenkins, so a Bitbucket account is needed as well. Bitbucket is a great option for developers, with one of its main benefits being the unlimited amount of private repositories you can create.

Application Development

Rather than getting into all the details of Spring annotations and how they work, I will instead focus, from a pure developer perspective, on the more challenging part of the whole setup; namely, installing and configuring Linux, Jenkins, and other tools needed for CI. All the code examples used in this tutorial, including the Spring Boot microservice application, are available on the Bickbucket repository for the project.

Our application composition is simple. We have a Spring Boot application entry point in our StorageWebserviceApplication.java file. The logic for uploading and downloading files is in StorageService.java. StorageController.java is a Rest controller, containing API endpoints used for file upload and download. Here is the project hierarchy:

We have chosen Gradle as a build tool, and it will package our application and compose the Docker image. So next, we will discuss the Gradle build file, service component, and the Dockerfile.

To be able to use AWS API, we need to include dependencies in our build file, as defined in the AWS documentation for using Gradle.

Summed up, the AWS dependency configuration part of our Gradle script will look like the following:

buildscript {
   	   	...
   	repositories {
         	mavenCentral()
   	}
   	dependencies {
...
         	classpath("io.spring.gradle:dependency-management-plugin:0.5.4.RELEASE")
   	}
}
..
apply plugin: "io.spring.dependency-management"
 
dependencyManagement {
	imports {
    	mavenBom ('com.amazonaws:aws-java-sdk-bom:1.10.47')
	}
}
 
dependencies {
   	..
   	compile ('com.amazonaws:aws-java-sdk-s3')
}

As stated previously, when uploading files to Amazon S3, we do so by uploading files to an S3 bucket.

To connect to the bucket, our Amazon S3 client needs to have credentials provided. Credentials are the access keys we created earlier. We define the access key ID and value in the application.properties file; we have named our bucket toptal-s3-example.

Our main service component is now as follows:

@Service
public class StorageService {
 
  	@Value("${aws.accesKeyId}")
  	private String awsAccessKeyId;
 
  	@Value("${aws.secretKey}")
  	private String awsSecretKey;
 
  	@Value("${aws.bucketName}")
  	private String awsBucketName;
  	
  	 private AWSCredentials credentials;
  	 private AmazonS3 s3client;;
 
  	@PostConstruct
	public void init(){
        	 credentials = new BasicAWSCredentials(awsAccessKeyId, awsSecretKey);
        	 s3client = new AmazonS3Client(credentials);
	}
 
 
  	public void uploadFile(MultipartFile file) throws IOException {
        	File fileForUpload = transformMultipartToFile(file);
        	s3client.putObject(new PutObjectRequest(awsBucketName, file.getOriginalFilename(), fileForUpload));
  	}
  	
  	public InputStream downloadFile(String amazonFileKey) throws IOException {
        	S3Object fetchFile = s3client.getObject(new GetObjectRequest(awsBucketName, amazonFileKey));
        	InputStream objectData = fetchFile.getObjectContent();
        	return objectData;
  	}
…

StorageService reads the credentials from the application.properties file and uses them to instantiate BasicAWSCredentials object, and subsequently the AmazonS3Client object. What follows is a simple matter of invoking putObject for file upload and getObject for file download, on the Amazon S3 client object.

We will run the service inside a Docker container and, during the Gradle build process, we will build the Docker image. We will do this by additionally configuring the build.gradle file, as follows:

buildscript {
   	...
   	dependencies {
         	...
         	classpath('se.transmode.gradle:gradle-docker:1.2')
   	}
}
 
.....
apply plugin: 'docker'
...
task buildDocker(type: Docker, dependsOn: build) {
 push = false
 applicationName = "storageservice"
 dockerfile = file('src/main/docker/Dockerfile')
 doFirst {
   	copy {
   	from jar
   	into stageDir
   	}
 }
}

The Buildscript part and apply plugin are pretty standard. We have also defined a buildDocker task which reads the Docker configuration stored in the src/main/docker/Dockerfile, and copies the JAR file to the Docker build.

Dockerfile contains a list of pure Docker commands with which we will prepare our image:

FROM frolvlad/alpine-oraclejdk8
ADD storageWebService-0.0.1-SNAPSHOT.jar storageService.jar
EXPOSE 8080
CMD ["java", "-Djava.security.egd=file:/dev/./urandom", "-jar", "/storageService.jar"]

A prerequisite for running our application is to have installed a Java Virtual Machine (JVM). Docker provides a list of images with Java installed, and we will choose one of the smallest, based on a minimal 5MB Alpine Linux. frolvlad/alpine-oraclejdk8 image has all that we need and is quite small (just 170 MB).

The FROM command sets the mentioned image as the base on which our own will be built. We ADD the built JAR file to the container filesystem under the name storageService.jar. Next, we define for Docker container to listen on port 8080 at runtime with the EXPOSE command. This, however, will not enable communication to 8080 from the host. When the image is done, and we want to run it, we will also need to publish the port on the container with the following command docker run -p 8080:8080 amazonRepository/storageservice, where amazonRepository is a repository we will configure later in this tutorial. With CMD, we define which commands will be executed when we run the container. Values in the brackets of the CMD command simply mean the following will be executed when we run the container:

java -Djava.security.egd=file:/dev/./urandom -jar /storageService.jar

The option -Djava.security.egd=file:/dev/./urandom is needed to help mitigate JVM delays during startup. If omitted, it will make the application boot up extremely slow due to a random number generation process needed during the boot process.

This sums up the “Application development” part. With this done, the service we have created here will be automatically started when we run a Docker container later on. So, let’s start installation and configuration of the other tools needed to set up the continuous integration process.

Application and System Operations

First off, we need a clean Linux server on which to set up the Jenkins CI tool. Note that the following instructions are specifically for Ubuntu 14.04. Keep in mind instructions may slightly differ for other Linux distributions. The Jenkins version used is 2.7.1 and screens and instructions may differ slightly depending on the version of Jenkins being used.

So, we go to our Linux server console and start installing the prerequisites.

JDK Prerequisite

We need to have a JDK installed. Following are instructions for installing JDK8.

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get install python-software-properties
sudo apt-get update
sudo apt-get install oracle-java8-installer
java -version 

Install Docker

For Jenkins to be able to trigger Docker builds, we need to install docker-engine as follows:

sudo apt-get install apt-transport-https ca-certificates

sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D

#create a docker list file
sudo vi /etc/apt/sources.list.d/docker.list

#add the following entry in the docker.list file (change trusty to #the release you are running on if you are running on different, ie. #xenial, precise...):
deb https://apt.dockerproject.org/repo ubuntu-trusty main
#save and exit the file
 
sudo apt-get update
apt-cache policy docker-engine
sudo apt-get install docker-engine

As we have now installed the Docker engine, with the following command we will start a hello-world Docker image to confirm Docker works correctly.

sudo docker run hello-world

Hello-world image output will look as follows, and with this, we can confirm that the engine is working ok.

Install AWS Command Line Interface (CLI)

Next, we will install the AWS CLI. Later on, in Jenkins job configuration, we will use the CLI to execute commands for AWS authentication and Docker image push to Amazon EC2 container registry.

To install the AWS CLI, we follow the guidelines described in details on the Amazon CLI documentation.

Of the two installation options, we will choose installation using Pip, a package management system used for installing and managing Python programs. We will install Pip and AWS CLI simply by running the following three commands:

#install Python version 2.7 if it was not already installed during the JDK #prerequisite installation
sudo apt-get install python2.7
#install Pip package management for python
sudo apt-get install python-pip
#install AWS CLI
sudo pip install awscli

AWS ECR

As the last step of the build process, we will be pushing our Docker image to the Amazon container registry. In the Amazon web services console, we find the AWS EC2 Container Service.

We select Repositories submenu on the left and click on Get started.

We are then presented with the first screen for configuring the repository where we enter the repository name and click on the Next Step button.

Clicking on Next Step then shows us a screen with instructions on how to push images to the repository.

We are presented with an example of how to build and push a Docker image to the registry, but we don’t need to concern ourselves with this now. With this, we have created a repository.

Install and Configure Jenkins

To install Jenkins, we enter the following commands in the shell:

#Download Jenkins key and pipe it to apt-key tool, apt-key command #add will read from input stream, as defined by „–„. When added
#apt will be able to authenticate package to be installed.
wget -q -O - https://jenkins-ci.org/debian/jenkins-ci.org.key | sudo apt-key add -
#create a sources list for jenkins
sudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'

#update your local package list
sudo apt-get update

#install jenkins
sudo apt-get install jenkins

When the installation finishes, Jenkins starts automatically. Check the service status with the following command:

sudo service jenkins status

Jenkins will be connecting to Bitbucket Git repository, and to do so, we need to install Git.

#install Git
sudo apt-get install git

Jenkins will be triggering Gradle build process, during which a Docker image will be created. To be able to do this, the Jenkins user needs to be added to the docker user group:

#add Jenkins user to docker user group
sudo usermod -aG docker jenkins

During the build process, Jenkins will be pushing Docker images to Amazon ECR. To enable this, we need to configure AWS for the Jenkins user.

First, we need to switch to the jenkins user. To do so, we need to set a password.

#change Jenkins password
sudo passwd jenkins

#switch to Jenkins user
su – jenkins

#configure AWS
aws configure

After entering the aws configure command, we start inputting the generated secret access key and key ID (these are the credentials we generated earlier in the process). In my case, the account’s region is us-west-2, and so I input that. We also set the default output format for AWS commands to be JSON.

We can now move on to configuring Jenkins through the web console accessible on port 8080.

When we access the URL, we are presented with the following Getting started screen.

As stated on the screen, we need to input the password. Having done this, the setup wizard prompts us to do the following:

• Choose which plugins to install – we will choose Install suggested plugins.
• Create first admin user by inputting user credentials

When done, click Save and Finish. With this, we have finished Jenkins setup configuration.

Before we start defining the build job, we need to add a few additional plugins. We will go to Manage Jenkins and click on Manage plugins. In the Available tab, we first find the Bitbucket plugin, check the box, and click on Download and install after restart.

You will then be presented with something like the following screen.

After plugin installs, we repeat the process for the following additional plugins that will be needed to set up the job:

• Gradle plugin
• Docker build step plugin
• Cloudbees Docker custom build environment plugin
• Amazon ECR plugin

The Docker build step plugin we use will send requests to the Docker daemon. For this purpose, we need to enable the TCP socket on port 2375. To do to so, we enter the Docker configuration file located at etc/default/docker.

sudo vi /etc/default/docker

Here we add the following line in the configuration:

DOCKER_OPTS='-H tcp://127.0.0.1:2375 -H unix:///var/run/docker.sock'

We save and exit the file and restart both Docker and Jenkins service.

sudo service docker restart
sudo service jenkins restart

After Jenkins reboots, we go to the Jenkins console and, from Manage Jenkins, we choose Configure System.

We find the Docker builder section and enter http://localhost:2375 for the REST API URL, click on Apply to confirm the change. We then click the Test Connection to hopefully confirm that everything is ok.

We save the configuration and proceed to Jenkins job configuration.

Job Configuration

We go to the Jenkins home page and create a New Item.

We choose a Freestyle project and enter the project name as shown on the following screen:

Clicking on OK, we are presented with the job configuration page. We want the project to be built on each push to our Bitbucket Git repository. To achieve this, we first need to define the repository we are connecting to.

Step 1: Source Code Management

Under source code management, we choose Git and enter the URL of our Bitbucket repository. The URL has the form of https://bitbucket.org/bitbucketUsername/repositoryName.

After we enter the URL, Jenkins will automatically try to test the connection. Since we didn’t yet input credentials, it will show an error indicating it cannot connect.

Open the drop down list Add, and click on Credentials Jenkins provider.

We are presented with the following screen, where we enter the username and password for our Bitbucket account

After adding the new credentials record, we make sure to select it in the credentials drop down, and this finishes the Source code management setup.

Step 2: Build Triggers

Check Trigger builds remotely and define an authentication token. Make sure to define a random and secure token.

Step 3: Bitbucket Webhook

Jenkins already provided the URL for us which we will use on Bitbucket. We go to our Bitbucket repository page and, in the settings menu, click on Web hooks. Subsequently clicking on Add webhook presents us with the following screen, which we fill in as follows:

URL has the following structure: http://JENKINS_URL _HOST:PORT/job/JOB_NAME/build?token=TOKEN.

Enter the values above respectively with the Jenkins URL, port it is running on, the name of the job you have created, and token you have previously defined.

After saving the Webhook, you will be provided with the following screen, which you can edit if needed, or view requests generated each time we push new code.

With this configuration, the webhook is triggered on each repository push, regardless of the branch. On the Jenkins side, we can define which branch push will trigger the build.

For Bitbucket to be able to push code to Jenkins, we need to reconfigure Jenkins global security to allow anonymous read access. Additionally, for our setup, we have to disable the default Jenkins option which prevents cross-site request forgery. To do this, go to Manage Jenkins and choose Configure global security. Check Allow anonymous read access and check Prevent cross-site forgery exploits. Then save the configuration.

Please note that this is only done for simplicity reasons. Full setup surpasses the coverage of this tutorial, and it would include further securing Jenkins behind a reverse proxy, on TLS connection, and enabling CSRF prevention.

Step 4: Gradle Build

We can now return to the Jenkins job and continue configuring it. In the build section, we add a build step: Invoke gradle script.

On this form, we enter the following:

As shown on the screen, we will use the Gradle wrapper, a convenient Gradle feature that doesn’t require you to have Gradle installed on the host. Make sure to check the Make gradlew executable box.

In the tasks, we specify build and buildDocker.

Step 5: Docker Tag Image

This part of the build tags a Docker image previously prepared by Gradle’s dockerBuild task. For this, we add a new build step to the job: Execute Docker command. We choose the Tag image command and set the image name, target repository where we will push the image, and tag:

Step 6: Docker Push to Amazon ECR

Lastly, we need to define how to push our image to the Amazon ECR. For this, we add a new Execute shell build step and set the commands to authenticate to AWS and to push the image to Amazon ECR:

#region for our account is us-west-2
aws ecr get-login --region us-west-2 | bash
#push the previously tagged image
docker push 058432294874.dkr.ecr.us-west-2.amazonaws.com/springbootdocker:${BUILD_NUMBER}

With this, we have finished our build process. After pushing new code to the repo, this job will activate, and we will have a new Docker image uploaded to the Docker registry “auto-magically”.

The image can be then pulled to wherever we have docker-engine installed, and can be run with the following command:

docker run -p 8080:8080 amazonRepository/springbootdocker

This command will start our Spring Boot microservice, with the following endpoints for uploading and downloading our files to the S3 bucket:

• http://hostnameURL:8080/api/storage/upload
• http://hostnameURL:8080/api/storage/download?fileName=xyz

Further Steps with Java and Continuous Integration

There are always more things to do. A lot of ground has been covered in this tutorial, but I would consider this only a starting point from which to learn further. Putting Jenkins behind a web proxy server, like Nginx, and establishing a TLS connection, are only two examples of what more could, and arguably should, be done.

Our Docker image is available on Amazon ECR and ready for deploying. We can now take it and deploy it manually. However, a finer solution would be to automate it further. CI is only the first step, and next step is Continuous Delivery. What about a high availability? Amazon AWS EC2 provides features for registering containers in the cloud on a clustered environment which is mandatory for production based service. A good working example of developing a continuous delivery process can be found on the following AWS blog post.

Conclusion

All in all, we have put in place a smooth and clean software development process. Utilizing the tools available, we have created an infrastructure that helps maximize our productivity. Now, we do not have to worry about the configuration of our Java service which is a simple web service with a REST endpoint. We let the Spring Boot convention take care of everything and focus only on the service logic. We utilize Jenkins to build a new Docker image each time we push our code to our Bitbucket Git repository, and in the end, we’ve set the cloud to be responsible for storing our Docker images and files. When we deploy our service contained within a Docker image, we will be care free of any restrictions of the operating system (as long as the operating system has a docker-engine installed).