Why Securing DevOps Is Hard

The promise of DevSecOps — embedding security throughout the software delivery lifecycle rather than bolting it on at the end — is widely accepted in principle. In practice, most organizations struggle to deliver on it. According to Julien Vehent, who spent a decade in information security before writing about the challenge, the problem runs deeper than tooling choices. It is simultaneously a cultural problem and a tooling problem, and neither side of the equation can be solved without addressing the other.

Traditional infrastructure gave security teams a natural checkpoint: every new server involved a human being who could manually apply controls and extend trust. Cloud-native infrastructure eliminates that checkpoint. Systems are provisioned automatically in response to load, often without any human in the loop. Security approaches built around manual verification simply do not scale to that model.

The Cultural Shift: Moving Security Left

Historically, security was treated as a compliance gate applied at the end of the development cycle — just before a service went to production. The DevSecOps model inverts this. “Shifting left” means moving security consideration as close to the design phase as possible, so that security requirements are built into a service from the start rather than retrofitted afterward.

The most effective mechanism for achieving this is embedding security engineers directly inside development and operations teams. These engineers still report to a central security function, but they operate day-to-day within the product team: adopting the same tools, the same rituals, and the same communication channels. By doing so, they inject security knowledge into the core of the development and operations cycle rather than acting as an external reviewer.

“You put security in the middle of dev and ops, not at the end as a compliance step.”

Julien Vehent, Author of Securing DevOps: Security in the Cloud

The cultural resistance to this model often comes from both directions. Security teams worry that adopting DevOps practices means abandoning hard-won controls. Development and operations teams may not be ready to have security engineers embedded in their daily work. And at the executive level, many organizations still treat security as a compliance function by default — a mindset that trickles down to engineering teams and makes the shift-left model politically difficult to implement.

Vehent is direct about where this needs to be resolved: “At an executive leadership level, organizations need to train their people to break those barriers and inject security at all phases of developing, running and operating cloud services. This has to come from the top.”

The Tooling Shift: A Near-Complete Reinvention

The cultural change is inseparable from a fundamental change in tooling. According to Vehent, the vast majority of security tooling in use a decade ago has been replaced or reinvented for cloud-native environments. Some categories of tools — such as intrusion detection systems — largely disappeared from cloud infrastructure initially but are beginning to reappear as those environments mature.

Secrets management is one clear example of where new purpose-built tooling has emerged. Managing API keys, database credentials, and certificates in a dynamic cloud environment is a fundamentally different problem than managing them on static on-premises infrastructure. Tools that address this include:

• HashiCorp Vault — a widely adopted open-source secrets management platform designed for dynamic infrastructure.
• Mozilla SOPS (Secrets OPerationS) — an open-source tool for encrypting secrets stored in files such as YAML, JSON, and ENV, using cloud KMS providers or PGP keys.
• AWS Secrets Manager and Parameter Store — native AWS services for storing and rotating secrets tied to IAM trust models.
• Google Cloud Secret Manager and Azure Key Vault — equivalent managed services on GCP and Azure respectively.

These tools share a common design philosophy: they leverage the trust models already built into cloud infrastructure — IAM roles, instance identity, and workload identity — rather than relying on static credentials or manually established trust relationships.

Security Teams Are Becoming Software Engineering Teams

One of the more significant structural changes in DevSecOps is the shift in the makeup of security teams themselves. Where security teams were historically composed largely of network engineers and compliance specialists, effective DevSecOps teams now require software engineering capability. Security engineers are expected to write the automation and tooling that enforces security controls across the pipeline — not just audit or review it.

This does not mean security specialists need to become full-stack developers. But the ability to write scripts, build integrations with CI/CD pipelines, and contribute to infrastructure-as-code repositories has become a practical requirement for security engineers working in cloud-native environments.

Integrating Security Into Existing Operational Tooling

Vehent’s experience at Mozilla offers a concrete illustration of what this integration looks like in practice. Early on, his team developed Mozilla InvestiGator, an in-house endpoint security tool designed to interrogate servers at scale. The tool worked, but it was built alongside the existing infrastructure rather than into it — a security layer placed on top of operational tooling rather than woven through it.

When Vehent moved to work on Mozilla’s AWS infrastructure, he noticed that many of the data and visibility needs the security team had were already being met by tools the operations team relied on for their own purposes. An asset inventory, for example, was something operations already maintained because they needed it to manage the environment. Rather than building a separate security-specific inventory, the security team could consume and extend the existing one.

This realization — that security answers can often be derived from operational tooling rather than separate security tooling — is central to the DevSecOps philosophy. It reduces tool sprawl, keeps security data in sync with operational reality, and gives security teams a foothold inside the operational workflow rather than alongside it.

Common Failure Modes

Vehent identifies a consistent pattern in organizations that fail to achieve genuine DevSecOps: security teams are not in a position to engage with development and operations teams early enough in the process. Lacking executive support, cultural buy-in, or embedded relationships with product teams, they end up reverting to the compliance-at-the-end model — not because they want to, but because they have no other access point.

The practical result is that security reviews happen too late to influence design decisions, vulnerabilities discovered at review time are expensive to remediate, and the relationship between security and engineering teams remains adversarial rather than collaborative.

Who Benefits From Learning DevSecOps

Vehent wrote Securing DevOps: Security in the Cloud with a specific reader in mind: junior security engineers looking to move into DevSecOps, or experienced developers and system administrators looking to add security depth to their skill set. The book assumes scripting and operations experience rather than starting from first principles in computer science.

In practice, the book also found an audience among senior security professionals — sometimes with decades of experience — who had not previously worked in a DevSecOps environment. For these readers, the value was less about learning specific tools and more about understanding what a DevSecOps workflow actually looks like day-to-day: how roles change, what skills to develop, and what a reasonable roadmap for adopting these practices looks like over the first few years.

Key Takeaways for DevOps and Security Teams

• Both sides must change. DevOps teams must mature to adopt security practices. Security teams must understand cloud-native operational models to implement controls effectively.
• Shift left requires embedding. Posting security engineers inside product teams — not alongside them — is the most reliable mechanism for moving security earlier in the lifecycle.
• Executive support is non-negotiable. Cultural change in large organizations does not happen bottom-up. Leadership must explicitly prioritize breaking down the barrier between security and engineering.
• Tooling has changed fundamentally. Cloud-native secrets management, identity-based trust models, and security automation replace many legacy control mechanisms. Teams that resist retraining risk operating with an approach that does not fit the environment.
• Extend operational tools before building new ones. Security data needs often overlap with operational data needs. Leveraging existing tooling reduces friction and keeps security integrated with the operational workflow.
• Security engineering is software engineering. Writing automation, building pipeline integrations, and contributing to infrastructure-as-code are now core competencies for DevSecOps practitioners.

Why Containers and Kubernetes Matter for HPC

Software containers and Kubernetes have become foundational tools for building, deploying, and managing enterprise applications at scale. They allow organisations to deliver software faster and more reliably while using compute resources more efficiently and reducing operational costs.

High-performance computing (HPC) is increasingly moving closer to the enterprise world. As a result, the HPC community stands to benefit from the container and Kubernetes ecosystem — particularly the ability to quickly allocate and deallocate large computational resources on demand. Unlike the traditional HPC model, where capacity was planned years in advance, modern compute infrastructure must be elastic and responsive to workload spikes.

Cloud providers and software vendors are continuously improving on-demand resource provisioning. In enterprise computing, packaging software in container images is now standard practice. Kubernetes has become the dominant container orchestrator in large enterprises. The HPC community — led largely by AI and machine learning workloads — is now adopting these patterns and applying them to batch jobs and interactive scientific applications.

The Challenge for HPC Teams

HPC leaders face a rapidly shifting technology landscape. Containerisation has become ubiquitous, Kubernetes has emerged as the standard for container orchestration, deep learning workloads continue to grow, and Site Reliability Engineering (SRE) practices have spread across data centres. Evaluating which of these technologies genuinely applies to HPC-style workloads — tightly coupled parallel jobs, massive batch throughput, specialised hardware — remains a real challenge.

A Brief Introduction to Kubernetes

When running a single container on a developer laptop, Docker alone is sufficient. But when deploying hundreds of containers across dozens or hundreds of machines, manual management quickly becomes impractical. Kubernetes addresses this by providing scheduling, container lifecycle management, networking, and self-healing capabilities in a scalable, extensible platform.

The core Kubernetes architecture consists of:

• Control plane (master): Contains the API server, scheduler, and controller manager. Controllers continuously compare the current state of resources against the desired state and take corrective action when they diverge.
• Worker nodes: Run the kubelet agent, which communicates with the control plane and manages containers via the Container Runtime Interface (CRI). Supported runtimes include containerd, CRI-O, and Docker (via shim).
• kube-proxy: Handles network proxying and load balancing rules on each node.

Kubernetes vs. Traditional HPC Workload Managers

Kubernetes and HPC workload managers such as SLURM, PBS Pro, or Univa Grid Engine both perform workload and resource management — but they were designed for fundamentally different workload profiles.

Strengths of HPC Workload Managers

Traditional HPC schedulers have been refined over decades for demanding scientific computing scenarios. Key capabilities include:

• Support for shared-memory and distributed-memory (MPI-based) parallel jobs
• Advance reservations — blocking resources ahead of time for scheduled workloads
• Fair-share scheduling to manage resource allocation across users, projects, and departments
• Resource reservation to accumulate capacity for large multi-node jobs
• Preemption — suspending lower-priority jobs to free resources for high-priority ones
• NUMA-aware scheduling for precise allocation of CPU cores and sockets
• Standards compliance for job submission and management (DRMAA, DRMAAv2)
• Proven scalability for millions of batch jobs per day on the world’s largest supercomputers

What traditional HPC managers lack are native support for microservice architectures, deeply integrated container lifecycle management, overlay networking, and rolling application updates.

Strengths of Kubernetes

Kubernetes was built from the ground up for containerised microservice applications. Its notable capabilities include:

• Management of Pods — groups of one or more co-located containers sharing network and storage namespaces
• Pluggable overlay networking via CNI plugins
• Self-healing through the controller loop (desired state vs. actual state)
• Declarative, version-controlled configuration via YAML manifests
• Built-in load balancing and service discovery
• Rolling updates and canary deployments
• Native integrations with monitoring and logging stacks (Prometheus, Fluentd, etc.)
• Persistent storage hooks through Persistent Volumes and CSI drivers

What Kubernetes currently lacks for HPC use cases is a high-throughput batch job queuing system with the sophisticated fair-share, reservation, and preemption rules that scientific computing requires. Additionally, many legacy HPC engineering applications are not designed to interface with Kubernetes APIs — though this is changing as new AI-centric frameworks such as Kubeflow are built natively for Kubernetes.

Combining Kubernetes and HPC Workload Managers

The most practical near-term approach is not to replace one system with the other, but to combine them. The recommended pattern works as follows:

1. Kubernetes manages HPC containers and supporting services — handling deployment, scaling, networking, and lifecycle across the cluster.
2. Inside those containers, an HPC workload manager (SLURM, Univa Grid Engine, etc.) runs — either connected to an existing HPC cluster or deployed as a self-contained instance within the containerised environment.

This layered approach preserves compatibility with existing HPC engineering applications and retains advanced batch scheduling capabilities, while the overall deployment gains the portability and operational consistency of Kubernetes.

Operational Benefits of This Architecture

Simplified Administration

HPC environments can involve large numbers of containers serving different purposes — compute nodes, head nodes, license servers, pre- and post-processing services. Kubernetes abstracts away the complexity of managing individual container runtimes, making scaling operations — adding or removing compute capacity — significantly simpler and more repeatable.

Hybrid and Multi-Cloud Portability

Portability is one of the core value propositions of containers, and Kubernetes extends that portability to entire fleets of containers. An HPC stack packaged for Kubernetes can run on-premises, on a public cloud provider, or across both simultaneously. Engineers interact with the same interfaces regardless of the underlying infrastructure, allowing organisations to make infrastructure decisions based on cost, performance, and available capabilities — not on application lock-in.

On-premises deployments can offer a true hybrid-cloud experience: consistent HPC application behaviour with the ability to burst to cloud resources on demand.

What’s Next for Kubernetes and HPC?

Adapting Kubernetes to the specific demands of HPC and engineering workloads is not trivial. MPI job patterns, tight inter-node communication latency requirements, and high-throughput batch scheduling are all areas where additional tooling — such as the Kubernetes Job API extensions, MPI Operator, or volcano batch scheduler — continues to mature.

The primary driver of this progress is the AI and machine learning community, whose workloads share many characteristics with traditional HPC (GPU clusters, parallel training jobs, large data movement) but have adopted Kubernetes natively. Projects like Kubeflow, volcano, and the Kubernetes SIG-Scheduling working group are expanding what is possible for batch and HPC-style workloads on Kubernetes.

For HPC practitioners, the practical path forward is to treat Kubernetes as the infrastructure and operations layer, while preserving proven HPC workload managers for the scheduling logic that scientific computing depends on — at least until native Kubernetes batch capabilities reach the required maturity.

About the Original Authors

Daniel Gruber, Burak Yenier, and Wolfgang Gentzsch are with UberCloud, a company that began developing HPC container technology and containerised engineering applications in 2013 to facilitate access to HPC workloads in shared on-premises and cloud environments. This article draws on their published white paper describing their experience with UberCloud HPC containers and Kubernetes.

The Problem With Too Many Choices

DevOps methodologies and continuous integration and delivery (CI/CD) practices have produced an explosion of automated deployment tools, each designed to streamline some part of the software delivery pipeline. The abundance of options is itself one of the biggest challenges teams face.

IT and development organizations typically assemble complex toolchains that orchestrate the full application lifecycle—from code authoring and integration through automated testing to production deployment. Many of these tools originated to solve a narrow problem (automated system configuration, for example, or code merging), but have since expanded to cover adjacent stages of the CI/CD pipeline, blurring the lines between development tooling and infrastructure tooling.

Feature creep, category overlap, and the absence of any industry-standard taxonomy for DevOps toolchains make tool selection genuinely difficult. Both proprietary and open-source technologies are widely used, and neither the categories nor the boundaries between them are stable. Despite this complexity, it is possible to identify broad groupings of tools that correspond to recognizable phases in the deployment process—and to apply concrete selection criteria once those groupings are understood.

The Four Core Categories of the DevOps Toolchain

No tool fits neatly into a single box, but most automated deployment tools fall into one or more of the following four broad categories.

1. Build Automation and Source Code Management

Source code management (SCM) and build automation tools let developers collaborate in parallel without creating conflicts or confusion in the codebase. Git has become the de facto standard version control system. Hosted Git services include:

• GitHub – widely adopted, especially in open-source communities
• AWS CodeCommit – managed Git service integrated with the AWS ecosystem
• Google Cloud Source Repositories – Google’s hosted Git offering
• Microsoft Azure Repos (formerly Visual Studio Team Services) – integrated with the Azure DevOps suite
• Atlassian Bitbucket – popular in shops already using Jira or Confluence

2. Continuous Integration

CI tools automatically merge contributions from multiple developers into a shared source repository, then trigger build and test workflows on every change. Key options include:

• Jenkins – one of the most widely deployed CI servers; highly extensible via plug-ins, enabling complex multi-stage pipelines covering build, test, and deployment
• CircleCI – cloud-native CI/CD with fast setup and YAML-based configuration
• Codeship – aimed at teams that want simplicity over deep customization
• TeamCity – JetBrains’ CI server, well-suited to Java and .NET shops
• Travis CI – popular in open-source projects; integrates tightly with GitHub

3. Application Deployment and Configuration Management

Configuration management platforms handle system installation, activation, patching, updates, and eventual decommissioning. They often overlap with CI tooling and can be extended via plug-ins. Common choices include:

• Ansible (Red Hat) – agentless, uses YAML playbooks; low barrier to entry
• Chef – uses a Ruby-based DSL for configuration “recipes”; mature ecosystem
• Puppet – declarative, model-based configuration; strong enterprise adoption
• SaltStack (Salt) – event-driven automation with fast execution over large fleets
• Capistrano – Ruby-based deployment tool, historically popular for web apps
• Fabric – Python library for remote shell commands and deployment tasks
• AWS OpsWorks – managed Chef and Puppet service within AWS
• Google Cloud Deployment Manager – declarative infrastructure management for GCP
• Microsoft Azure Automation – runbook-based automation using PowerShell or Python

4. Infrastructure Deployment (Infrastructure as Code)

Infrastructure as Code (IaC) tools treat server and network provisioning the same way developers treat application code: as versioned, reviewable, repeatable definitions. This approach enforces consistency between environments, reduces manual steps that introduce human error, and makes infrastructure auditable. Key tools include:

• Terraform (HashiCorp) – cloud-agnostic IaC tool using HashiCorp Configuration Language (HCL); the production-scale successor to Vagrant
• Packer (HashiCorp) – automates the creation of machine and container images across multiple platforms
• AWS CloudFormation – native IaC for AWS using JSON or YAML templates
• Chef Habitat – application automation focused on containerized and cloud-native workloads

Note that tools from Category 3—Ansible, Chef, and Puppet in particular—can also perform infrastructure automation once the appropriate client-side agents or modules are installed in machine images, further blurring category boundaries.

How to Select the Right Automated Deployment Tools

The following practical criteria can guide tool selection regardless of which specific products are on your shortlist.

Think Holistically About the Entire CI/CD Stack

No single tool covers every stage of the pipeline. Plan for how tools will connect before committing to any one product. For example, a consultancy focused on agile delivery might combine Terraform for infrastructure orchestration, Ansible for provisioning and configuration, Jenkins for the CI code pipeline, and a Makefile as the glue that ties each stage together. Define the hand-off points between tools before you evaluate individual options.

Match Tools to Your Language and Infrastructure Stack

Select tools that support the programming languages your developers already use, and that integrate cleanly with the infrastructure your operations team already manages. If your teams are already productive with Jenkins, evaluate whether it can serve additional teams or pipelines before introducing a competing CI tool.

Factor in Team Expertise and Budget

Jenkins is powerful but has a steep configuration curve. If your DevOps budget is modest or your team lacks the bandwidth to learn a complex new tool, a simpler hosted option such as Codeship or TeamCity may deliver faster value. Operational simplicity is a legitimate selection criterion—especially for smaller teams.

Evaluate Configuration Syntax and DSL Familiarity

Each tool uses a different syntax or domain-specific language (DSL):

• Ansible uses YAML—structured and human-readable, but whitespace-sensitive and potentially unfamiliar to some team members
• Chef uses a Ruby-based DSL for its configuration recipes—an advantage for teams with Ruby experience
• Terraform uses HCL, which is declarative and relatively readable but requires learning its own resource model
• CloudFormation uses JSON or YAML, which is portable but can become verbose for large stacks

Choose syntax that your team can realistically maintain, review, and debug. Configuration drift caused by poorly understood automation is itself a source of human error.

Align With Your Target Infrastructure Platform

If you are deploying primarily to a single cloud provider, the native tooling for that platform is worth evaluating first:

• AWS users should evaluate CodePipeline, CodeDeploy, CodeCommit, and OpsWorks. CodePipeline integrates with GitHub and Jenkins and uses JSON to define pipeline structure. OpsWorks is essentially managed Chef, meaning automation code can be redeployed outside AWS with minimal changes.
• Microsoft Azure users may prefer Azure DevOps (formerly VSTS) and Azure Automation with PowerShell runbooks.
• Multi-cloud or hybrid environments favor cloud-agnostic tools such as Terraform and Ansible, which can target on-premises servers, private clouds, and multiple public clouds from a single codebase.

The trade-off with native cloud tools is vendor lock-in at the pipeline level. However, because the underlying automation code is often portable—OpsWorks uses standard Chef cookbooks, CodePipeline integrates with third-party tools—the practical lock-in risk may be lower than it first appears.

Plan for Cross-Cloud Compatibility From the Start

Organizations committed to hybrid or multi-cloud strategies should prioritize tools that work across environments. Absorbing the short-term cost of learning a cloud-agnostic DSL—such as Terraform’s HCL or Ansible’s YAML—pays long-term dividends when workloads need to move between providers or when on-premises infrastructure must be managed alongside cloud resources using a consistent toolchain.

Why Automation Matters for Reducing Human Error

Manual deployment processes are inherently error-prone. Steps get skipped, environment-specific configuration values get entered incorrectly, and runbooks drift out of sync with reality. Automated deployment tools address this by encoding deployment logic as versioned, testable, repeatable code. When a deployment fails, the failure is reproducible and debuggable. When it succeeds, the same code can be applied consistently across development, staging, and production environments—eliminating an entire class of “works on my machine” defects.

The goal is not to replace human judgment but to remove humans from the execution of repetitive, high-stakes steps where fatigue and distraction introduce risk. Engineers remain responsible for designing, reviewing, and approving the automation—making those decisions the focus of human attention rather than the mechanics of running commands on production servers.

Summary

Automated deployment tools fall broadly into four categories: source code management and build automation, continuous integration, application deployment and configuration management, and infrastructure deployment (IaC). The boundaries between categories are porous, and many tools serve multiple roles. Effective tool selection requires a holistic view of the CI/CD pipeline, an honest assessment of team expertise and budget, attention to language and infrastructure compatibility, and a deliberate decision about how much vendor-specific integration is acceptable given your multi-cloud strategy.