Source – zdnet.com

Typically, when people think of initiating DevOps practices, they think about automating infrastructure delivery. However, organizations that have successfully adopted DevOps typically go through a number of other specific steps before reaching that point, according to new research from Puppet.

“In a DevOps evolution, there are many paths to success but many more to failure,” Alanna Brown, Puppet’s director of product marketing, said to ZDNet. Organizations of all kinds, she said, “share the same set of challenges as they’re starting to adopt DevOps practices. If they lack the necessary support, if there are organizational roadblocks… all these things can kill early momentum and lead to cynicism.”

So with Puppet’s seventh annual State of DevOps report, shared before its release with ZDNet, Puppet decided to use its findings to develop a prescriptive path forward for organizations that may be having trouble scaling DevOps across the business.

The report, which will be published on Sept. 12, surveyed about 3,000 tech professionals worldwide about the specific practices they implemented on their DevOps journeys. The findings suggest there are five stages to adopting and scaling DevOps. Puppet ran a factor analysis to determine what two specific business practices represented the defining characteristics of those five stages, revealing 10 key practices along with several other contributing practices.

These stages and their corresponding practices help offer a linear path to scaling DevOps. Stages Two and Three could happen alternatively or simultaneously, but those steps are needed to get to Stage Four– which is where automating infrastructure delivery comes in.

“The key is that there is a path to success,” Brown said.

Here’s a look at the five stages and the defining practices of each stage:

1. NORMALIZING THE TECH STACK

“This is critical because enterprises are just inherently complex,” Brown said. In just the past five years, an explosion in technological development and new tools has created a “one of everything” problem, she added. Specifically, at this stage, teams are using version control and deploying on a standard set of operating systems.

2. STANDARDIZE AND REDUCE VARIATION

Building on Stage 1, teams at this stage typically work on deploying on a single operating system and building on a standard set of technologies, including databases, key value stores and message queues.

3: EXPANDING DEVOPS PRACTICES

“We start to see DevOps practices spread beyond dev and ops teams to other functional boundaries,” Brown said. “Individuals can do work without approval from outside their team, deployment patterns are being reused.”

Working without outside approval is more efficient but “can only happen after a certain level of trust is built,” Brown added. “Trust is built with success — showing you can deploy an application 10 times without having an incident.”

4. AUTOMATING INFRASTRUCTURE DELIVERY

At this crucial stage — what many organizations typically consider the beginning of a DevOps initiative — teams begin automating resources for broader use, not just use within their own divisions. Provisioning and automating resources across teams “highlights the need to get executive buy in,” at this stage, said Andi Mann, chief technology advocate at Splunk, a sponsor of this year’s survey.

Specifically, the defining practices are automating system configurations and automating provisioning.

5. PROVIDING SELF-SERVICE CAPABILITIES

Self service takes root once an organization has laid the groundwork with the other four stages and has established a high degree of trust internally. The defining practices of this stage are automating incident response and making resources available via self service.

To get to automated incident response, Brown said, “you need to address several manual chokepoints. The team that often is remediating the incident, they often don’t have access to the data they need, they need to get their access signed off…Getting to this automated point requires deep collaboration across multiple teams.”

That collaboration is a “really big cultural indicator,” she said, proving that the organization has cultivated the trust needed for DevOps.

*FOUNDATIONAL PRACTICES*

In addition to detailing these stages and defining practices, Puppet used the data to identify five “foundational practices” — practices that have a high impact across all five stages of a DevOps journey.

These include reusing deployment patterns when building applications or services (also highlighted in Stage 3), as well as reusing testing patterns. They also include using a configuration management tool and having teams contribute improvements to tooling provided by other teams. Lastly, successful organizations typically have monitoring and alerting configurable by the team operating the service.

The study refers to these as “foundational” practices because a plurality of survey respondents who successfully scaled DevOps were employing these practices early on in their journey.

Meanwhile, the foundational practices all reflect some of the core values of DevOps, Mann said. For example, he said, “We see things like monitoring and alerting being configurable by the team operating the service- it’s an empowerment model. Allowing them to understand their role in the system and giving them empowerment to create positive outcomes for their organization means this is a core part of what DevOps is.”

Before following along, ensure Salt is installed on at least one master and one minion, and that you have a basic understanding of core Salt concepts: minions (managed nodes), pillars (secure configuration data), and grains (system properties). The examples reference server names walkerrowe.copahost.com and eurovps — substitute these with your own server names.

Start Salt on the Master and Minion

Start the Salt services on the master and minion nodes respectively:

systemctl start salt-master
systemctl start salt-minion

While the examples below work with a single server, setting up at least two servers gives you a more realistic sense of how Salt behaves in production environments.

Verify Minion Connectivity with test.ping

The Salt test.ping command confirms that a minion is reachable and responsive. Note that this is not equivalent to the Linux ping utility — it does not send ICMP packets. Instead, it verifies that the Salt minion process is running and can communicate with the master.

salt '*' test.ping

A healthy minion returns True. The '*' glob targets all connected minions.

Run Ad Hoc Commands with cmd.run

Salt can execute arbitrary shell commands across minions without needing a state file. Use cmd.run to check disk usage on all minions:

salt '*' cmd.run 'df -h'

The '*' wildcard targets minions by hostname. Salt also supports targeting by grain values, node groups, compound matchers, and other methods — giving you fine-grained control over which nodes receive a command.

Install and Remove Software

Salt manages software installation declaratively through state files. The two key files are top.sls (which maps minions to state files) and individual state files such as webserver.sls.

Configure the Top File

The top file at /srv/salt/top.sls defines which states apply to which minions. A basic example that applies a webserver state to all minions looks like this:

base:
  '*':
    - webserver

Write a State File to Install Apache

Create /srv/salt/webserver.sls to install the Apache HTTP server using Salt’s pkg module. Both files use YAML format with two-space indentation:

apache2:
  pkg:
    - installed

These state files collectively define highstate — the desired configuration that minions should reach when state.highstate is applied. Apply highstate to a specific minion:

salt 'walkerrowe.copahost.com' state.highstate

Salt installs Apache and reports a summary of changes made, resources checked, and whether any failures occurred.

Remove a Package with pkg.purged

To remove Apache and its configuration files, update webserver.sls to use the purged state. The dot notation pkg.purged is shorthand that calls the purged function — it removes both the package binary and its configuration files:

apache2:
  pkg.purged

Run highstate again to apply the removal:

salt 'walkerrowe.copahost.com' state.highstate

Use Jinja Templating for OS-Conditional Logic

Salt state files support Jinja2 templating, which you can use to apply different configurations based on grain values such as the OS distribution. The following example installs Apache on Ubuntu and Nginx on CentOS:

{% if grains['osfinger'] == 'Ubuntu-14.04' %}
apache2:
  pkg:
    - installed
{% elif grains['osfinger'] == 'CentOS-6' %}
nginx:
  pkg:
    - installed
{% endif %}

The grains['osfinger'] value combines the OS name and major version (e.g., Ubuntu-20.04). Always verify the exact grain value on your systems using salt-call grains.item osfinger.

Install a Package from a Custom Repository

Salt can manage package repository definitions through the pkgrepo state module. This example adds the Kubernetes repository on a CentOS minion, then targets by grain (OS property) rather than hostname.

Update /srv/salt/top.sls to match on the os grain:

base:
  'os:CentOS':
    - match: grain
    - kubernetes

Create /srv/salt/kubernetes.sls to define the repository using Salt’s pkgrepo.managed state (this example targets yum-based systems):

base:
  pkgrepo.managed:
    - name: kubernetes
    - humanname: kubernetes
    - baseurl: http://yum.kubernetes.io/repos/kubernetes-el7-x86_64
    - file: /etc/yum.repos.d/kubernetes.repo
    - key_url: https://packages.cloud.google.com/yum/doc/yum-key.gpg

Run state.highstate and then verify the repository file was created on the minion:

ls /etc/yum.repos.d/kubernetes.repo
/etc/yum.repos.d/kubernetes.repo

For apt-based systems such as Ubuntu or Debian, consult the Salt pkg state documentation for the equivalent configuration using pkgrepo.managed with apt parameters.

Write a Custom Grain

Grains are system properties that Salt collects automatically (OS, CPU architecture, IP addresses, etc.). You can extend grains by writing custom Python modules. This example creates a grain that exposes memory usage statistics from the free command.

Note: This grain module is written for Ubuntu. The output format of free can differ between Linux distributions and versions, so the index positions used to parse the output may need adjustment on other systems.

Create the Grain Module

Create the grains directory on the Salt master and open a new Python file:

mkdir /srv/salt/_grains
vim /srv/salt/_grains/fmem.py

Add the following Python code. The function calls free, splits the output into tokens, and maps specific token positions to grain keys:

import subprocess

def memf():
    x = subprocess.check_output(["free"])
    y = x.split()
    grains = {}
    grains['totalmem'] = y[7]
    grains['usedmem']  = y[8]
    grains['freemem']  = y[9]
    return grains

Sync and Verify the Grain

Push the custom grain module to the minion using saltutil.sync_grains. The salt-call command runs Salt locally on the minion you are logged into:

salt-call saltutil.sync_grains

Salt should confirm the grain was synced:

local:
    - grains.fmem

You can then retrieve the custom grain values:

salt-call grains.item totalmem usedmem freemem

The output will include the three custom memory grain keys alongside Salt’s standard grain output.

Next Steps

The commands covered here represent a cross-section of what Salt can do: executing ad hoc remote commands, applying declarative state management, managing package repositories by grain targeting, and extending Salt’s inventory system with custom grains. As you grow more comfortable with these primitives, explore Salt’s pillar system for secure data management, Salt orchestration for multi-node workflows, and the Salt event bus for reactive automation.

Teams evaluating Salt should also compare the open-source and enterprise (Salt, formerly SaltStack Enterprise) editions, and benchmark Salt’s approach against alternatives such as Ansible, Chef, and Puppet to determine the best fit for their infrastructure scale and workflow requirements.