This article is based on a post originally published on devops.com. The topic has been expanded and clarified for a DevOps engineering audience.
DevOps can feel frustratingly vague. It is not defined by a specific toolchain, nor does it mandate a rigid set of processes. At its core, DevOps is a cultural philosophy that prioritises automation, collaboration, and continuous improvement. That ambiguity makes it difficult to answer a very practical question: how do you know whether your DevOps practice is actually working?
The answer lies in key performance indicators (KPIs). Choosing the right KPIs gives teams concrete, measurable evidence of progress, highlights bottlenecks, and grounds cultural conversations in data. This article focuses on a specific, often-overlooked context: Z DevOps, the practice of applying DevOps principles to IBM Z mainframe software delivery.
What Is Z DevOps?
Z DevOps refers to the application of DevOps culture and practices within the IBM Z (mainframe) ecosystem. Mainframes are not a legacy curiosity: a substantial share of enterprise-critical transactions — including banking, insurance, and retail — still run on IBM Z systems, and vast amounts of structured business data reside there.
Despite this, mainframe environments have historically lagged behind cloud and distributed systems in adopting modern software delivery practices. Z DevOps is the effort to close that gap — introducing continuous integration, automated testing, pipeline-based delivery, and cross-team collaboration into mainframe development workflows.
One practical challenge in Z DevOps is that many generic DevOps tools were designed for Linux/cloud infrastructure and do not natively support z/OS environments, COBOL codebases, or mainframe-specific build and deploy processes. Teams pursuing Z DevOps therefore need purpose-built or Z-aware tooling on top of sound KPI practices.
Why KPIs Matter in a Culture-First Practice
Because DevOps is fundamentally cultural, it is tempting to evaluate it through qualitative means alone — surveys, retrospectives, anecdotal feedback. Those inputs are valuable, but they are insufficient on their own. KPIs provide:
- Objectivity: Numbers are harder to argue with than opinions when making the case for investment or change.
- Trend visibility: A single data point is noise; a series of measurements over time reveals whether things are improving, stagnating, or regressing.
- Accountability: Teams and individuals can align their work to specific, measurable outcomes.
- Continuous improvement: You cannot improve what you do not measure. KPIs surface opportunities that anecdotal observation misses.
Key KPIs for Z DevOps
The following KPIs are relevant regardless of where you are on your Z DevOps journey — whether you are just introducing pipeline automation or optimising a mature delivery process.
1. Deployment Frequency (Mean Time to Delivery)
DevOps prioritises faster, smaller, and more frequent software releases. Tracking how many deployments you complete per week or per month — sometimes called Mean Time to Delivery (MTTD) — gives you a baseline and lets you measure improvement over time.
Continuous deployment at a literally constant cadence is an unrealistic goal for most mainframe environments. What matters is the trend: are releases becoming more frequent and less risky over time? Even moving from quarterly releases to monthly, and then to bi-weekly, represents meaningful DevOps progress on Z.
2. Deployment Success Rate
Not every deployment succeeds. Failures occur due to environment mismatches, incomplete testing, configuration errors, and many other causes. Tracking the ratio of successful deployments to total deployment attempts tells you how reliable your pipeline is.
A sustained low success rate is a signal to investigate pipeline quality, test coverage, or process gaps — not just to blame individual teams. Tracking this metric over time lets you validate whether pipeline improvements are actually reducing failures.
3. Deployment Scale
Deployments vary enormously in scope. A minor performance patch and a major feature release are both “deployments,” but comparing their success rates or durations without accounting for scale produces misleading conclusions.
Useful proxies for deployment scale include:
- Lines of code or number of modules changed
- Percentage of the deployment containing new features versus bug fixes
- Number of dependent systems affected
Capturing scale data alongside other deployment KPIs provides context that makes the rest of your metrics interpretable.
4. Test Failure Rate
Automated testing is a cornerstone of any DevOps pipeline. In Z DevOps, this includes unit tests, integration tests, and regression tests covering mainframe application logic. Tracking how often tests fail — and at which stage of the pipeline — helps you identify whether:
- Developers are receiving insufficient feedback on code quality before committing
- Test suites are poorly maintained or not representative of real conditions
- Velocity pressure is causing code to be submitted before it is ready
A rising test failure rate is an early warning sign that pipeline quality is degrading, often before deployment success rates begin to fall.
5. Per-Team Delivery Performance
Organisation-wide metrics can obscure significant variation between teams. One team might be deploying frequently and reliably while another struggles with the same pipeline. Breaking down deployment frequency, success rate, and test failure rate by team enables targeted intervention:
- Teams with low performance can receive coaching, tooling support, or process review.
- High-performing teams can serve as internal examples and mentors.
This is especially important in large mainframe shops where COBOL, PL/I, Assembler, and Java teams may coexist and have very different maturity levels.
6. Inter-Team Communication Activity
DevOps explicitly values transparency and collaboration across development, operations, security, and business teams. While communication quality is inherently qualitative, communication volume and patterns can be tracked as a proxy metric.
Data points to consider include:
- Frequency of cross-team messages in collaboration tools
- Number of shared retrospectives or joint planning sessions
- Incident response times that depend on cross-team escalation
This metric has significant limitations — high message volume does not guarantee effective communication — but combined with qualitative feedback, it adds a useful data layer to cultural assessments.
Collecting DevOps KPIs in the Z Ecosystem
Collecting the KPIs above in a Z environment is more involved than in a cloud-native context. Generic DevOps observability platforms were not built with z/OS in mind and may not integrate with mainframe build systems, job schedulers, or deployment tooling.
IBM’s UrbanCode Velocity (now part of the IBM DevOps portfolio) is one tool specifically cited for Z DevOps KPI collection. It provides:
- Release automation templates designed for mainframe delivery pipelines
- Integration with Z-specific tooling to gather pipeline telemetry
- Dashboards and reporting for the KPIs described above
- A unified view that can span both mainframe and distributed pipeline stages
Using a Z-aware platform means your KPI data reflects actual mainframe pipeline behaviour rather than being approximated from generic CI/CD logs that may not capture z/OS job outcomes accurately.
Building a KPI Practice: Practical Recommendations
- Start with a baseline. Before optimising anything, collect at least one full release cycle of data for each KPI so you have a starting point for comparison.
- Pick a small, meaningful set of metrics. Tracking every possible data point creates noise. Start with deployment frequency, success rate, and test failure rate, then expand as your practice matures.
- Review KPIs regularly in retrospectives. Data without discussion does not drive change. Incorporate KPI review into your sprint or release retrospectives.
- Avoid weaponising metrics. KPIs should identify systemic issues, not blame individuals. Ensure teams understand that the goal is improvement, not performance management.
- Iterate on your KPIs. As your Z DevOps practice evolves, the metrics that matter most will change. Revisit your KPI set periodically and retire those that no longer provide useful signal.
Conclusion
DevOps is a cultural practice, but culture is shaped by feedback — and KPIs are one of the most reliable feedback mechanisms available. For teams doing Z DevOps, the challenge is both selecting the right metrics and using tooling that actually integrates with the mainframe environment. Tracking deployment frequency, success rate, scale, test failure rates, per-team performance, and inter-team communication gives you a multi-dimensional view of DevOps health. Pair that data with the right Z-aware tooling and a commitment to acting on what you learn, and you have the foundation for genuine, sustained improvement.