The late great W. Edwards Deming, statistician, professor, author, lecturer, and consultant said, “In God we trust. All others must bring data.” The point is, if you want people to trust your information, then you need to present measured data. This is true for countless circumstances. As a DevOps consultant, I am often asked by my clients what data should be used to show the value of their DevOps capabilities, in a way that is acceptable to the stakeholders of those DevOps capabilities. A lot of excellent research results, published by DORA and others, describes metrics that demonstrate the value that DevOps provides to application development. More and more enterprises are embracing the idea of providing DevOps capabilities, such as pipelines, tools, hardened containers and tech stacks, as a centralized service to support their many application teams. The question then becomes, “What metrics should be used to show the value of DevOps-as-a-Service (DaaS)?”

Before measuring the value of anything, you need to know two things: “What is it?” and “Why is it important?”

What is DevOps-as-a-Service?
In high-level terms, DaaS is DevOps capabilities made available to the users of those capabilities through portals and APIs. There is no standard set of services; DevOps capabilities offered vary from organization to organization. In my SlideShare, DevOps-as-a-Service (DaaS) Values, I illustrate a generic blueprint for DaaS.

The DaaS blueprint has four layers, as follows:

• Stakeholders – There are three types of stakeholders: business, app-dev users and DaaS teams.
• Service access layer – This layer consists of user portals, APIs, and access security.
• Service layer – The service layer consists of user services and management services. User services include things such as repositories, CI/CD pipelines, sandboxes, test data, configuration data, guides and training. Management services include service catalogs, issues and change requests, SLOs, SLIs, governance policies, experience monitoring and automated tasks.
• Resources layer – This layer includes application stacks, infrastructure resources, tools, infrastructure code and topology data.

Why is DaaS Important?
The value of DaaS, as perceived by different stakeholders, can be summarized as follows:

• Business – DaaS makes the business competitive.
• App-dev Users – DaaS improves application development.
• DaaS team(s) – DaaS improves management of DevOps capabilities.

What Service Level Objectives (SLOs) Demonstrate the Value of DaaS?
The SLOs need to be chosen to best fit the requirements of each application. The following are examples.

Business value can be demonstrated by:
ROI based on tracking cost savings relative to a benchmark comparing prior non-DaaS, trend cost per developer and per release.
SLO agility of services – Lead time for changes, and frequency of releases.
SLO reliability of services – change failure rate, time to restore services.
SLO governance, compliance and security scores.

App-dev user value can be demonstrated by:
SLO availability of DaaS, service level indicators (SLIs) depend on service components and architecture.
SLO service response time.
SLO experience scores.
SLO change request response time.

DaaS team value can be demonstrated by:
SLO scope of applications – number of apps supported, number of users per month.
SLO number of requests served per month.
SLO critical events per month.

How Does an Organization Develop DaaS Metrics?
Start by picking an application that is supported by DaaS. Do not try to boil the ocean and measure all applications at once. Then, appoint a DaaS SLO/SLI team to work on metrics development for the application. The team should include stakeholders from business, app-dev and DaaS support teams associated with the application.

DaaS SLO/SLI team members determine the SLOs of DaaS that each class of stakeholder values the most, and then decomposes the services to identify sources of data that will form the most important service level indicators (SLIs).

With the SLIs in hand, instrumentation, telemetry and analysis requirements can be determined and used to design monitoring agents, communication, analyzers, alerts and dashboards.

Then, roles, policies, playbooks and responsibilities will complete the solution. Test the solution for the chosen application. As you gain confidence in the solution, adapt and expand the solution for use with other applications.

Continue to monitor the progress and performance of the SLO metrics, and continuously improve them with experience and as newer technologies become available.

What This Means

DevOps-as-a-Service provides many valuable capabilities to business, app-dev users and DaaS teams. Metrics that demonstrate value to each class of stakeholder should be determined for each application that consumes the capabilities offered by each unique DaaS. A DaaS SLO/SLI team can determine the best metrics and system design for the implementation for an example application. The SLOs can be adapted and evolved for other applications in the organization, and used as a basis for continuous improvement.

The rise of COVID-19 defined 2020 for nearly every industry. Cloud computing is no exception, though growth in the sector continued this year. Gartner estimates that $257.9 billion will be spent on public cloud services in 2020, up 6.3% from 2019. IDC reports that almost half of IT spending is cloud-based, “reaching 60% of all IT infrastructure and 60% to 70% of all software, services and technology spending in 2020.” These projections came in mid-2020, which proves that cloud adoption and spending were not greatly impacted by the pandemic.

You can make the argument that cloud computing itself is the trend of 2020. Blockchain, quantum computing, machine learning and AI all deservedly earn a lot of headlines in the tech press. But at a more basic level, cloud computing is changing the way we do business. Whether it is the way they store their data, improvements to agility and go-to-market for faster release of new products and services, or how they develop and operate services remotely in today’s “locked-down world,” cloud computing is benefitting all businesses in every sector.

In short, cloud computing had a strong year in 2020 despite COVID-19 and appears to have a very bright future. Several key trends, perhaps overshadowed by the pandemic, helped define a dynamic year for the sector. These include the following:

Continued Dominance of ‘The Big Four’ – All of the big four cloud providers—AWS, Azure, Google Cloud and Alibaba Cloud—reported strong growth and collectively grew 40% in Q320. Together they still account for 63% of the entire cloud sector.
Rise of Managed Services – The global cloud managed services market, in particular, is growing rapidly and is expected to reach $116 billion by 2026, growing from $62.4 billion in 2020, according to a study conducted by Markets and Markets.
Serverless and Containers – There used to be some talk about serverless and containers replacing the use of traditional virtual machines (VM). But I mostly see customers realizing that VM infrastructure is not going to disappear anytime soon, though container use certainly rose over the last 12 months.
Automation – The push toward greater automation continued in 2020. I noticed users accepting greater automation in their infrastructure environments, and there was simply more demand for it across the board. This is great news as one of the major advantages of a DevOps approach is automating away the manual tasks that bog down critical projects.
Growth in home use – This may seem an obvious one, but with the COVID-19 pandemic leading companies across the globe to new “work from home” policies, virtual desk infrastructure (VDI) usage exploded.
Security – Especially with a remote workforce, cloud security faced new challenges this year. One example of this playing out is in Snowflake’s massive IPO.
Other trends of note this year included the continued growth of hybrid and multi-cloud and desktop as a service (DaaS). So, while COVID-19 truly defined 2020 for most people, the cloud sector was busy with the trends discussed above. It will certainly be interesting to see how these trends play out in the year ahead. Whatever new trends arise, it seems likely that cloud computing will continue to transform the way we all do business.