Jeremy Unruh, head of developer efficiency and platform engineering for Credit Karma, says one of his team’s goals is to speed up the company’s engineering. Given the complexities of many platforms and the technologies they use such as Kubernetes, he says the typical product engineer must focus on shipping features. That can include experimenting and shifting traffic before customers see those features, Unruh says. “Part of my team’s motive is to abstract that away and provide a seamless experience where they really don’t have to think about anything but what they’re developing.”

Unruh says part of his challenge when he joined Credit Karma about three years ago was to increase efficiency of releasing code across the company. The engineers there had been using an older Jenkins-style system, he says, which served as a generic job runner. Developing products on that system meant clearing a few hurdles along the way, Unruh says, including jumping through a remote desktop running on a Windows computer. On top of that, teams building new microservices were required to write custom deployment code to move production forward, he says. That would be the basis for the job for the system to execute the service, Unruh says.

That meant everything was different because every team took their own approach, he says, which slowed them down. “It linearly required 15 steps just to deploy your service into production,” Unruh says. “It was really cumbersome and there was no way for us to standardize.”

Looking for ways to improve efficiency, he wanted to eliminate the need to jump to another host just to access the system. Unruh says he also sought to end the need for custom code for deploying a service. “I just build a service and I can deploy it,” he says.

That is where Falcon came in, Unruh says, which is used by every engineer in the company, who number upward of 700. He says many early users indicated that the tool cut their time in half. Now Falcon manages the release lifecycle through all of Credit Karma’s environments, Unruh says.

Falcon has also streamlined the manager approval process for features, he says, and increased visibility. It allows the teams to weigh traffic to services, which helps detect anomalies and errors with the new code, Unruh says, while only affecting a minimal amount of Credit Karma’s customers.

The years since the introduction of Falcon have prepared the company for new phases to its DevOps strategy, he says, which include multiple instances of services running across multiple datacenters as the company expands. Credit Karma has also invested in another product called Flare built by Unruh’s team. “Flare became the big warehouse,” he says. “Every time something’s deployed, every time an action happens, any time we do a poll request and we’re waiting for reviews — all those metrics on how long things take, how many vulnerabilities happen in a security scan, all that data goes into Flare’s warehouse.”

That gives Credit Karma more visualization into the health of services from a code lifecycle perspective, he says. The plan for 2021 is to integrate Falcon and Flare together, Unruh says. “That will give us true CI/CD [continuous integration/continuous delivery] where the machine is making the decision.”.

Such automation translates into production cycles progressing to a point where human engineers can be hands-off and let the system take over automatically while the engineer turns their attention to other tasks.

Change naturally invites some questions, and Unruh says engineers at Credit Karma early on wanted to understand how Falcon would fulfill all its promises while also being one-size-fits all. He met with leaders within the organization to show early mockups that demonstrated what the experience would be to sell them on it. Unruh also took in feedback during the development of Falcon, which led to snowballing adoption of it by the engineers.

When evaluating how to approach the creation of Falcon, he says they weighed building everything themselves, turning to open source, or other resources to find the right approach to get deliverables out faster. Credit Karma includes a feedback tool embedded in all the products built internally for engineering, Unruh says, to capture details quickly to help teams be more responsive. “That’s been invaluable,” he says, “especially for my teams. It gives us continuous feedback, which is really important if you’re supporting engineering from a DevOps perspective.”

Kubernetes is quickly consolidating its place as the leading container orchestration platform for cloud-native applications, with adoption at 59% among enterprise IT professionals as of March. But while Kubernetes delivers agility, flexibility and scalability for DevOps teams, it also creates complexity that can be an enigma for SecOps teams—especially when something goes wrong.

When it comes to detecting threats and tracking down breaches in Kubernetes, security teams’ key asset is the Kubernetes API server audit log. The audit log captures every successful or unsuccessful API server call, whether by human users, automation pipelines, controllers, operators, system components and, in general, API server operations performed by principals identified as service accounts. In other words, the audit log is the single source of truth for what’s happening inside a Kubernetes infrastructure.

But just because the audit log tracks all the activity in your Kubernetes environment doesn’t make it easy to find what you’re looking for. The larger your infrastructure, the more events there will be to parse through. The Kubernetes audit log is also extremely verbose, capturing a number of identifying attributes for each action, such as detailed information about what resource was accessed, who accessed it or what IP address the request came from. When the Kubernetes infrastructure at hand involves multiple clusters, which is the most common use pattern, it becomes a daunting task to place effective security monitoring controls across all clusters and manual security audit log monitoring becomes extremely impractical.

One simple approach to automate Kubernetes Audit log monitoring, which is quite useful for compliance purposes, is through the use of simple static policies that trigger alerts when a violation occurs. For example, an alert could be triggered if a human user accesses a workload that processes sensitive data. This simplified approach is useful when you have a precise notion, as in cases of regulatory compliance of what use patterns are allowed. However, in real-life deployments, adversaries never limit themselves to what is allowed.

Another approach is threshold-based security monitoring, which captures when API events, or combinations of events, rise above a certain threshold. However, this tactic can be easily bypassed by an adversary simply by controlling the rate of compromise attempts, spreading the attack out over time to avoid crossing the threshold.

Threshold-based based monitoring is also prone to false positives that can really wear out security and teams and create alert fatigue. For example, periodic batch processing workloads, which create a frenzy of API calls, would break through pre-configured thresholds but aren’t actually a threat. To separate threats from routine deployment events, you need to understand the context, something threshold-based monitoring alone cannot provide.

With significant, high-maintenance efforts, and potentially high rates of false alerts, threshold-based monitoring can help security or operations teams to catch known threats and vulnerabilities, but it won’t catch unknown threats or vulnerabilities. Again, to identify a threat, you need to be able to understand the context of any anomalous behavior: who is making the API calls, from where and what resources are they accessing. You also need to observe this data over time to establish what normal behavior looks like so you can differentiate it from anomalous behavior.

As a powerful, elastic and flexible cloud-native application infrastructure that drives many automatic processes, Kubernetes is inherently complex. To detect a security threat in Kubernetes, you need an adaptive security monitoring solution powered by machine learning. Through time-based observation of the various actors within a Kubernetes cluster that leverage the API server audit log, machine learning can learn and detect anomalous patterns in actor activity that simple tools will miss.

Learning the behavior of your clusters is no trivial task. Profiles for different users, components and automated services need to be built over time, and with so many moving parts, this is impractical to do manually. At the same time, new vulnerabilities are constantly coming to light, and there will always be vulnerabilities that are waiting to be uncovered. You can set alerts for vulnerabilities you know, but to catch unknown threats you need a strong baseline of activity in your cluster to monitor against, something that the right machine learning algorithms can readily capture and adapt to over time.

While machine learning is a broad topic, it is quite clear that the traditional approach of supervised or unsupervised offline model creations would not yield satisfying detection results, simply because different environments are accessed and behave differently. There has to be a component that learns and adapts to the specific Kubernetes environment being monitored.

Here are two security use cases that machine learning can reveal through the analysis of the Kubernetes audit log stream:

Stolen Service Account Tokens and User Credentials

Machine learning-based audit log monitoring tools can alert security teams of suspected credential theft. For example, if the same cluster credentials are reused from multiple different geographic or network locations within a short period of time, it can be flagged as an anomalous behavior and, if combined with additional abnormalities, could indicate stolen credentials.

By tracking and learning the geographical attributes and the access patterns of individual users or principals, machine learning can detect this anomalous behavior and flag it for security teams. In response, the security team can take actions to reduce or eliminate the blast radius—such as limiting access to the API server from the specific suspected IP address or modifying the Kubernetes role-based access control (RBAC) policy to reduce access privileges for the specific account.

Misconfigured Kubernetes RBAC

In addition to detecting threat actors that attempt to pivot within the Kubernetes cluster, machine learning can also leverage the audit log to detect attempted exploits of known or unknown vulnerabilities if the RBAC permissions are misconfigured or over-permissive. On the flip side, once RBAC policies are properly configured, reducing RBAC privileges to the required minimum, machine learning can alert when unauthorized users make unsuccessful API calls to access sensitive resources, indicating a failed exploit attempt.

Ultimately, the audit log is an incredibly rich source of information on activity within your Kubernetes environment and a valuable tool for both detecting threats and vulnerabilities and forensically tracing breaches. Although the complexity prohibits manual monitoring, machine learning can give security teams real-time observability into security incidents within their Kubernetes environments and enable informed investigations and audits that get to the root of problems more quickly.

Technologies such as COBOL, IBM mainframes, and older operating systems remain deeply embedded in industries like banking, insurance, and retail. The financial sector’s continued reliance on mainframe infrastructure illustrates how mission-critical these systems remain — even decades after they were first deployed. The challenge for engineering and operations teams is not whether to adopt DevOps, but how to apply DevOps principles incrementally to environments never designed for them.

DevOps and Legacy Systems: Not a Misfit

A common objection is that CI/CD pipelines, Agile practices, and Test-Driven Development only work for greenfield projects. In reality, these practices can be introduced incrementally into legacy environments. The path is not painless, but it is viable. Legacy codebases typically carry significant technical debt, fragile coupling between components, and missing test coverage — all of which make wholesale transformation risky.

A pragmatic approach frames the transformation around five principles that should permeate every change made to the legacy environment:

• Automation — Eliminate repetitive manual steps wherever possible.
• Standardisation — Establish consistent, documented processes for builds, deployments, and environments.
• Shift-left — Move quality and security checks earlier in the development lifecycle.
• Communication — Break down silos between development, testing, and operations teams.
• Feedback — Create continuous loops of measurement and improvement at every stage.

Automated Build and Deployment

Legacy environments frequently lack any formal CI process. Builds are triggered manually, deployment steps are documented (if at all) in tribal knowledge, and there is no branching strategy enforcing code review before changes reach production. This slows release cycles and increases the risk of undetected defects reaching end users.

Introducing version control and a defined branching strategy is typically the first concrete step. Three branching patterns that work well for legacy codebases are:

• Development → Release → Master — Used for ongoing active development.
• Maintenance/Hotfix → Master — Used for emergency bug fixes and critical defect patches.
• Feature Branch → Develop → Release → Master — Used when adding new features that need to be integrated into the main release branch.

Once version control and pull-request review workflows are in place, a CI server (Jenkins, GitLab CI, or a similar tool) can be introduced to automate build compilation and early-stage testing on every commit.

Automate Unit Test Coverage

Legacy codebases developed under Waterfall processes often have little or no automated unit test coverage. Attempting to retrofit comprehensive tests all at once is unrealistic — it introduces risk and consumes capacity that could otherwise go toward feature work. A more sustainable strategy is to improve test coverage incrementally, focusing first on the most critical or most frequently changed modules.

Static analysis tools can provide immediate value without requiring test authorship. SonarQube performs static code analysis to surface bugs, code smells, and potential security vulnerabilities across more than 25 languages, including COBOL and C — making it particularly suited to legacy stacks. Establishing a SonarQube quality gate as part of the CI pipeline gives teams an objective, automated signal on code quality with every build.

Automate Infrastructure Provisioning

In legacy environments, hardware and infrastructure are often tightly coupled and managed entirely by IT operations teams following manual runbooks. This creates bottlenecks whenever developers need a new test environment or need to reproduce a production configuration.

Introducing infrastructure-as-code (IaC) allows teams to define environment configurations in source-controlled files and provision or destroy environments on demand. The initial investment involves codifying what was previously done manually — essentially turning runbooks into machine-executable playbooks. Tools such as Ansible and Chef are well-suited to this transition because they can manage configuration of both modern and legacy systems and integrate into existing IT workflows without requiring a full cloud migration.

Self-service environments reduce the idle time developers spend waiting for infrastructure to be provisioned and ensure that test environments reliably mirror production configuration, reducing the frequency of environment-related defects.

Shift-Left and Automate Test Suites

In traditional legacy development workflows, testing occurs after developers have completed and handed off code — often in a dedicated testing phase late in the release cycle. Defects found at this stage are expensive to fix and cause delays that compound through the pipeline.

Shifting testing left means introducing automated integration, functional, and non-functional tests that run in lower environments (development, staging) as early as possible in the pipeline. Defining a minimum acceptance threshold — a pass rate below which code cannot be promoted to the next environment — forces quality to be addressed continuously rather than deferred.

For teams starting from zero test automation, a pragmatic first step is identifying the five to ten most business-critical user journeys and writing automated acceptance tests for those scenarios. This creates an immediate safety net without requiring a complete test suite before any value is delivered.

Automated Deployment of Code

Deployment automation is not limited to modern application stacks. The principle — moving code through a defined set of environment stages using a repeatable, scripted process — applies equally to legacy applications. Tools such as AWS CodeDeploy, Octopus Deploy, and Azure Pipelines support deployment of a wide variety of application types. Containerisation technologies can also package legacy applications for more portable, consistent deployment across environments.

Even before full end-to-end pipeline automation is achieved, several capabilities should be prioritised:

• Automated smoke tests that run immediately after a deployment to verify basic functionality.
• Automated infrastructure provisioning as part of the deployment pipeline.
• Notifications to development, testing, and QA teams upon deployment completion.
• A deployment dashboard providing the entire team visibility into what was deployed, when, and to which environment.
• A fast, tested rollback mechanism so that bad deployments can be reversed quickly with minimal service impact.

Monitoring Applications in Real Time

Legacy systems frequently lack production monitoring beyond basic server health checks. When incidents occur, teams rely on log trawling and manual root-cause analysis — a slow and unreliable process that extends mean time to recovery (MTTR).

Introducing application performance monitoring (APM) into legacy production environments closes this feedback gap. Tools such as New Relic, Datadog, or open-source alternatives like Prometheus with Grafana can provide real-time visibility into application behaviour, error rates, and latency. With this data, teams can detect regressions introduced by new deployments quickly and correlate production behaviour with specific code changes — accelerating both deployment confidence and incident response.

Monitoring the Entire Development Lifecycle

Application monitoring in production is one part of the picture. DevOps transformation in a legacy context also requires continuous measurement of the development process itself — tracking metrics such as lead time from commit to deployment, defect escape rate, and deployment frequency across all environments.

This lifecycle-level visibility helps teams identify where bottlenecks and quality problems are concentrated — whether in code review, environment setup, testing, or deployment — and provides the data needed to justify and prioritise further automation investments. Legacy system transformation is not a single project with an end date; it is an ongoing discipline of measurement, feedback, and incremental improvement.

Addressing the Mindset and Cultural Challenges

Technical tooling is rarely the hardest part of DevOps adoption in a legacy environment. Teams accustomed to working within established manual workflows frequently underestimate how much time and cognitive load those workflows consume, and can be resistant to change — particularly when the business case is not clearly communicated.

Management buy-in is essential. Leaders need to sponsor the transformation, communicate the rationale, and create space for teams to learn new practices without the pressure of delivering at the same velocity immediately. Framing DevOps adoption in terms of improving developer experience — less manual toil, faster feedback, reduced incident stress — tends to resonate more with engineering teams than purely business-centric arguments.

Identifying quick wins early in the programme builds momentum. A team that experiences a reduction in deployment time or catches a critical defect through an automated test for the first time develops organic advocacy for continued investment.

Is Now the Right Time to Start?

The gap between organisations with mature DevOps practices and those still operating on fully manual legacy processes continues to widen. Digital-native competitors can ship software in hours; organisations constrained by manual legacy release processes may still measure releases in weeks or months.

The good news is that organisations running legacy systems often hold significant advantages that digital-native companies lack — decades of domain data, established customer relationships, and deep operational experience. Combining these strengths with DevOps practices gives legacy-heavy organisations a credible path to competing effectively in the digital market.

The transformation does not require a single large-scale programme. Starting with a single team, a single application, or even a single pipeline stage — and iterating from there — is a proven and lower-risk approach. The most important step is the first one.

Training a machine learning model is the part data scientists tend to enjoy. You gather a large, labeled dataset, choose an algorithm, tune hyperparameters, and measure accuracy. The hard part begins afterward: integrating that model—often written in Python or R—into a production system built in a completely different language, serving real users, and expected to behave reliably under load.

The gap between a working notebook and a production-grade ML system is where projects most frequently fail. A recommendation engine bolted onto a medical records system with ad-hoc glue code is a recipe for operational debt. The four considerations below are a practical checklist for teams preparing to ship an ML system to production.

1. Requirements Readiness: Does the System Actually Meet Its Goals?

Before promoting any model to production, establish whether it satisfies the original requirements. This sounds obvious, but ML systems make it easy to confuse a well-trained model with a production-ready one. Ask:

• What are the acceptance criteria? Accuracy, precision, recall, AUC, or business-level KPIs?
• Has the model been evaluated against a held-out test set that was never used during training or hyperparameter tuning?
• Have you defined and run integration tests that cover both the ML component and the surrounding application code?
• Are there documented requirements for maintenance releases, not just the initial launch?

Without clear, measurable requirements, there is no objective way to declare the system ready—or to detect when it regresses.

2. Real-World Output Validation: Is the System Producing Expected Results?

A model trained on historical data may perform well on holdout sets but behave unexpectedly against live production data. Factors such as data drift, schema changes in upstream sources, or edge cases not represented in training data can all cause silent degradation.

Recommendations for ongoing output validation include:

• Shadow mode testing: Run the new model in parallel with the existing system and compare outputs before committing to a full cutover.
• Continuous evaluation in production: Log predictions and periodically compare them against ground truth as it becomes available (e.g., did the patient actually have the predicted condition?).
• Alerting on output distribution shifts: Monitor the statistical distribution of model outputs. A sudden shift can indicate a data pipeline problem or model decay even before accuracy metrics confirm it.

The goal is to move beyond a one-time validation event and treat model quality as an ongoing operational metric.

3. Performance and Scalability: Will It Hold Up Under Real Load?

ML inference can be computationally expensive, and systems that perform well in a single-user development environment can collapse under concurrent production load. Performance testing for ML systems should address:

• Latency under concurrency: What is the p50, p95, and p99 inference latency at realistic concurrent user counts?
• Throughput limits: What is the maximum requests-per-second the system can handle before latency degrades beyond acceptable thresholds?
• Real-time and embedded constraints: Systems with hard real-time requirements—such as embedded control software—need guaranteed response times. Functional or interpreted languages commonly used in ML may not provide the deterministic latency these environments require without careful architectural choices (e.g., compiling models to native code, using ONNX, or serving via a high-performance runtime).
• Horizontal scalability: Can the inference service be scaled out by adding instances, or does it have shared state that creates bottlenecks?

Load testing should be a gate in the deployment pipeline, not an afterthought discovered at launch.

4. Reliability and Acceptable Output Variance: How Correct Does It Need to Be?

Unlike traditional software where a function either returns the right answer or it does not, ML systems produce probabilistic outputs. This requires a different approach to reliability. Define upfront:

• Acceptable error bounds: Is a prediction within 5% of the true value acceptable, or must results be exact?
• Handling of outlier predictions: What happens when the model produces a clearly implausible result? Should the system fall back to a rule-based default, surface a warning, or block the output entirely?
• Failure modes: What does the system do if the model service is unavailable? A graceful degradation strategy prevents a slow ML backend from taking down the entire application.
• Auditability: In regulated industries such as healthcare or finance, the ability to explain and audit individual predictions is often a hard requirement, not a nice-to-have.

Bridging the Gap with DevOps Practices

The four considerations above are easier to manage when DevOps principles are applied to the ML lifecycle—a practice increasingly called MLOps.

Unified Version Control and Continuous Integration

A common failure pattern is maintaining the ML model code and the application integration code in separate repositories with independent release cycles. This makes it difficult to reproduce a known-good state or roll back a bad deployment. Treating model artifacts, training code, feature pipelines, and application code as a unified versioned system—built and tested together in a CI pipeline (for example, using Jenkins, GitHub Actions, or GitLab CI)—eliminates this ambiguity. Every deployment becomes a traceable, reproducible artifact.

Automated Deployment and Rollback

Manual deployments of ML systems are error-prone, particularly when coordinating changes across a model serving layer and an application layer. Automated deployment pipelines with built-in rollback capability mean that when a bad model version reaches production—and eventually one will—the recovery path is a single action rather than a manual debugging session under pressure.

Production Monitoring and Model Decay Detection

Standard application monitoring covers infrastructure metrics: CPU, memory, error rates, and latency. ML systems require an additional layer: monitoring the quality of model outputs over time. As the real-world distribution of inputs shifts away from the training distribution (a phenomenon called data drift or concept drift), model performance can silently degrade. Automated alerts triggered by output quality metrics allow teams to catch decay early and trigger a retraining or rollback before users are significantly affected.

Summary

Deploying a machine learning system to production is an engineering discipline that extends well beyond model accuracy. The four core considerations—requirements readiness, real-world output validation, performance and scalability, and reliability—provide a structured framework for evaluating production readiness. Applying DevOps practices such as unified CI/CD pipelines, automated rollback, and continuous quality monitoring transforms a fragile, manually-managed ML deployment into a maintainable, observable production system.