This article is based on a first-person account from the engineering team at Descartes Labs, originally published on TechBeacon.
Background: Geospatial Machine Learning at Scale
At Descartes Labs, the engineering team built a data refinery to collect, process, and analyze sensor data to quantify physical changes on Earth. The platform runs on Kubernetes in Google Cloud, powering global-scale machine learning across more than 10 petabytes of geospatial data.
At any given moment, tens of thousands of CPUs may be extracting meaning from complex geospatial datasets, with models generating insights within minutes of satellites passing overhead — including near-real-time wildfire detection workloads.
Supporting a rapidly evolving API platform with time-sensitive components creates real pressure on the delivery pipeline. The lessons the team learned along the way offer a practical blueprint for any organization running mission-critical workloads on Kubernetes.
Empower Your Application Developers
Descartes Labs has approximately 30 software engineers and three site reliability engineers (SREs). With limited SRE capacity, the team quickly identified three scaling problems with a traditional, SRE-centric deployment model:
- Manually creating deployment pipelines for each application is error-prone and does not scale.
- Placing SREs in the critical path for configuring application-specific pipelines creates a bottleneck.
- Making SREs responsible for day-to-day pipeline operations is unsustainable at scale.
The solution was to adopt Spinnaker, an open-source continuous delivery platform that supports multiple cloud providers and has rich, actively developed support for Kubernetes via its V2 provider. Spinnaker gave the team a foundation to build self-service tooling without a large engineering investment.
Pipeline Templates and Pipelines as Code
Spinnaker pipelines are defined in JSON and can be created or updated programmatically using the spin CLI. However, a one-size-fits-all pipeline template rarely works in practice: different applications may require schema migrations, ConfigMap changes, custom environment variables, and varied rollout strategies.
The Descartes Labs team addressed this by building pipeline templates using Jinja, the Python-based open-source templating language. By establishing conventions around application naming and Kubernetes namespacing, they could render complete Spinnaker pipeline definitions from lightweight, per-application JSON configuration files.
This produced a clean ownership model:
- Jinja templates are managed by the SRE team and protected via code owners in version control.
- Configuration files live alongside application code, giving application developers full ownership without SRE involvement.
Configuring a new application with the standard pipeline suite — target cluster, horizontal pod autoscaler settings, container image prefix, and basic environment variables — requires fewer than 20 lines of JSON. The application team can review and merge this configuration entirely independently.
The team notes that their Jinja-based approach reflected the maturity of Spinnaker at the time they built it. Today, Managed Pipeline Templates and Sponnet offer more native, first-class templating capabilities within Spinnaker and may be preferable starting points for new adopters.
Pushing Operational Responsibility to Application Developers
Application developers understand their own services best. Shifting pipeline operational responsibility to them — rather than concentrating it in the SRE team — improves both responsiveness and scaling. Spinnaker provides several features that make this shift practical and safe:
- Authorization: Pipeline execution can be restricted to specific teams, preventing accidental cross-team deployments.
- Audit trail: Every manual trigger or approval is attributed to a specific user, giving clear accountability.
- Rich diagnostics: Developers can view deployment status, pod health, and container logs directly within the Spinnaker UI, without needing direct access to the Kubernetes cluster.
That last point is significant: Spinnaker’s diagnostics allow teams to restrict developer access to production Kubernetes clusters while preserving the visibility engineers need to diagnose incidents independently.
The Value of High-Velocity Continuous Delivery
A common assumption is that deploying more frequently introduces more errors. The Descartes Labs team found the opposite: higher deployment frequency corresponds to smaller, better-understood changes that are easier to test, debug, and roll back if needed.
The team practices trunk-based development, triggering deployments automatically on merge to the main branch. Core applications can be deployed multiple times per day, with a lead time for changes — the interval from code merge to receiving production traffic — of under one hour.
Key characteristics of this model:
- Each deployment carries a small, well-scoped diff, making the blast radius of any defect smaller.
- Unit and integration tests run against production, so regressions surface quickly.
- When a change does introduce a problem, the short lead time means a fix can often be deployed faster than a rollback would resolve it — the team generally rolls forward rather than back.
- A one-click rollback pipeline is included in every application’s Spinnaker template as a safety net.
This experience aligns with broader industry research. The Accelerate: State of DevOps Report (published by DORA) has consistently found that higher deployment frequency and shorter lead times correlate with lower change failure rates and faster recovery from incidents — not higher instability.
A Path to Fully Automated Production Deployment with Canary Analysis
Canary deployments gradually shift traffic to a new application version, allowing teams to validate behavior against real production load before completing the rollout. Spinnaker has native support for automated canary analysis, comparing metrics between the canary and baseline deployments to make a pass/fail decision automatically.
However, standard Kubernetes service routing distributes traffic evenly across all pods regardless of version. This makes fine-grained, percentage-based traffic splitting — a prerequisite for meaningful canary analysis — difficult to achieve natively.
Descartes Labs addressed this by integrating Istio, the open-source service mesh, into their Spinnaker and Kubernetes stack. Istio improves the canary story in two concrete ways:
- Traffic control independent of pod count: Istio’s traffic management APIs allow Spinnaker to shift specific percentages of traffic between the canary and baseline deployments without needing to scale pod counts to match the desired traffic ratio.
- Layer 7 metrics out of the box: Istio automatically exposes HTTP-level metrics — response codes, request latencies, error rates — that Spinnaker’s automated canary analysis engine can use as signals to approve or reject a canary promotion.
Together, these capabilities enable a fully automated promotion path: a new build triggers a canary deployment, Istio routes a small percentage of production traffic to it, Spinnaker evaluates the resulting metrics, and the deployment either promotes automatically or halts without human intervention.
Key Takeaways for Your Organization
The Descartes Labs approach demonstrates that safe, high-velocity continuous delivery is achievable even with a small SRE team, if the architecture is designed around self-service and developer empowerment. The core lessons are:
- Build self-service pipeline infrastructure. Use templates and configuration-as-code so application teams can onboard and operate their own pipelines without SRE involvement in every change.
- Push pipeline ownership to developers. Tools like Spinnaker provide the authorization, audit, and diagnostic features needed to do this safely.
- Deploy frequently and in small batches. Trunk-based development with short lead times reduces risk per deployment and makes recovery fast.
- Use a service mesh to unlock advanced canary analysis. Istio’s traffic management and L7 metrics make automated, metrics-driven canary promotion practical on Kubernetes.
- Pair rollback tooling with a roll-forward culture. One-click rollback pipelines are a safety net, but small, frequent changes make rolling forward often the faster and cleaner option.
Combining Kubernetes, Istio, and Spinnaker provides a deployment ecosystem that balances safety with velocity — and scales across growing engineering teams without proportionally growing SRE headcount.
Original source: TechBeacon. This article is based on a first-person account from the Descartes Labs engineering team.