Managing Multi-Cloud Production Kubernetes Clusters at Brandwatch

Originally reported by InfoQ based on an interview with Andy Hume, Director of Engineering, Application Infrastructure at Brandwatch. Source: infoq.com

Overview: Six Clusters Across Three Environments

Brandwatch runs approximately 150 production services across six independent Kubernetes clusters. Those clusters span three distinct environments: Amazon EKS, Google Kubernetes Engine (GKE), and self-managed clusters hosted in their own data centers in London. The team began their Kubernetes journey on GKE in early 2016, finding it a strong fit for many of their services before expanding to a true multi-cloud, multi-site setup.

Operating at this scale across heterogeneous infrastructure surfaces challenges that teams running a single managed cluster rarely encounter: divergent control-plane APIs, inconsistent networking models, varied upgrade cadences, and the need to keep CI/CD tooling cluster-agnostic. The following sections cover how Brandwatch’s team has addressed each of these areas.

Managing Self-Hosted Clusters with Rancher

While EKS and GKE abstract away control-plane management, Brandwatch still operates self-managed Kubernetes clusters in their own data centers. For these, the team uses Rancher 2 to provision and manage the clusters. Their familiarity with Rancher goes back to 2015–2016 when they ran Rancher 1 before Kubernetes became their orchestration layer. When Rancher 2 pivoted to a Kubernetes-native model, it was a natural progression.

That said, Hume noted that Rancher 2 introduces complexity, and as the team plans to scale Kubernetes across more bare-metal infrastructure they are evaluating solutions that offer lower operational overhead and tighter integration with their existing network design.

Developer Workflow: Freedom at the Edges, Standards at the Core

Brandwatch’s platform team deliberately avoids mandating specific local development tools. Individual teams have adopted different approaches depending on their preferences, including:

• Skaffold — for iterative local development with continuous build and deploy loops against a live cluster.
• Telepresence — for proxying a local process into a remote cluster, useful when services have many dependencies.
• Docker Compose — for teams preferring a fully local, container-based environment without a cluster dependency.

This flexibility is intentional. Forcing a single local development workflow across diverse teams with different stacks tends to create friction rather than efficiency. The platform team focuses its standardization effort where it matters most: the CI/CD pipeline and deployment artifacts.

CI/CD Pipeline: ConcourseCI, Declarative Manifests, and GitOps-Style Deployments

All teams publish container images and associated metadata to a central system that manages deployments and change control. The CI/CD layer runs on ConcourseCI, which itself executes on Kubernetes. Around this foundation, the team has built custom tooling that automates the deployment lifecycle.

Toolchain Breakdown

• Custom kubectl wrapper — A declarative YAML-based wrapper around kubectl that captures service metadata and standardizes how resources are applied.
• Helm — Used for templating Kubernetes manifests, but not for executing production rollouts. This avoids Helm’s release management complexity in production while still benefiting from its templating capabilities.
• Kustomize — Used to manage all cluster-level (platform) services, allowing environment-specific overrides without duplicating base manifests.

Manifests as the Single Source of Truth

Brandwatch keeps all Kubernetes manifests in a repository separate from application source code. This separation simplifies the mechanics of rollouts and rollbacks considerably. When a team deploys a new image, they update the image reference in the manifest repository. The manifests in Git are the authoritative source of truth — rolling back an application is as straightforward as deploying an older manifest revision, which also reverts the image tag automatically.

This approach shares principles with GitOps patterns (as popularized by tools like Flux and Argo CD), though Brandwatch has implemented it with custom tooling rather than adopting those specific platforms.

Ingress: Choosing nginx Over Managed Load Balancers

Both GKE and EKS offer native Ingress implementations backed by their respective managed load balancers (GCP’s HTTP(S) Load Balancer and AWS ALB). Despite this, Brandwatch runs the nginx Ingress controller on all six clusters — including the managed ones — and bypasses the cloud-native load balancers for HTTP traffic routing.

The primary reasons are operational consistency and control over HTTP headers:

• Security headers — The team can centrally add security-specific response headers (e.g., Strict-Transport-Security, X-Content-Type-Options) and strip internal headers without relying on each application to implement this correctly. The security team has a single, auditable location for these policies.
• Cloud LB limitations — The GCP HTTP Load Balancer imposes size limits on incoming HTTP request headers. This caused compatibility issues with some of Brandwatch’s legacy applications being migrated into Kubernetes, making the managed LB a blocker.
• Cross-cluster consistency — A uniform nginx configuration across all clusters reduces cognitive overhead when debugging ingress behavior or applying policy changes.

Monitoring and Alerting with Prometheus

Each cluster runs its own Prometheus instance, deployed via the Prometheus Operator. Alerting responsibility is distributed: individual development teams are responsible for creating and responding to alerts for their own services using the Prometheus and Alertmanager stack within each cluster.

The primary alerting focus is availability and correctness of customer-facing applications. In addition, the health of Kubernetes workloads (pod restarts, deployment failures, resource pressure) is monitored as a proxy for overall cluster health.

Notably, the team does not proactively alert on node-level metrics for their managed (EKS and GKE) node pools. Their operational model for a failing node is pragmatic: terminate it and let the cluster autoscaler provision a replacement. Hume described the managed node pools on GKE and EKS as resilient and straightforward to manage in failure scenarios.

Autoscaling: Working Around Cluster Autoscaler Latency

Brandwatch uses the Kubernetes Cluster Autoscaler to provision new nodes when workloads demand it. However, the team found the autoscaler’s response time too slow for burst workloads — by the time a new node is ready and pods are scheduled, latency-sensitive or throughput-sensitive work may already be failing or queuing excessively.

They address this with two complementary strategies:

• Application-level resilience — Services are built with queue-and-retry logic so that work is not dropped while new nodes and pods are spinning up. The application absorbs the latency rather than failing hard.
• Predictive pre-scaling — For known workload spikes (scheduled batch jobs, anticipated traffic peaks), the team pre-scales the cluster ahead of time, avoiding the reactive autoscaler lag entirely.

Key Takeaways for Multi-Cloud Kubernetes Operations

Brandwatch’s architecture offers several transferable lessons for teams managing Kubernetes at scale across multiple clouds or on-premises environments:

• Standardize the deployment contract, not the developer experience. Mandate how artifacts are published and deployed; leave local tooling choices to teams.
• Separate application manifests from application source. This makes rollbacks deterministic and keeps Git the authoritative source of cluster state.
• Use Helm for templating, not release management, in production. Kustomize handles environment-level customization without Helm’s operational overhead.
• Prefer a consistent ingress layer across all clusters. nginx Ingress provides uniform behavior and centralized policy enforcement that cloud-native LBs may not.
• Distribute monitoring responsibility, but standardize the stack. Per-cluster Prometheus with team-owned alerts scales better than a centralized alerting model for large organizations.
• Design applications to tolerate autoscaler lag. Cluster Autoscaler is not instantaneous; queue-and-retry patterns and predictive scaling fill the gap.