Kubernetes adoption has grown dramatically, but operational complexity remains one of the biggest barriers to confident day-to-day use. One area that consistently trips up engineering teams is alerting: out-of-the-box alert configurations either generate a flood of false positives or miss genuine problems entirely. The root cause is that Kubernetes is a highly dynamic, self-healing platform — static thresholds designed for traditional infrastructure rarely translate well.
The five strategies below address the most common alerting pain points in Kubernetes environments, covering everything from API-level monitoring to kube-system internals.
1. Detect Application Issues by Tracking the API Gateway for Microservices
Because Kubernetes is highly abstracted from the physical layer, low-level metrics such as raw CPU usage and memory consumption are poor early-warning indicators on their own. When a single pod begins to strain, the Kubernetes auto-healing mechanism will typically scale out or restart that pod before a human-defined CPU threshold is ever breached. Alerting on those thresholds therefore produces noise without actionable signal.
A more reliable approach is to monitor your API gateway and ingress controllers. API-level metrics — error rates, request latency, and request throughput — reflect the actual health of a service from the consumer’s perspective. A rising error rate or increasing p99 latency almost always precedes the infrastructure-level symptoms that traditional alerts are tuned to catch.
Keep these considerations in mind:
- Baselines vary widely — each pod, cluster, region, and data centre can have a legitimately different normal pattern. Static thresholds cannot account for this variance.
- Ingress controllers are your friend — controllers such as NGINX Ingress or Traefik expose service-level metrics that can be scraped by Prometheus and used to create granular, per-service alerts.
- Anomaly detection over fixed thresholds — because baselines shift continuously, threshold-free anomaly detection (whether rules-based or ML-assisted) is better suited to catching real degradation events.
2. Stop Measuring Individual Containers in Isolation
One of Kubernetes’ defining features is that it creates, destroys, and reschedules containers continuously. An individual container’s resource consumption metrics are therefore short-lived and highly variable — making per-container alerting both noisy and misleading. By the time an alert fires on a specific container, that container may already have been replaced.
The more useful level of abstraction is the full container set — for example, all replicas of a given Deployment or DaemonSet. Aggregate metrics across the group reveal trends that individual container metrics obscure.
cAdvisor (Container Advisor), which is embedded in the kubelet, automatically collects per-container resource usage data including CPU, memory, filesystem, and network I/O. When these metrics are aggregated at the Deployment or namespace level in Prometheus and visualised in Grafana, patterns become far clearer.
The one exception to avoiding per-container metrics is outlier detection: if a single container within a replica set is consuming significantly more memory than its peers, that is a signal worth investigating. Tools that support statistical outlier detection can surface this without requiring you to set a fixed threshold for each container individually.
3. Distinguish Transient Hiccups from Systemic Trends
Under normal operating conditions, pods fail. Kubernetes is designed around this assumption — it restarts failed pods automatically. The operational question is not whether a pod failed, but why it failed and whether that failure is recurring.
Every Kubernetes pod carries a status field that reports its position in the pod lifecycle. Possible states include:
- Pending — the pod has been accepted but is not yet running (often due to insufficient resources or an image pull issue).
- Running — the pod is bound to a node and at least one container is running.
- Succeeded — all containers exited successfully and will not be restarted.
- Failed — all containers have terminated and at least one exited with a non-zero status.
- Unknown — the node hosting the pod cannot be reached.
When a pod enters the Failed state, you can query the reason — for example, OOMKilled indicates the container exceeded its memory limit. The actionable insight comes from aggregating these reasons over time. A single OOMKill is noise; a recurring pattern of OOMKills across a service is a signal that memory limits need adjustment or that there is a memory leak.
Set up your monitoring stack to track pod failure reasons as labelled metrics. Alert not on individual failures, but on failure rates or recurring failure patterns within a time window.
4. Never Ignore High Disk Usage
High disk usage is one of the few Kubernetes alerts that should always be treated as urgent. Unlike CPU or memory pressure, which Kubernetes can partially compensate for through scheduling and limits, a full disk can cause silent, cascading failures: log writes fail, container image pulls fail, and etcd — which stores all cluster state — becomes unable to commit new entries.
Most teams already set a disk usage alert at 75–80% capacity, which is sensible. However, a static threshold only tells you that you have a problem — it gives you no lead time to fix it. A better approach is to alert on the rate of change in disk usage. A node that jumps from 40% to 70% disk usage overnight is more concerning than one that has been sitting at 72% for weeks.
Pay particular attention to disk usage in StatefulSets. If you have containerised databases or other stateful workloads, disk exhaustion in a StatefulSet’s Persistent Volume can cause data loss or corruption — not just a service restart.
Useful Prometheus metrics for disk monitoring include kubelet_volume_stats_used_bytes and node_filesystem_avail_bytes.
5. Always Monitor the kube-system Namespace
The kube-system namespace hosts the control plane components that make everything else work: CoreDNS, kube-proxy, the API server, the scheduler, and etcd. Problems here affect every workload running on the cluster. Despite this, many teams focus their alerting exclusively on application namespaces and treat kube-system as a black box.
Most kube-system components expose a /metrics endpoint that can be scraped by Prometheus. In addition to the general resource usage metrics discussed in tip 2, monitor the following component-specific KPIs:
DNS KPIs (CoreDNS)
- Total DNS request rate — a sudden spike can indicate a resource bottleneck, a scaling limit being approached, or an application bug generating excessive DNS lookups (such as resolving the same hostname in a tight loop).
- DNS request latency — elevated DNS latency degrades user-facing response times and can break internal service discovery. CoreDNS exposes histogram metrics that make p99 latency easy to alert on. Common Prometheus metrics:
coredns_dns_requests_totalandcoredns_dns_request_duration_seconds.
etcd KPIs
- Failed proposals (
etcd_server_proposals_failed_total) — etcd uses a Raft consensus algorithm. Failed proposals indicate that the cluster is struggling to reach agreement, often due to network issues or an unhealthy member. Alert early: etcd can tolerate losing a minority of members, but too many failures lead to unrecoverable downtime. - Snapshot duration (
etcd_debugging_snap_save_total_duration_seconds) — etcd periodically snapshots its state to disk. Slow snapshots are an early indicator of disk I/O problems on the node hosting etcd. - Message send failures (
etcd_network_peer_sent_failures_total) — rising peer message failures suggest network degradation between etcd members. Left unchecked, this can split the cluster and halt all API server operations.
Conclusion
Effective Kubernetes alerting is not about monitoring everything — it is about monitoring the right things at the right level of abstraction. Static thresholds and per-container metrics are a poor fit for a dynamic, self-healing platform. Shifting your focus to API-level signals, aggregate container metrics, pod failure trends, disk usage rate-of-change, and kube-system health will give you earlier, more actionable warnings with significantly less noise.
The monitoring stack most commonly used for this purpose in the Kubernetes ecosystem is the Prometheus + Alertmanager + Grafana combination, often deployed via the kube-prometheus project, which includes a curated set of default alerts and dashboards covering many of the KPIs described above. Whether you use that stack or a managed observability platform, the principles here apply regardless of tooling.