Kubernetes is quickly consolidating its place as the leading container orchestration platform for cloud-native applications, with adoption at 59% among enterprise IT professionals as of March. But while Kubernetes delivers agility, flexibility and scalability for DevOps teams, it also creates complexity that can be an enigma for SecOps teams—especially when something goes wrong.

When it comes to detecting threats and tracking down breaches in Kubernetes, security teams’ key asset is the Kubernetes API server audit log. The audit log captures every successful or unsuccessful API server call, whether by human users, automation pipelines, controllers, operators, system components and, in general, API server operations performed by principals identified as service accounts. In other words, the audit log is the single source of truth for what’s happening inside a Kubernetes infrastructure.

But just because the audit log tracks all the activity in your Kubernetes environment doesn’t make it easy to find what you’re looking for. The larger your infrastructure, the more events there will be to parse through. The Kubernetes audit log is also extremely verbose, capturing a number of identifying attributes for each action, such as detailed information about what resource was accessed, who accessed it or what IP address the request came from. When the Kubernetes infrastructure at hand involves multiple clusters, which is the most common use pattern, it becomes a daunting task to place effective security monitoring controls across all clusters and manual security audit log monitoring becomes extremely impractical.

One simple approach to automate Kubernetes Audit log monitoring, which is quite useful for compliance purposes, is through the use of simple static policies that trigger alerts when a violation occurs. For example, an alert could be triggered if a human user accesses a workload that processes sensitive data. This simplified approach is useful when you have a precise notion, as in cases of regulatory compliance of what use patterns are allowed. However, in real-life deployments, adversaries never limit themselves to what is allowed.

Another approach is threshold-based security monitoring, which captures when API events, or combinations of events, rise above a certain threshold. However, this tactic can be easily bypassed by an adversary simply by controlling the rate of compromise attempts, spreading the attack out over time to avoid crossing the threshold.

Threshold-based based monitoring is also prone to false positives that can really wear out security and teams and create alert fatigue. For example, periodic batch processing workloads, which create a frenzy of API calls, would break through pre-configured thresholds but aren’t actually a threat. To separate threats from routine deployment events, you need to understand the context, something threshold-based monitoring alone cannot provide.

With significant, high-maintenance efforts, and potentially high rates of false alerts, threshold-based monitoring can help security or operations teams to catch known threats and vulnerabilities, but it won’t catch unknown threats or vulnerabilities. Again, to identify a threat, you need to be able to understand the context of any anomalous behavior: who is making the API calls, from where and what resources are they accessing. You also need to observe this data over time to establish what normal behavior looks like so you can differentiate it from anomalous behavior.

As a powerful, elastic and flexible cloud-native application infrastructure that drives many automatic processes, Kubernetes is inherently complex. To detect a security threat in Kubernetes, you need an adaptive security monitoring solution powered by machine learning. Through time-based observation of the various actors within a Kubernetes cluster that leverage the API server audit log, machine learning can learn and detect anomalous patterns in actor activity that simple tools will miss.

Learning the behavior of your clusters is no trivial task. Profiles for different users, components and automated services need to be built over time, and with so many moving parts, this is impractical to do manually. At the same time, new vulnerabilities are constantly coming to light, and there will always be vulnerabilities that are waiting to be uncovered. You can set alerts for vulnerabilities you know, but to catch unknown threats you need a strong baseline of activity in your cluster to monitor against, something that the right machine learning algorithms can readily capture and adapt to over time.

While machine learning is a broad topic, it is quite clear that the traditional approach of supervised or unsupervised offline model creations would not yield satisfying detection results, simply because different environments are accessed and behave differently. There has to be a component that learns and adapts to the specific Kubernetes environment being monitored.

Here are two security use cases that machine learning can reveal through the analysis of the Kubernetes audit log stream:

Stolen Service Account Tokens and User Credentials

Machine learning-based audit log monitoring tools can alert security teams of suspected credential theft. For example, if the same cluster credentials are reused from multiple different geographic or network locations within a short period of time, it can be flagged as an anomalous behavior and, if combined with additional abnormalities, could indicate stolen credentials.

By tracking and learning the geographical attributes and the access patterns of individual users or principals, machine learning can detect this anomalous behavior and flag it for security teams. In response, the security team can take actions to reduce or eliminate the blast radius—such as limiting access to the API server from the specific suspected IP address or modifying the Kubernetes role-based access control (RBAC) policy to reduce access privileges for the specific account.

Misconfigured Kubernetes RBAC

In addition to detecting threat actors that attempt to pivot within the Kubernetes cluster, machine learning can also leverage the audit log to detect attempted exploits of known or unknown vulnerabilities if the RBAC permissions are misconfigured or over-permissive. On the flip side, once RBAC policies are properly configured, reducing RBAC privileges to the required minimum, machine learning can alert when unauthorized users make unsuccessful API calls to access sensitive resources, indicating a failed exploit attempt.

Ultimately, the audit log is an incredibly rich source of information on activity within your Kubernetes environment and a valuable tool for both detecting threats and vulnerabilities and forensically tracing breaches. Although the complexity prohibits manual monitoring, machine learning can give security teams real-time observability into security incidents within their Kubernetes environments and enable informed investigations and audits that get to the root of problems more quickly.

Last year, the Cloud Native Computing Foundation (CNCF) initiated a process of conducting third-party security audits for its own projects. The aim of these security audits was to improve the overall security of the CNCF ecosystem.

CoreDNS, Envoy and Prometheus are some of the CNCF projects which underwent these audits, resulting in identification of several security issues and vulnerabilities in the projects. With the help of the audit results, CoreDNS, Envoy and Prometheus addressed their security issues and later, provided users with documentation for the same.

CNCF CTO Chris Aniszczyk says “The main takeaway from these initial audits is that a public security audit is a great way to test the quality of an open source project along with its vulnerability management process and more importantly, how resilient the open source project’s security practices are.” He has also announced that, later this year, CNCF will initiate a bounty program for researchers who identify bugs and other cybersecurity shortcomings in their projects.

After tasting initial success, CNCF formed a Security Audit Working Group to provide security audits to their graduated projects, using the funds provided by the CNCF community. CNCF’s graduated projects include Kubernetes, Envoy, Fluentd among others. Due to the complexity and wide scope of the project, the Working group appointed two firms called the Trail of Bits and Atredis Partners to perform Kubernetes security audits. Trail of Bits implements high-end security research to identify security vulnerabilities and reduce risk and strengthen the code. Similarly, Atredis Partners also does complex and research-driven security testing and consulting.

Kubernetes security audit findings

Three days ago, the Trail of Bits team released an assessment report called the Kubernetes Security Whitepaper, which includes all the key aspects of the Kubernetes attack surface and security architecture. It aims to empower administrators, operators, and developers to make better design and implementation decisions. The Security Whitepaper presents a list of potential threats to Kubernetes cluster.

Kubernetes cluster vulnerabilities

A Kubernetes cluster consists of several base components such as kubelet, kube-apiserver, kube-scheduler, kube-controller-manager, and a kube-apiserver storage backend. Components like controllers and schedulers in Kubernetes assist in networking, scheduling, or environment management. Once a base Kubernetes cluster is configured, the Kubernetes clusters are managed by operator-defined objects. These operator-defined objects are referred as abstractions, which represents the state of the Kubernetes cluster. To provide an easy way of configuration and portability, the abstractions also include the component-agnostic. This again increases the operational complexity of a Kubernetes cluster.

Since Kubernetes is a large system with many functionalities, the security audit was conducted on selected eight components within the larger Kubernetes ecosystem:

• Kube-apiserver
• Etcd
• Kube-scheduler
• Kube-controller-manager
• Cloud-controller-manager
• Kubelet
• Kube-proxy
• Container Runtime

The Trail of Bits team firstly identified three types of attackers within a Kubernetes cluster:

• External attackers (who did not have access to the cluster)
• Internal attackers (who had transited a trust boundary)
• Malicious Internal users (who abuse their privilege within the cluster)

The security audits resulted in total 37 findings, including 5 high severity, 17 medium severity, 8  low severity and 7 informational in the access control, authentication, timing, and data validation of a Kubernetes cluster. Some of the findings include:

• Insecure TLS is in use by default
• Credentials are exposed in environment variables and command-line arguments
• Names of secrets are leaked in logs
• No certificate revocation
• seccomp is not enabled by default

Recommendations for Kubernetes cluster administrators and developers

The Trail of Bits team have proposed a list of best practices and guideline recommendations for cluster administrators and developers.

Recommendations for cluster administrators

• Attribute Based Access Controls vs Role Based Access Controls: Role-Based Access Controls (RBAC) can be configured dynamically while a cluster is operational. In contrast, Attribute Based Access Control (ABAC) are static in nature. This increases the difficulty of ensuring proper deployment and enforcement of controls.
• RBAC best practices: Administrators are advised to test their RBAC policies to ensure that the policies defined on the cluster are backed by an appropriate component configuration and that the policies properly restrict behavior.
• Node-host configurations and permissions: Appropriate authentication and access controls should be in place for the cluster nodes as an attacker with network access can use Kubernetes components to compromise other nodes.
• Default settings and backwards compatibility: Kubernetes contains many default settings which negatively impact the security of a cluster. Hence, cluster operators and administrators must ensure that the component and workload settings are rapidly changed and redeployed, in case of a compromise or an update.
• Networking: Due to the complexity of Kubernetes networking, there are many recommendations for maintaining a secure network. Some of them include: proper segmentation, isolation rules of the underlying cluster hosts should be defined. An executing control-plane components host should be isolated to the greatest extent possible.
• Environment considerations: The security of a cluster’s operating environment should be addressed. If a cluster is hosted on a cloud provider, administrators should ensure that best-practice hardening rules are implemented.
• Logging and alerting: Centralized logging of both workload and cluster host logs is recommended to enable debugging and event reconstruction.

Recommendations for developers

• Avoid hardcoding paths to dependencies: Developers are advised to be conservative and cautious when handling external paths. Users should be warned if a path was not found, and have an option to specify it through a configuration variable.
• File permissions checking: Kubernetes should provide users the ability to perform file permissions checking, and enable this feature by default. This will help prevent common file permissions misconfigurations and help promote more secure practices.
• Monitoring processes on Linux: A Linux process is uniquely identified in the user-space via a process identifier or PID. A PID will point to a given process as long as the process is alive. If it dies, the PID can be reused by another spawned process.
• Moving processes to a cgroup: While moving a given process to a less restricted cgroup, it is necessary to validate that the process is the correct process after performing the movement.
• Future cgroup considerations for Kubernetes: Both Kubernetes and the components it uses (runc, Docker) have no support for cgroups. Currently, it is not an issue, however, it would be good to track this topic as it might change in the future.
• Future process handling considerations for Kubernetes: Tracking and participating in the development of a processes (or threads) on Linux is highly recommended.

Kubernetes security audit sets precedent for other open source projects

By conducting security audits and open sourcing the findings, Kubernetes, a widely used container-orchestration system, is setting a great precedent to other projects. This shows Kubernetes’ interest in maintaining security in their ecosystem. Though the number of security flaws found in the audit may upset a Kubernetes developer, it surely assures them that Kubernetes is trying its best to stay ahead of potential attackers. The Security Whitepaper and the threat model, provided in the security audit is expected to be of great help to Kubernetes community members for future references.

Developers have also appreciated CNCF for undertaking great efforts in securing the Kubernetes system.

Background: Why the CNCF Commissioned a Kubernetes Security Audit

On 6 August 2019, the Cloud Native Computing Foundation (CNCF) published a blog post detailing the results of a formal, third-party security audit of Kubernetes — the foundation’s flagship project and the most widely adopted open-source container orchestration platform. The CNCF had previously piloted its security audit programme with three smaller projects: CoreDNS, Envoy, and Prometheus. Following the success of that pilot, the programme was extended to Kubernetes itself.

Making the full audit findings publicly available — rather than sharing them only with maintainers — was a significant decision. It reflects a commitment to transparency and sets a concrete example for how large open-source communities can handle security at scale.

What the Audit Covered: Scope and Methodology

The auditors examined Kubernetes across five security control families:

• Networking — traffic flows, network policy enforcement, and exposure of internal APIs
• Cryptography — cipher suite choices, certificate management, and transport-layer security
• Authentication and Authorization — RBAC configuration, admission control, and identity verification
• Secrets Management — how sensitive data such as credentials and tokens is stored and accessed
• Multi-tenancy Isolation — namespace boundaries, workload separation, and privilege escalation paths

Given the sheer size and complexity of the Kubernetes codebase, the audit deliberately favoured breadth over depth. The goal was to produce a holistic security health assessment across all five areas, rather than exhaustively audit a single subsystem. Short-term and long-term remediation goals were recommended for each control category.

The Five High-Severity Findings

The audit identified five findings rated as high severity. Each touches a fundamental security concern:

• Access control bypass of PodSecurityPolicy — An attacker or misconfigured workload could circumvent the policies intended to restrict pod behaviour.
• No mechanism for certificate revocation — Kubernetes did not provide a built-in way to revoke TLS certificates, meaning compromised credentials could remain valid indefinitely.
• Unauthenticated HTTPS connections — Certain HTTPS endpoints did not enforce mutual authentication, leaving them open to impersonation or interception.
• Time-of-check/time-of-use (TOCTOU) race condition with PID handling — A race condition in process ID management could be exploited to interfere with workload scheduling or isolation.
• Improperly patched directory traversal in kubectl cp — A previously reported path traversal vulnerability in the kubectl cp command had not been fully remediated, potentially allowing a malicious container to overwrite arbitrary files on the host.

Beyond these five high-severity issues, the report also surfaced a range of lower-severity recommendations: cleaning up technical debt in the codebase, expanding test coverage, improving documentation, and hardening insecure defaults. These housekeeping recommendations are often the unglamorous but essential work that makes future vulnerability patching faster and safer.

Breadth vs. Depth: The Right Approach for a Project of This Scale

A common critique of broad security audits is that they trade depth for coverage. For a project as large and multifaceted as Kubernetes, however, the breadth-first approach makes practical sense. A targeted deep-dive into a single component — say, the API server’s authentication logic — would yield detailed findings for that subsystem while leaving the rest of the attack surface unexamined.

The holistic view produced by this audit is more useful for prioritisation. Maintainers and the CNCF security SIG can now see where the overall security posture is weakest, allocate resources accordingly, and commission follow-on deep-dive audits of the areas that warrant them. Think of it as a general health check that identifies which specialist referrals to make next.

Benefits for Maintainers: Proactive vs. Reactive Security

One of the most important takeaways from this audit is the value of finding your own vulnerabilities before others do. When a security flaw is discovered and disclosed by an external researcher or, worse, exploited in the wild, maintainers are forced into crisis mode: patches must ship under time pressure, communications must be carefully managed, and users are left exposed in the interim.

A planned security audit changes that dynamic entirely. Maintainers know about the findings in advance. Fixes can be prioritised thoughtfully against other roadmap commitments. Coordinated disclosure timelines can be agreed upon. The result is a more orderly and effective remediation process — and a lower risk of security incidents affecting end users.

Benefits for the Community: Transparency Builds Trust

By open-sourcing the full audit findings — including the white paper and threat model — the CNCF gave the wider Kubernetes community something valuable: the ability to make informed decisions. Organisations running Kubernetes in production can review the findings, assess their own exposure to the identified issues, and take compensating controls where necessary while waiting for upstream fixes.

This transparency also delivers a second benefit: it signals that the project takes security seriously as a first-class concern, not an afterthought. For enterprise adopters evaluating Kubernetes against competing platforms, a published, third-party security audit is meaningful evidence of security maturity.

Finally, the audit artefacts — particularly the threat model — are useful educational resources for anyone looking to deepen their understanding of Kubernetes internals or cloud-native security concepts more broadly.

Key Lessons for Other Open-Source Projects

The Kubernetes security audit offers a replicable model that other open-source projects — especially those handling sensitive workloads or infrastructure — should consider adopting:

1. Commission third-party audits proactively. Internal code review, however thorough, cannot fully replicate the perspective of an independent security team with no prior assumptions about the codebase.
2. Scope audits across control families, not just individual components. A cross-cutting assessment reveals systemic weaknesses that component-level reviews miss.
3. Publish the findings. Open-sourcing the results — not just the patches — builds community trust and accelerates collective learning.
4. Pair findings with actionable goals. Short-term and long-term remediation targets give maintainers and contributors a clear roadmap rather than an undifferentiated list of problems.
5. Treat the threat model as a living document. A threat model produced during an audit is a starting point, not a final answer. It should be updated as the project evolves.

Getting Involved: The CNCF Security SIG

If you want to follow the ongoing security work in the Kubernetes ecosystem, or contribute to future security audits, the CNCF maintains a Security Special Interest Group (SIG Security). The SIG coordinates security reviews, policy guidance, and disclosure processes across CNCF projects. Details on joining can be found through the official CNCF GitHub organisation and community pages.

Conclusion

The CNCF’s decision to commission, and then openly publish, a comprehensive security audit of Kubernetes was a landmark moment for cloud-native security. It demonstrated that even the largest and most complex open-source projects can take a structured, transparent, and proactive approach to security. The five high-severity findings identified in the audit serve as a reminder that no project — regardless of its maturity or community size — is immune to serious vulnerabilities. More importantly, the audit shows that the right response to that reality is not silence, but structured, open engagement with the security community.

For DevOps and platform engineering teams running Kubernetes, the practical takeaway is clear: review the published findings against your own cluster configurations, follow the upstream remediation progress, and consider commissioning your own security assessments for the internal platforms and tools your teams build and maintain.