Kubernetes networking is widely regarded as one of the hardest aspects of running a production Kubernetes environment. The platform imposes strict requirements on how pods communicate: every pod must be reachable from every other pod without network address translation (NAT), and each pod must be aware of its own IP address. Meeting those requirements across multiple hosts, at scale, demands purpose-built tooling.
Why Kubernetes Networking Is Hard
The fundamental challenge is that containers are ephemeral and can be scheduled on any node in a cluster. Maintaining direct, routable connectivity between all pods — regardless of which physical or virtual host they land on — requires a networking layer that goes well beyond what a default Linux network stack provides out of the box. Add multi-tenancy, network policy enforcement, and the need for observability, and the problem becomes genuinely complex.
How CNI Plugins Solve the Problem
Kubernetes delegates the networking problem to the Container Network Interface (CNI) — an open standard for plugin-based networking in Linux containers. CNI plugins are responsible for:
- Allocating unique IP addresses to every pod.
- Creating and managing network interfaces inside container namespaces.
- Setting up routing rules so pods on different hosts can reach each other.
- Optionally enforcing network policies that control traffic flow.
CNI is not Kubernetes-specific. The same plugins also work with OpenShift, Amazon ECS, and other container runtimes. Most plugins achieve cross-host connectivity by building an overlay network on top of the existing physical or cloud network infrastructure. This overlay assigns each container a unique IP from a shared address space and transparently routes traffic between containers without requiring changes to the underlying network.
The overlay approach offers two practical benefits: cluster administrators gain a single, consistent network layer that is far easier to monitor and secure, and individual hosts no longer need hand-crafted iptables rules for every container-to-container communication path.
Tool 1: Flannel — Simple Overlay Networking
Flannel, originally developed by CoreOS, is one of the oldest and most widely adopted CNI plugins. Its design philosophy is simplicity: it provides a straightforward overlay network that satisfies Kubernetes networking requirements without exposing operators to unnecessary complexity.
How Flannel Works
- Flannel assigns a subnet to each node. Every pod on that node receives an IP from that subnet, giving every pod a unique, cluster-wide address.
- Subnet-to-host mappings are stored in etcd, the same key-value store Kubernetes uses for cluster state.
- A daemon called flanneld runs on every node. It watches etcd for subnet assignments and programs route rules in the Linux kernel’s routing table accordingly.
- For multi-host traffic, the recommended default backend is VXLAN. The flanneld daemon collects packets destined for remote hosts, wraps them in UDP/VXLAN encapsulation (an L3 wrapper), and sends them across the physical network. The receiving node unwraps the packet and delivers it to the correct pod.
- A software bridge (e.g., the Linux bridge) handles local switching; the VXLAN device handles forwarding at Layer 2 for off-host traffic.
When to Use Flannel
Flannel is an excellent choice when you need basic pod-to-pod connectivity and want minimal operational overhead. It does not natively support Kubernetes NetworkPolicy, so if fine-grained traffic control is a requirement, you will need to pair it with a separate policy engine or choose a different plugin.
Tool 2: Calico — High-Performance L3 Networking with Policy
Project Calico takes a fundamentally different architectural approach. Rather than building an overlay, Calico configures a pure Layer 3 network using BGP (Border Gateway Protocol) to exchange routing information directly between nodes. Each node acts as a BGP router, advertising the pod CIDRs it owns to the rest of the cluster.
How Calico Works
- Packets travel between pods without any encapsulation wrapper, which reduces overhead and makes packet captures with standard tools (tcpdump, Wireshark) straightforward.
- BGP route distribution is handled by BIRD, a widely used open-source BGP daemon.
- In environments where BGP is not supported by the underlying network fabric, Calico can fall back to VXLAN or IP-in-IP encapsulation modes.
- Calico enforces network policy by programming iptables (or eBPF) rules on each node, giving fine-grained control over ingress and egress traffic.
Advanced Policy Capabilities
The standard Kubernetes NetworkPolicy API covers ingress rules based on pod labels and ports, but has historically lacked support for egress policies and CIDR-based rules. Calico extends this with its own GlobalNetworkPolicy and NetworkPolicy custom resources, which add:
- Egress policy enforcement to control outgoing connections from pods.
- CIDR-based rules for integrating with external networks.
- Namespace-scoped policies that prevent cross-namespace communication by default.
- Integration with Istio service mesh, allowing Calico to enforce policy at both the network infrastructure layer and the service mesh layer simultaneously.
When to Use Calico
Calico is a strong choice when performance and advanced network policy are priorities. The L3 routing approach scales well to large clusters. Commercial support is available through Tigera, the company that maintains the project.
Tool 3: Weave Net — Ease of Use with Strong Performance
Weave Net, from Weaveworks, occupies the middle ground: it aims to combine operational simplicity with solid performance and built-in security features.
How Weave Net Works
- Weave Net creates a mesh overlay network that connects every node in the cluster to every other node.
- Like Flannel, it uses VXLAN encapsulation for cross-host traffic, but it performs the encapsulation directly in the kernel via a fast datapath. This avoids the overhead of packets traveling through userspace, which provides a meaningful performance advantage over purely userspace implementations.
- When the underlying network topology does not support the fast datapath, Weave automatically falls back to a userspace mode called sleeve mode, ensuring connectivity is maintained even if performance is reduced.
- A built-in DNS component called weaveDNS provides automatic service discovery, name resolution, load balancing, and fault tolerance within the cluster without additional configuration.
Security Features
Weave Net ships with traffic encryption enabled between hosts. It uses NaCl encryption for sleeve mode traffic and IPsec ESP for fast datapath traffic. Network policies are automatically configured at installation time; operators only need to define their rules rather than setting up the policy infrastructure from scratch.
When to Use Weave Net
Weave Net is well-suited to teams that want good performance and security without the operational complexity of managing a BGP-based network like Calico. Its automatic configuration and built-in service discovery make it accessible to operators who are newer to Kubernetes networking.
Comparison Summary
| Feature | Flannel | Calico | Weave Net |
|---|---|---|---|
| Networking approach | Overlay (VXLAN) | L3 BGP (or VXLAN fallback) | Mesh overlay (kernel VXLAN) |
| Encapsulation | Yes (VXLAN/UDP) | No (by default) | Yes (kernel-space) |
| NetworkPolicy support | No (native) | Yes (extended) | Yes |
| Built-in encryption | No | No (requires WireGuard or external) | Yes (NaCl / IPsec) |
| Service discovery | No | No | Yes (weaveDNS) |
| Operational complexity | Low | Medium–High | Low–Medium |
| Best for | Simple clusters, getting started | Large-scale, policy-heavy environments | Balanced performance and ease of use |
Choosing the Right CNI Plugin
There is no universally correct answer. The right plugin depends on your team’s operational maturity, the size and complexity of your cluster, and your network policy requirements:
- Start with Flannel if you need something working quickly and your cluster does not require network policy enforcement.
- Choose Calico if performance, scalability, and fine-grained egress/ingress policy are priorities, and your team is comfortable with BGP concepts or willing to learn them.
- Choose Weave Net if you want a balance of performance, built-in encryption, and service discovery without significant configuration effort.
It is also worth noting that this list is not exhaustive. Other CNI plugins — such as Cilium (eBPF-based), Canal (Flannel + Calico policy), and Multus (multi-network support) — address specialized use cases and are worth evaluating as your requirements evolve. Modern Kubernetes deployments often combine multiple tools rather than relying on a single solution for all networking needs.
Original source referenced: techgenix.com (article from June 2019). Some product capabilities described here may have evolved since the original publication date. Always consult current official documentation before making architectural decisions.