LabHub

Blog

Kubernetes Network Policy with Calico and Cilium Microsegmentation

2026-03-12 · 21 min read · #kubernetes#network-policy#calico#cilium#security#microsegmentation#zero-trust

한국어English日本語

Kubernetes Network Policy

Introduction

A Kubernetes cluster operates by default as a flat network in which every Pod may talk to every other Pod. That is convenient for development but a fatal weakness from a security standpoint. Once a single Pod is compromised, the attacker can reach every service in the cluster freely, with no firewall in between. This is called a lateral movement attack, and it is one of the most frequent categories of security incident in container environments.

According to Red Hat's 2024 State of Kubernetes Security report, about 90% of the organizations surveyed said they had experienced a security incident in a container or Kubernetes environment. In a large share of those cases the damage spread because there was no isolation at the network level. In a microservice architecture in particular the number of communication paths between services grows explosively, so traditional perimeter security alone cannot stop internal threats.

Microsegmentation is the answer to this problem. It is a strategy that subdivides the network at the workload level so that each service performs only the communication it has been explicitly permitted. In Kubernetes it is implemented through the NetworkPolicy API and CNI plugins such as Calico and Cilium. This article covers the whole process of building a Zero-Trust network in production, starting from the basic NetworkPolicy spec and moving through Calico's GlobalNetworkPolicy and Cilium's L7 policies and eBPF-based enforcement.

Basic Structure of Kubernetes NetworkPolicy

Reading the NetworkPolicy API v1 Spec

Kubernetes NetworkPolicy is a namespace-scoped resource in the networking.k8s.io/v1 API group. It controls inbound (Ingress) and outbound (Egress) traffic for Pods, and any Pod with at least one policy applied to it switches to a whitelist model in which only the traffic named in that policy is allowed.

The core components are as follows.

The important point is that NetworkPolicy is additive. When several policies apply to the same Pod, the union of the traffic each policy allows becomes the final allowed range. There is no notion of conflict or priority between policies.

Establishing a Baseline with a Default-Deny Policy

The first step toward Zero-Trust is to block all traffic by default. Apply a default-deny policy per namespace, then explicitly allow only the communication you need.

# default-deny-all.yaml
# Block all Ingress/Egress traffic inside the namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

With this policy applied, every Pod in the production namespace has both inbound and outbound traffic blocked. DNS lookups are blocked too, so service discovery stops working. A policy that allows DNS traffic must therefore be applied alongside it.

# allow-dns.yaml
# Allow egress to the DNS service in the kube-system namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
      ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

Policies That Allow Cross-Namespace Communication

In a microservice environment, services in different namespaces need to talk to each other. For example, expressing in YAML the case where a web server in the frontend namespace has to reach an API server in the backend namespace gives the following.

# allow-frontend-to-backend.yaml
# Allow Ingress from the frontend namespace to the backend API server
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-api
  namespace: backend
spec:
  podSelector:
    matchLabels:
      app: api-server
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: frontend
          podSelector:
            matchLabels:
              app: web
      ports:
        - protocol: TCP
          port: 8080

This policy allows access on TCP port 8080 to Pods carrying the app: api-server label in the backend namespace, only from app: web Pods in the frontend namespace. Note that putting namespaceSelector and podSelector inside the same from entry makes them an AND condition, while splitting them into separate from entries makes them an OR condition.

Calico Network Policy in Depth

Calico Compared With the Built-in NetworkPolicy

Calico is a CNI plugin that fully supports the built-in Kubernetes NetworkPolicy API while adding further capabilities. The main differences are as follows.

CapabilityKubernetes NetworkPolicyCalico NetworkPolicy
Policy scopeNamespaceNamespace + cluster-wide (Global)
Policy orderNone (additive)Explicit order field
Policy tierNoneMulti-tier hierarchy
Deny ruleNo explicit DenyExplicit Deny action
Log actionNot supportedLogging on policy match
Staged PolicyNot supportedSimulation before production rollout
Host EndpointNot supportedNode-level firewall policy
FQDN/DNS policyNot supportedDNS-name-based egress control
L7 policyNot supportedSupported in Calico Enterprise

Cluster-Wide Default-Deny with GlobalNetworkPolicy

The built-in NetworkPolicy applies per namespace only, so a default-deny policy has to be added by hand every time a new namespace appears. Calico's GlobalNetworkPolicy solves this by applying to the whole cluster at once.

# calico-global-default-deny.yaml
# Cluster-wide default-deny (kube-system excluded)
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: default-deny
spec:
  tier: default
  order: 1000
  selector: all()
  namespaceSelector: '!kubernetes.io/metadata.name == "kube-system"'
  types:
    - Ingress
    - Egress

This policy applies a default block to every namespace except kube-system. order: 1000 means low priority, so more specific allow policies are evaluated first. In Calico, the lower the order value, the higher the priority.

Policy Tiering Architecture

Calico's tier system is a powerful mechanism for managing policies hierarchically. Each tier forms an independent policy evaluation pipeline, and traffic that an upper tier does not explicitly Allow or Deny is handed on to the next tier (Pass).

Three or four tiers are typical in practice.

  1. SecurityOps tier: owned by the security team. Top-priority rules such as blocking malicious IPs and regulatory compliance policies
  2. Platform tier: owned by the platform team. DNS allowances, monitoring agent traffic, shared infrastructure policies
  3. Application tier: owned by the development teams. Per-application service-to-service communication rules
  4. Default tier: the baseline default-deny policy
# tier-setup.yaml
# Create the SecurityOps tier - the top-priority policy layer owned by the security team
apiVersion: projectcalico.org/v3
kind: Tier
metadata:
  name: security-ops
spec:
  order: 100
---
# Create the Platform tier - the shared infrastructure policy layer
apiVersion: projectcalico.org/v3
kind: Tier
metadata:
  name: platform
spec:
  order: 200
---
# Create the Application tier - the service policy layer owned by the development teams
apiVersion: projectcalico.org/v3
kind: Tier
metadata:
  name: application
spec:
  order: 300
---
# Platform tier: policy allowing DNS and monitoring traffic
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: platform.allow-dns-and-monitoring
spec:
  tier: platform
  order: 100
  selector: all()
  types:
    - Egress
  egress:
    # Allow DNS traffic
    - action: Allow
      protocol: UDP
      destination:
        ports:
          - 53
    - action: Allow
      protocol: TCP
      destination:
        ports:
          - 53
    # Allow Prometheus metric scraping
    - action: Allow
      protocol: TCP
      destination:
        selector: app == "prometheus"
        ports:
          - 9090
    # Hand the remaining traffic to the next tier
    - action: Pass

The advantage of this structure is separation of concerns. The security team manages compliance policies in the SecurityOps tier, the platform team infrastructure policies in the Platform tier, and the development teams per-service policies in the Application tier, each independently.

Safe Production Rollout with Staged Policy

Applying a new network policy straight to production can cause an unexpected outage. Calico's Staged Policy runs a policy in simulation mode without enforcing it, so you can see in advance which traffic would be affected.

A Staged Policy is simply a change of resource type to StagedGlobalNetworkPolicy or StagedNetworkPolicy. Such a policy neither blocks nor allows traffic for real, but it produces a record in the Calico logs saying that this traffic matched the policy.

The production rollout procedure is as follows.

  1. Deploy the policy as a StagedGlobalNetworkPolicy
  2. Analyse the impact in Calico Enterprise or in the logs (at least 24 to 48 hours)
  3. Confirm there is no unexpectedly blocked traffic
  4. Change the resource type to GlobalNetworkPolicy to enforce it for real
  5. Monitor after enforcement and roll back immediately if anything looks wrong

Cilium Network Policy and eBPF

The CiliumNetworkPolicy CRD

Cilium is a high-performance CNI plugin built on eBPF (extended Berkeley Packet Filter). It supports the built-in Kubernetes NetworkPolicy and adds the CiliumNetworkPolicy and CiliumClusterwideNetworkPolicy CRDs for defining fine-grained policies at the L3/L4/L7 level.

Cilium's biggest differentiator is its eBPF-based data plane. Unlike traditional iptables-based implementations, eBPF programs process packets in the kernel, so performance barely degrades as the number of policy rules grows. iptables evaluates rules as a linear chain, so latency rises in proportion to the rule count, whereas eBPF keeps close to O(1) performance through hash-map lookups.

L3/L4/L7 Policy Enforcement Architecture

Cilium policies operate at three layers.

L7 policies are handled by the Envoy proxy embedded in Cilium. That makes fine-grained control possible, such as "this Pod may only issue GET /api/v1/users requests, and POST /api/v1/admin requests are blocked".

# cilium-l7-http-policy.yaml
# L7 HTTP policy: method/path based access control for the API server
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: api-server-l7-policy
  namespace: backend
spec:
  endpointSelector:
    matchLabels:
      app: api-server
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: web-frontend
      toPorts:
        - ports:
            - port: '8080'
              protocol: TCP
          rules:
            http:
              # Allow the user lookup API only
              - method: GET
                path: '/api/v1/users.*'
              # Allow the user creation API
              - method: POST
                path: '/api/v1/users'
              # Allow the health check endpoint
              - method: GET
                path: '/healthz'
    - fromEndpoints:
        - matchLabels:
            app: admin-dashboard
      toPorts:
        - ports:
            - port: '8080'
              protocol: TCP
          rules:
            http:
              # The admin dashboard may use every method
              - method: '.*'
                path: '/api/v1/.*'

This policy lets web-frontend use only the user-related read/write APIs and the health check, while admin-dashboard can reach every API endpoint. An L7 policy violation returns an HTTP 403 response, and Hubble shows the details of the blocked request.

Controlling External Access with DNS-Based FQDN Policies

In production, Pods often have to reach external APIs such as payment gateways and third-party services. IP-based egress policies break when the external service changes its IP. Cilium's FQDN-based policies solve this by controlling external access by DNS name.

# cilium-fqdn-egress-policy.yaml
# DNS-based egress policy: only allowed external domains are reachable
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: payment-service-egress
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: payment-service
  egress:
    # Allow DNS lookups (required for FQDN policies to work)
    - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
      toPorts:
        - ports:
            - port: '53'
              protocol: ANY
          rules:
            dns:
              - matchPattern: '*'
    # Allow the payment gateway APIs only
    - toFQDNs:
        - matchName: api.stripe.com
        - matchName: api.paypal.com
      toPorts:
        - ports:
            - port: '443'
              protocol: TCP
    # Allow access to the internal database
    - toEndpoints:
        - matchLabels:
            app: postgresql
            role: primary
      toPorts:
        - ports:
            - port: '5432'
              protocol: TCP

This policy limits the outbound traffic of the payment-service Pod to port 443 on api.stripe.com and api.paypal.com plus port 5432 on the internal PostgreSQL database. The rule allowing DNS lookups must be included for the FQDN policy to work correctly. Cilium intercepts DNS responses to learn the IP mapping for a domain and thereafter allows traffic only to those IPs.

The Identity-Based Security Model

Cilium's security model is fundamentally different from a traditional IP-based firewall. It assigns a security identity to each Pod or endpoint and evaluates policies against that identity rather than the source IP of the packet.

The advantages of this approach are as follows.

Calico and Cilium Compared

Detailed Comparison Table

The table below summarises the key differences between the two solutions.

Comparison itemCalicoCilium
Data planeiptables (default) / eBPF (option)eBPF (default)
L3/L4 policyFully supportedFully supported
L7 policySupported in the Enterprise editionSupported in the open source edition
FQDN policySupported (DNS Policy)Supported (toFQDNs)
Policy tierSupported (Tier CRD)Not supported (single layer)
Global policyGlobalNetworkPolicyCiliumClusterwideNetworkPolicy
Staged PolicySupportedNot supported (policy-audit-mode instead)
ObservabilityCalico Enterprise Flow LogsHubble (open source, built in)
Service meshSeparate setup requiredCilium Service Mesh built in
Multi-clusterCalico FederationCluster Mesh
EncryptionWireGuardWireGuard / IPsec
Performance (large policy sets)Latency proportional to rule count on iptablesO(1) lookup via eBPF hash maps
MaturityVery high (2016~)High (2018~)
Learning curveModerateHigh (requires understanding eBPF)

Selection Criteria and Recommendations by Environment

When to choose Calico

When to choose Cilium

In practice both solutions are sufficiently proven in production, so choose according to your team's technology stack and operational requirements. What matters is that a default-deny policy must be applied whichever solution you choose, and that you have an impact analysis process for policy changes.

Production Operations Guide

Zero-Trust Network Implementation Checklist

Work through the following checklist in order when implementing a Zero-Trust network in production.

  1. Standardise namespace labelling: apply a consistent label scheme to every namespace and Pod. Define standard labels such as app, env, team and tier.
  2. Analyse the current traffic pattern: collect existing communication patterns with Hubble or Calico Flow Logs for at least one week.
  3. Apply the default-deny policy: apply it in audit/staged mode first, then switch to enforcement mode after analysing the impact.
  4. Allow DNS and essential infrastructure traffic: allow egress to infrastructure services such as kube-dns, monitoring agents and log collectors.
  5. Write per-service allow policies: based on the collected traffic patterns, write policies that allow only the minimum communication each service needs.
  6. Test and verify the policies: automate policy linting and simulation in the CI/CD pipeline.
  7. Roll out gradually: apply in the order development environment - staging environment - production canary - full production.
  8. Monitor continuously: watch policy violations, blocked traffic and new communication patterns in real time.

Policy Testing and Simulation Strategy

A network policy must be tested before it reaches production. The following three levels of testing are recommended.

Level 1: Static Analysis

Use kubectl and policy lint tools to check for YAML syntax errors and selector mistakes. kube-linter or conftest can also verify compliance with your organisation's policy standards automatically.

Level 2: Simulation (Dry-Run)

Use Calico's Staged Policy or Cilium's policy-audit-mode. In Cilium, audit mode can be enabled in the agent configuration as follows.

# cilium-config.yaml
# Cilium agent configuration: enable policy audit mode
apiVersion: v1
kind: ConfigMap
metadata:
  name: cilium-config
  namespace: kube-system
data:
  enable-policy: 'default'
  policy-audit-mode: 'true'
  monitor-aggregation: 'medium'
  hubble-metrics-server: ':9965'
  hubble-metrics: 'dns,drop,tcp,flow,port-distribution,icmp,httpV2'

In audit mode, traffic that violates a policy is not blocked; instead an action: audit event is recorded in the Hubble logs. Analyse those events to confirm the policy is correct, then turn audit mode off.

Level 3: Integration Test

Run E2E tests on a test cluster configured identically to the real environment to confirm that service-to-service communication works. Use tools such as curl, netcat and nmap to verify concretely what is allowed and what is blocked.

Policy Monitoring with Hubble and Calico Enterprise

Hubble (Cilium)

Hubble is the observability platform built into Cilium; it visualises network flows in real time. The hubble observe command shows the traffic that policies have blocked or allowed.

The main monitoring metrics are as follows.

Calico Enterprise Flow Logs

Calico Enterprise provides detailed flow logs for all allowed and blocked traffic. They can be shipped to Elasticsearch or Splunk to build dashboards, and they are useful for comparing traffic patterns before and after a policy change.

Troubleshooting: Debugging Unapplied Policies and Blocked Communication

Here are the network policy problems that occur most often in production, and how to debug them.

Problem 1: the policy is applied but traffic is not blocked

Problem 2: legitimate traffic is blocked

Problem 3: intermittent communication failures

Failure Cases and Recovery Procedures

Case 1: A Lateral Movement Attack While Running Without Default-Deny

Situation: at an e-commerce platform, an attacker broke into a single Pod through a vulnerability in a public web service. Because no network policy was applied at all, the attacker moved freely to the Redis cache, the internal API servers and even the database, and exfiltrated customer data.

Root cause analysis:

Recovery and improvements:

Case 2: A Service Outage Caused by the Wrong Policy Order

Situation: the security team added a new blocking policy in a Calico tier but set the order value wrongly, so it was applied at a higher priority than the DNS allow policy in the Platform tier. As a result DNS lookups failed across the whole cluster and all service-to-service communication stopped.

Root cause analysis:

Recovery procedure:

  1. Delete the offending policy immediately with calicoctl delete gnp
  2. Confirm the DNS service is back to normal (recovery within about 30 seconds)
  3. Rewrite the policy as a StagedGlobalNetworkPolicy and run the simulation
  4. Document the policy order value scheme: SecurityOps(100~199), Platform(200~299), Application(300~399)
  5. Add a CI/CD validation gate to prevent order value collisions between tiers

Lesson: core infrastructure traffic such as DNS, kube-apiserver and monitoring must be protected in the topmost tier, and a new blocking policy must always go through Staged mode.

Rollback Strategy

Fast rollback is essential when a network policy causes an outage. Prepare the following three levels of rollback.

  1. Per-policy rollback: revert only a specific policy to its previous version. Under GitOps management, a git revert is synchronised automatically.
  2. Namespace-level rollback: delete every NetworkPolicy in that namespace to restore the default-allow state. Reapply the policies one by one after the service recovers.
  3. Cluster-wide emergency rollback: delete every NetworkPolicy and all Calico/Cilium policies. This is the last resort, used only in an emergency where you have to accept the security risk and prioritise service availability.

Operational Considerations

Namespace Labelling Strategy

The effectiveness of a network policy depends on the consistency of the labelling scheme. The following standard label schema is recommended.

Label keyDescriptionExample values
appApplication nameapi-server, web-frontend
versionDeployment versionv1, v2
envEnvironmentproduction, staging, dev
teamOwning teamplatform, backend, data
tierArchitecture tierfrontend, backend, database
complianceCompliance levelpci-dss, hipaa, sox

Labels must be applied to namespaces as well, because namespaceSelector matches on namespace labels. Kubernetes 1.22 and later adds the kubernetes.io/metadata.name label automatically so name-based selection is possible, but it is better to manage additional classification labels explicitly.

Assessing the Blast Radius of a Policy Change

Follow this process when changing a network policy.

  1. Change impact analysis: work out in advance how many Pods the policy applies to and which service-to-service communication paths are affected.
  2. Peer review: have the security team and the service operations team review the policy change together.
  3. Canary rollout: apply it to one namespace or a subset of Pods first and observe the impact.
  4. Heightened monitoring: watch the blocked-traffic logs closely for at least 30 minutes after the change.
  5. Automatic rollback conditions: build a mechanism that rolls back to the previous policy automatically when blocked traffic exceeds a threshold.

Automating Policy Validation in the CI/CD Pipeline

Network policies are managed as code, so they can be validated automatically in the CI/CD pipeline. Integrate the following stages into the pipeline.

Lint stage: perform YAML schema validation with kubeval or kubeconform. Validate the Calico/Cilium CRD schemas as well.

Policy stage: check compliance with organisational standards using conftest and OPA Rego policies. For example, verify that every namespace has a default-deny policy, and that no egress policy contains a wildcard CIDR (0.0.0.0/0).

Simulation stage: apply the policies to a test cluster and verify service communication with E2E tests.

Approval stage: put a manual approval gate for the security team before production rollout.

# github-actions-network-policy-ci.yaml
# GitHub Actions: network policy validation pipeline
name: Network Policy Validation
on:
  pull_request:
    paths:
      - 'k8s/network-policies/**'

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: YAML Lint
        run: |
          yamllint k8s/network-policies/

      - name: Schema Validation
        run: |
          kubeconform \
            -schema-location default \
            -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' \
            k8s/network-policies/*.yaml

      - name: Policy Compliance Check
        uses: open-policy-agent/conftest-action@v2
        with:
          files: k8s/network-policies/
          policy: policies/network/

      - name: Integration Test
        run: |
          kind create cluster --config test/kind-config.yaml
          kubectl apply -f k8s/network-policies/
          ./test/verify-connectivity.sh

This pipeline automatically runs YAML linting, schema validation, compliance checks and integration tests whenever a network policy file changes, which guarantees a safe policy deployment.

Closing Thoughts

Kubernetes network security is not optional. The basic NetworkPolicy API alone is enough to implement meaningful microsegmentation, and Calico's tiering architecture and Staged Policy, together with Cilium's L7 policies and eBPF performance, provide an even stronger security posture in production.

The most important thing is to start. Aiming for a perfect policy design usually ends with nothing applied at all. Applying default-deny first, observing traffic patterns, and refining the policies gradually is the realistic and effective iterative approach.

A Zero-Trust network is not achieved with a single tool or policy; it comes out of an operational culture of continuous observation, analysis and improvement. Use the technical methods covered in this article as a basis for establishing and executing a network policy strategy that fits your organisation's security requirements.

References

Read the original: https://www.youngju.dev/blog/kubernetes/2026-03-12-kubernetes-network-policy-calico-cilium-microsegmentation.en

Comments

No comments yet.

Sign in to leave a comment