Kubernetes Istio Ambient Mesh Sidecarless Guide

• Introduction
• Understanding the Istio Ambient Mesh Architecture
  • A Two-layer Data Plane
  • The Traffic Flow in Detail
• ztunnel: The L4 Processing Layer
  • What ztunnel Is
  • ztunnel's Capabilities in Detail
  • ztunnel's Resource Profile
  • Checking ztunnel's Status
• Waypoint Proxy: The L7 Processing Layer
  • The Role of the Waypoint Proxy
  • Deploying a Waypoint Proxy
  • Scaling the Waypoint Proxy
• Sidecar vs Ambient
  • Which Mode to Choose
• Installation and Configuration
  • Prerequisites
  • Installing Istio with the Ambient Profile
  • Adding a Namespace to the Mesh
  • Installing with Helm
• Configuring Traffic Policy
  • L4 AuthorizationPolicy
  • L7 AuthorizationPolicy (Waypoint Required)
  • Traffic Splitting with a VirtualService
  • PeerAuthentication Configuration
• Observability Integration
  • Collecting Prometheus Metrics
  • Monitoring the Key Metrics
  • Kiali Integration
  • Grafana Dashboards
• Migration Strategy
  • Moving from Sidecars to Ambient Mode
  • Cautions During the Migration
• Troubleshooting
  • Common Problems and How to Fix Them
• Production Checklist
  • Before Installation
  • Network Configuration
  • Security Configuration
  • Observability
  • High Availability
  • Migration (When Switching from Sidecars)
• Failure Cases and Recovery Procedures
  • Case 1: Service Interruption During a ztunnel DaemonSet Update
  • Case 2: L7 Policy Latency from Waypoint Proxy Overload
  • Case 3: Communication Failures in Mixed Sidecar/Ambient Mode
  • Case 4: The Effect of an istiod Failure on the Ambient Mesh
• References

Introduction

A service mesh is the core infrastructure layer that transparently manages service-to-service communication in a microservice architecture. Istio is the most widely used open-source project in the field, but the traditional sidecar proxy model carried two fundamental problems: operational complexity and resource overhead.

Injecting an Envoy sidecar into every Pod carries costs you cannot ignore: 50-100MB of memory overhead per Pod, 2-5 seconds of startup delay, and a rolling restart of every workload whenever the sidecar is upgraded. In a large cluster running thousands of Pods, the sidecars themselves become a substantial infrastructure cost.

Istio Ambient Mesh solves this problem at the root. It removes the sidecar entirely and introduces a two-layer architecture - a node-level shared proxy (ztunnel) and an optional L7 proxy (the waypoint proxy) - achieving over 90% savings in memory usage and over 50% savings in CPU usage. Since it reached Beta in Istio 1.22, adoption in production environments has been growing quickly.

In this article, we cover Ambient Mesh with a practical focus: its architectural principles, installation, traffic policy, observability, the strategy for migrating from sidecars, and the failure cases and recovery procedures you meet in production operations.

Understanding the Istio Ambient Mesh Architecture

A Two-layer Data Plane

Ambient Mesh's core design philosophy is separation of concerns. In the traditional sidecar model a single Envoy proxy handled everything from L4 to L7; Ambient Mesh splits that into two clearly separated layers.

+------------------------------------------------------------------+
|                    Istio Ambient Mesh architecture                  |
+------------------------------------------------------------------+
|                                                                    |
|  [Service A Pod]    [Service B Pod]    [Service C Pod]             |
|       |                   |                   |                    |
|       v                   v                   v                    |
|  +----------------------------------------------------------+     |
|  |              ztunnel (DaemonSet, on every node)            |     |
|  |  - mTLS tunneling (HBONE)                                 |     |
|  |  - L4 authentication / authorization                      |     |
|  |  - L4 telemetry                                           |     |
|  |  - TCP connection management                              |     |
|  +----------------------------------------------------------+     |
|       |                                                            |
|       | HBONE (tunneling based on HTTP CONNECT)                    |
|       v                                                            |
|  +----------------------------------------------------------+     |
|  |      Waypoint Proxy (optional, per namespace/SA)          |     |
|  |  - HTTP routing                                           |     |
|  |  - L7 authentication / authorization policy               |     |
|  |  - L7 telemetry                                           |     |
|  |  - traffic mirroring, canary deployment                   |     |
|  |  - header-based routing                                   |     |
|  +----------------------------------------------------------+     |
|       |                                                            |
|       | HBONE                                                      |
|       v                                                            |
|  +----------------------------------------------------------+     |
|  |              ztunnel (destination node)                   |     |
|  +----------------------------------------------------------+     |
|       |                                                            |
|       v                                                            |
|  [Destination Pod]                                                 |
|                                                                    |
+------------------------------------------------------------------+

L4 Secure Overlay (ztunnel): enabled by default for all mesh traffic. It provides mTLS encryption, L4 access control and TCP-level telemetry. It is implemented as an extremely lightweight Rust-based proxy.

L7 Processing (Waypoint Proxy): deployed optionally, only where L7 features are needed - HTTP routing, header-based policy, gRPC metric collection. It uses the standard Envoy proxy.

Thanks to this design, services that do not need L7 features (a large share of all services) get the mesh's security and observability from ztunnel alone, without paying for an L7 proxy.

The Traffic Flow in Detail

Let us walk through how traffic is handled in Ambient Mesh, step by step.

L4-only flow (no waypoint):
  Pod A ---> ztunnel(node1) ===HBONE===> ztunnel(node2) ---> Pod B

Flow that needs L7 processing:
  Pod A ---> ztunnel(node1) ===HBONE===> Waypoint Proxy ===HBONE===> ztunnel(node2) ---> Pod B

1. The ztunnel on the same node transparently intercepts the traffic leaving the source Pod
2. ztunnel encrypts and tunnels the traffic with the HBONE protocol (based on HTTP CONNECT)
3. If the destination has an L7 policy configured, it goes to the waypoint proxy first
4. The waypoint proxy applies the L7 policy (routing, authorization and so on) and forwards it to the destination node's ztunnel
5. The destination node's ztunnel decrypts it and delivers it to the destination Pod

ztunnel: The L4 Processing Layer

What ztunnel Is

ztunnel (Zero Trust Tunnel) is the lightweight proxy that forms Ambient Mesh's foundation layer. Deployed as a DaemonSet on each node, it handles all the workload traffic on that node.

Key characteristics:

• Written in Rust: it delivers C/C++-level performance while guaranteeing memory safety
• Deliberately limited scope: it is designed to handle only L3/L4 functions, which keeps the attack surface small
• Does not parse HTTP traffic: it neither reads nor modifies the workload's HTTP headers, which is better for security
• The HBONE protocol: it implements mTLS communication with a tunneling protocol based on HTTP CONNECT

ztunnel's Capabilities in Detail

# the scope of what ztunnel provides
ztunnel_capabilities:
  security:
    - automatic mTLS encryption/decryption
    - SPIFFE-based workload identity issuance
    - applying L4 AuthorizationPolicy
  networking:
    - TCP connection proxying
    - HBONE tunneling
    - load balancing (L4)
  observability:
    - TCP connection metrics (bytes, connection count, duration)
    - L4 access logs
    - Prometheus metric exposure
  not_supported:
    - HTTP routing
    - header-based policy
    - gRPC protocol parsing
    - traffic mirroring/splitting

ztunnel's Resource Profile

ztunnel is lighter than a sidecar Envoy by an incomparable margin. Here is how the resource usage compares in a typical production environment.

Checking ztunnel's Status

# check the ztunnel DaemonSet status
kubectl get daemonset -n istio-system ztunnel

# check the ztunnel Pod logs
kubectl logs -n istio-system -l app=ztunnel --tail=50

# check the ztunnel proxy status (on a specific node)
kubectl exec -n istio-system $(kubectl get pod -n istio-system -l app=ztunnel --field-selector spec.nodeName=worker-1 -o jsonpath='{.items[0].metadata.name}') -- curl -s localhost:15000/config_dump

# list the workloads ztunnel manages
kubectl exec -n istio-system $(kubectl get pod -n istio-system -l app=ztunnel -o jsonpath='{.items[0].metadata.name}') -- curl -s localhost:15020/debug/workloadz

Waypoint Proxy: The L7 Processing Layer

The Role of the Waypoint Proxy

The waypoint proxy is an optional component you deploy only where L7-level traffic management is needed. It uses the standard Envoy proxy engine and is deployed per namespace or per service account.

Here is how it differs from ztunnel in the essentials.

• It understands the HTTP/gRPC protocols and parses headers
• It applies request-level routing, retries and timeouts
• It enforces L7 AuthorizationPolicy (path- and method-based)
• It collects HTTP metrics and distributed tracing data

Deploying a Waypoint Proxy

# deploy a waypoint proxy for a namespace
istioctl waypoint apply -n my-namespace

# deploy a waypoint for a specific service account
istioctl waypoint apply -n my-namespace --name sa-waypoint --for service

# check the waypoint proxy status
kubectl get gateway -n my-namespace

The waypoint proxy is managed as a Gateway resource of the Kubernetes Gateway API.

# an example waypoint proxy Gateway resource
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: my-namespace-waypoint
  namespace: my-namespace
  labels:
    istio.io/waypoint-for: service
spec:
  gatewayClassName: istio-waypoint
  listeners:
    - name: mesh
      port: 15008
      protocol: HBONE

Scaling the Waypoint Proxy

In production you have to tune the waypoint proxy's resources and replica count appropriately.

# HPA configuration for the waypoint proxy
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: my-namespace-waypoint
  namespace: my-namespace
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-namespace-waypoint
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 80

Sidecar vs Ambient

Here is a comparison of the two modes from several angles. Sidecar mode and ambient mode can be mixed per namespace within the same cluster, but the two modes must never be applied to the same namespace at once.

Which Mode to Choose

When ambient mode is the right fit:

• When resource efficiency matters in a large cluster
• When most services need only L4 security (mTLS, network policy)
• When you want to adopt the mesh gradually without restarting Pods
• When you want to reduce the operational burden of managing sidecars

When sidecar mode is the right fit:

• When complete per-Pod network isolation is required
• When fine-grained proxy customization through EnvoyFilter is required
• When multi-cluster or VM networking is central
• When L7 policy has to apply to every service by default

Installation and Configuration

Prerequisites

# check the Kubernetes cluster version (1.27+ recommended)
kubectl version --short

# install istioctl (latest version)
curl -L https://istio.io/downloadIstio | sh -
cd istio-*
export PATH=$PWD/bin:$PATH

# check the istioctl version
istioctl version

Installing Istio with the Ambient Profile

# install Istio with the ambient profile
istioctl install --set profile=ambient --skip-confirmation

# confirm the installation
kubectl get pods -n istio-system

# expected output:
# NAME                      READY   STATUS    RESTARTS   AGE
# istiod-xxxx               1/1     Running   0          2m
# ztunnel-xxxxx             1/1     Running   0          2m
# ztunnel-yyyyy             1/1     Running   0          2m
# ztunnel-zzzzz             1/1     Running   0          2m
# istio-cni-node-xxxxx      1/1     Running   0          2m
# istio-cni-node-yyyyy      1/1     Running   0          2m
# istio-cni-node-zzzzz      1/1     Running   0          2m

The ambient profile installs the following components:

• istiod: the control plane (Pilot, Citadel and Galley combined)
• ztunnel: the L4 proxy on each node (DaemonSet)
• istio-cni: the CNI plugin for traffic redirection (DaemonSet)

Adding a Namespace to the Mesh

# add the namespace to the ambient mesh (no Pod restart needed!)
kubectl label namespace my-app istio.io/dataplane-mode=ambient

# check the labels
kubectl get namespace my-app --show-labels

# exclude a specific Pod from the mesh
kubectl label pod my-pod -n my-app istio.io/dataplane-mode=none

# remove the namespace from the mesh (no Pod restart needed!)
kubectl label namespace my-app istio.io/dataplane-mode-

A caution: you must never apply both the sidecar mode label (istio-injection=enabled) and the ambient mode label (istio.io/dataplane-mode=ambient) to the same namespace.

Installing with Helm

In production, using Helm for fine-grained configuration management is recommended.

# add the Istio Helm repository
helm repo add istio https://istio-release.storage.googleapis.com/charts
helm repo update

# install istio-base (CRDs)
helm install istio-base istio/base -n istio-system --create-namespace

# install istiod (ambient profile)
helm install istiod istio/istiod -n istio-system \
  --set profile=ambient \
  --set pilot.resources.requests.memory=256Mi \
  --set pilot.resources.requests.cpu=200m

# install ztunnel
helm install ztunnel istio/ztunnel -n istio-system \
  --set resources.requests.memory=64Mi \
  --set resources.requests.cpu=50m

# install istio-cni
helm install istio-cni istio/cni -n istio-system \
  --set ambient.enabled=true

Configuring Traffic Policy

L4 AuthorizationPolicy

These are the L4-level access control policies handled by ztunnel. They work without a waypoint proxy.

# L4 AuthorizationPolicy: allow access from specific services only
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: my-app
spec:
  targetRefs:
    - kind: Service
      group: ''
      name: backend
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - 'cluster.local/ns/my-app/sa/frontend'
      to:
        - operation:
            ports: ['8080']
---
# L4 default-deny policy
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: deny-all
  namespace: my-app
spec:
  targetRefs:
    - kind: Service
      group: ''
      name: backend
  action: DENY
  rules: []

L7 AuthorizationPolicy (Waypoint Required)

Fine-grained access control based on HTTP path and method becomes available once a waypoint proxy is deployed.

# deploy the waypoint proxy first
# istioctl waypoint apply -n my-app

# L7 AuthorizationPolicy: HTTP path-based control
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: api-access-control
  namespace: my-app
spec:
  targetRefs:
    - kind: Service
      group: ''
      name: api-server
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - 'cluster.local/ns/my-app/sa/web-frontend'
      to:
        - operation:
            methods: ['GET']
            paths: ['/api/v1/products/*']
    - from:
        - source:
            principals:
              - 'cluster.local/ns/my-app/sa/admin-service'
      to:
        - operation:
            methods: ['GET', 'POST', 'PUT', 'DELETE']
            paths: ['/api/v1/*']

Traffic Splitting with a VirtualService

# traffic splitting for a canary deployment (waypoint required)
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata:
  name: reviews-canary
  namespace: my-app
spec:
  hosts:
    - reviews
  http:
    - match:
        - headers:
            x-canary:
              exact: 'true'
      route:
        - destination:
            host: reviews
            subset: v2
          weight: 100
    - route:
        - destination:
            host: reviews
            subset: v1
          weight: 90
        - destination:
            host: reviews
            subset: v2
          weight: 10
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: reviews-dr
  namespace: my-app
spec:
  host: reviews
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2

PeerAuthentication Configuration

# apply strict mTLS (enabled by default in ambient mode)
apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: strict-mtls
  namespace: my-app
spec:
  mtls:
    mode: STRICT

Observability Integration

Collecting Prometheus Metrics

In Ambient Mesh both ztunnel and the waypoint proxy expose Prometheus metrics.

# Prometheus ServiceMonitor for ztunnel
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: ztunnel-monitor
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: ztunnel
  endpoints:
    - port: http-monitoring
      path: /metrics
      interval: 15s
---
# Prometheus ServiceMonitor for waypoint proxies
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: waypoint-monitor
  namespace: my-app
spec:
  selector:
    matchLabels:
      gateway.istio.io/managed: istio.io-mesh-controller
  endpoints:
    - port: http-envoy-prom
      path: /stats/prometheus
      interval: 15s

Monitoring the Key Metrics

# check the ztunnel metrics
kubectl exec -n istio-system $(kubectl get pod -n istio-system -l app=ztunnel -o jsonpath='{.items[0].metadata.name}') -- curl -s localhost:15020/metrics | grep istio_tcp

# key ztunnel metrics:
# istio_tcp_connections_opened_total     - number of TCP connections opened
# istio_tcp_connections_closed_total     - number of TCP connections closed
# istio_tcp_sent_bytes_total             - bytes sent
# istio_tcp_received_bytes_total         - bytes received
# istio_tcp_connection_duration_seconds  - connection duration

# check the waypoint proxy metrics (L7 metrics included)
kubectl exec -n my-app $(kubectl get pod -n my-app -l gateway.istio.io/managed=istio.io-mesh-controller -o jsonpath='{.items[0].metadata.name}') -- curl -s localhost:15090/stats/prometheus | grep istio_request

# key waypoint metrics:
# istio_requests_total                   - total request count (L7)
# istio_request_duration_milliseconds    - request latency (L7)
# istio_request_bytes                    - request size
# istio_response_bytes                   - response size

Kiali Integration

Kiali supports Ambient Mesh and can visualize the service graph including ztunnel and the waypoint proxies.

# install Kiali (with ambient support)
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.24/samples/addons/kiali.yaml

# open the Kiali dashboard
istioctl dashboard kiali

Grafana Dashboards

# install the Grafana + Prometheus stack
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.24/samples/addons/prometheus.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.24/samples/addons/grafana.yaml

# open the Grafana dashboard
istioctl dashboard grafana

The key panels to have on a Grafana dashboard for Ambient Mesh:

• TCP connection count and byte traffic per ztunnel node
• HTTP request rate and error rate per waypoint proxy
• mTLS handshake success/failure rate
• HBONE tunnel latency

Migration Strategy

Moving from Sidecars to Ambient Mode

Migrating from the existing Istio sidecar mode to ambient mode has to be done in stages. The two modes can coexist in the same cluster, so a gradual namespace-by-namespace switch is possible.

Step 1: preparation

# check that the current Istio version supports ambient (1.22+)
istioctl version

# install the ambient profile components (keeping the existing istiod)
istioctl install --set profile=ambient --skip-confirmation

# confirm that ztunnel and istio-cni are working correctly
kubectl get daemonset -n istio-system ztunnel
kubectl get daemonset -n istio-system istio-cni-node

Step 2: start with non-critical namespaces

# validate in a test namespace first
# 1. remove the sidecar label
kubectl label namespace test-app istio-injection-

# 2. add the ambient label
kubectl label namespace test-app istio.io/dataplane-mode=ambient

# 3. restart the Pods to remove the existing sidecars
kubectl rollout restart deployment -n test-app

# 4. confirm the sidecars are gone and traffic goes through ztunnel
kubectl get pods -n test-app
# every Pod should show as 1/1 (no sidecar)

# 5. confirm mTLS communication
istioctl proxy-status

Step 3: deploy a waypoint in namespaces that need L7 features

# deploy a waypoint in namespaces that were using L7 policies
istioctl waypoint apply -n test-app

# confirm the existing L7 policies (VirtualService, AuthorizationPolicy and so on)
# work correctly through the waypoint
kubectl get gateway -n test-app

Step 4: switch the production namespaces once validated

# switch the production namespace (same procedure)
kubectl label namespace production istio-injection-
kubectl label namespace production istio.io/dataplane-mode=ambient
kubectl rollout restart deployment -n production

# deploy a waypoint if one is needed
istioctl waypoint apply -n production

Cautions During the Migration

HBONE compatibility: for a Pod in sidecar mode to talk to a Pod in ambient mode, the sidecar must have the ENABLE_HBONE flag set. Before the migration you have to update the sidecars to an HBONE-capable version and restart them.

Waypoint awareness: sidecars do not know about waypoint proxies. When a client in sidecar mode sends a request to a server in ambient mode, it can bypass the waypoint. Switching the services along the same communication path to the same mode is recommended wherever possible.

CNI compatibility: when Cilium is your CNI, Cilium by default tries to remove other CNI plugins. You have to prevent that with the cni.exclusive=false setting.

Troubleshooting

Common Problems and How to Fix Them

Problem 1: Pod-to-Pod communication fails (mTLS handshake error)

# Symptom: connections between Pods are refused or time out
# Cause: ztunnel is not working correctly, or there is a certificate problem

# Diagnosis 1: check the ztunnel status
kubectl get pods -n istio-system -l app=ztunnel
kubectl logs -n istio-system -l app=ztunnel --tail=100 | grep -i error

# Diagnosis 2: check the certificate status
istioctl proxy-status

# Diagnosis 3: check the workload identity
kubectl exec -n istio-system $(kubectl get pod -n istio-system -l app=ztunnel -o jsonpath='{.items[0].metadata.name}') -- curl -s localhost:15020/debug/workloadz

# Solution: restart ztunnel
kubectl rollout restart daemonset ztunnel -n istio-system

Problem 2: The L7 policy is not applied

# Symptom: the HTTP path/method rules in the AuthorizationPolicy are ignored
# Cause: the waypoint proxy is not deployed, or is not wired up correctly

# Diagnosis 1: check whether a waypoint proxy exists
kubectl get gateway -n my-app
istioctl waypoint list -n my-app

# Diagnosis 2: check the waypoint proxy Pod status
kubectl get pods -n my-app -l gateway.istio.io/managed=istio.io-mesh-controller

# Diagnosis 3: check the waypoint proxy logs
kubectl logs -n my-app -l gateway.istio.io/managed=istio.io-mesh-controller --tail=100

# Solution: redeploy the waypoint
istioctl waypoint delete -n my-app
istioctl waypoint apply -n my-app

Problem 3: CNI plugin conflict

# Symptom: ztunnel does not intercept traffic, and Pods behave as if outside the mesh
# Cause: a conflict between istio-cni and the existing CNI plugin

# Diagnosis: check the istio-cni logs
kubectl logs -n istio-system -l k8s-app=istio-cni-node --tail=100

# Fix (on Cilium):
# disable Cilium's exclusive CNI setting
helm upgrade cilium cilium/cilium -n kube-system \
  --set cni.exclusive=false

# restart istio-cni
kubectl rollout restart daemonset istio-cni-node -n istio-system

Problem 4: Trouble talking to services outside the mesh

# Symptom: a Pod inside the ambient mesh cannot reach a service outside the mesh
# Cause: ztunnel intercepts the external traffic too and tries to HBONE-tunnel it

# Diagnosis: look for errors about external destinations in the ztunnel logs
kubectl logs -n istio-system -l app=ztunnel --tail=200 | grep "external"

# Solution: register the external service explicitly with a ServiceEntry
cat <<EOF | kubectl apply -f -
apiVersion: networking.istio.io/v1
kind: ServiceEntry
metadata:
  name: external-api
  namespace: my-app
spec:
  hosts:
    - api.external-service.com
  location: MESH_EXTERNAL
  ports:
    - number: 443
      name: https
      protocol: TLS
  resolution: DNS
EOF

Problem 5: ztunnel memory usage spikes

# Symptom: the ztunnel Pod's memory usage keeps climbing
# Cause: a large number of concurrent connections, or a memory leak

# Diagnosis: check the current memory usage
kubectl top pods -n istio-system -l app=ztunnel

# Diagnosis: check the connection count
kubectl exec -n istio-system $(kubectl get pod -n istio-system -l app=ztunnel -o jsonpath='{.items[0].metadata.name}') -- curl -s localhost:15020/metrics | grep tcp_connections

# Solution: adjust the ztunnel resource limits
helm upgrade ztunnel istio/ztunnel -n istio-system \
  --set resources.limits.memory=256Mi \
  --set resources.requests.memory=128Mi

Production Checklist

These are the items you must check before deploying Ambient Mesh to a production environment.

Before Installation

• Confirm the Kubernetes cluster version is 1.27 or later
• Confirm the Istio version is 1.22 or later (ambient Beta)
• Verify CNI plugin compatibility (Calico, Cilium and so on)
• Confirm the Gateway API CRDs are installed
• Confirm there is spare node capacity (for the ztunnel DaemonSet)

Network Configuration

• Confirm the sidecar-mode and ambient-mode namespaces do not overlap
• Confirm the HBONE port (15008) is allowed by the inter-node firewall
• Confirm the ztunnel metrics port (15020) is reachable from Prometheus
• Confirm ServiceEntries are configured for services outside the mesh

Security Configuration

• Confirm PeerAuthentication is set to STRICT mTLS
• Confirm the L4 AuthorizationPolicy default-deny is in place
• Confirm a waypoint proxy is deployed in every namespace that needs L7 policy
• Confirm SPIFFE ID-based service-to-service access control is configured

Observability

• Confirm Prometheus is configured to collect the ztunnel and waypoint metrics
• Confirm the Grafana dashboard has ambient-mesh-specific panels
• Confirm the Kiali version supports ambient mode
• Confirm alert rules are set for ztunnel failures and waypoint error rates

High Availability

• Confirm the ztunnel DaemonSet's updateStrategy is RollingUpdate
• Confirm the waypoint proxy has at least 2 replicas
• Confirm a PodDisruptionBudget is set on the waypoint proxy
• Confirm istiod has at least 2 replicas

Migration (When Switching from Sidecars)

• Confirm validation was completed in a non-critical namespace first
• Confirm the sidecar-mode Pods are on an HBONE-capable version
• Confirm the existing VirtualService/DestinationRule work correctly through the waypoint
• Confirm the rollback procedure is documented

Failure Cases and Recovery Procedures

Case 1: Service Interruption During a ztunnel DaemonSet Update

Situation: while updating the ztunnel DaemonSet, the old ztunnel Pod on a node terminated before the new one started, and all mesh traffic on that node stopped.

Cause: maxUnavailable was set too high, or the new ztunnel Pod was slow to start.

Recovery procedure:

# 1. identify the affected nodes
kubectl get pods -n istio-system -l app=ztunnel -o wide

# 2. pause the update (where possible)
kubectl rollout pause daemonset ztunnel -n istio-system

# 3. recover ztunnel manually on the affected node
kubectl delete pod -n istio-system $(kubectl get pod -n istio-system -l app=ztunnel --field-selector spec.nodeName=affected-node -o jsonpath='{.items[0].metadata.name}')

# 4. confirm ztunnel has recovered
kubectl get pods -n istio-system -l app=ztunnel -o wide

# Preventive measure: set maxUnavailable to 1
kubectl patch daemonset ztunnel -n istio-system -p '{"spec":{"updateStrategy":{"rollingUpdate":{"maxUnavailable":1}}}}'

Case 2: L7 Policy Latency from Waypoint Proxy Overload

Situation: traffic concentrated on one namespace's waypoint proxy and response latency spiked.

Cause: the waypoint proxy replica count was too low for the traffic volume.

Recovery procedure:

# 1. check the waypoint proxy status
kubectl get pods -n affected-namespace -l gateway.istio.io/managed=istio.io-mesh-controller
kubectl top pods -n affected-namespace -l gateway.istio.io/managed=istio.io-mesh-controller

# 2. scale out immediately
kubectl scale deployment -n affected-namespace $(kubectl get deployment -n affected-namespace -l gateway.istio.io/managed=istio.io-mesh-controller -o jsonpath='{.items[0].metadata.name}') --replicas=5

# 3. apply autoscaling with an HPA (see the HPA example above)

# 4. if necessary, temporarily relax the L7 policy down to L4
# removing the waypoint disables the L7 policy and falls back to L4 (ztunnel)
# istioctl waypoint delete -n affected-namespace
# (caution: this disables every L7 policy)

Case 3: Communication Failures in Mixed Sidecar/Ambient Mode

Situation: during the migration, calls from a service in a sidecar-mode namespace to a service in an ambient-mode namespace failed intermittently.

Cause: the sidecar was an older version without HBONE protocol support, or the sidecar did not know about the waypoint proxy and sent traffic straight to the backend.

Recovery procedure:

# 1. check whether the sidecar supports HBONE
kubectl exec -n sidecar-namespace deployment/my-app -c istio-proxy -- pilot-agent request GET /debug/config_dump | grep ENABLE_HBONE

# 2. if HBONE is disabled, restart the sidecar to pick up the latest configuration
kubectl rollout restart deployment -n sidecar-namespace

# 3. if the problem persists, switch that namespace to ambient mode too
kubectl label namespace sidecar-namespace istio-injection-
kubectl label namespace sidecar-namespace istio.io/dataplane-mode=ambient
kubectl rollout restart deployment -n sidecar-namespace

Case 4: The Effect of an istiod Failure on the Ambient Mesh

Situation: every istiod Pod became unhealthy and certificate renewal stopped.

Cause: insufficient istiod resources, an etcd connection problem, or a webhook misconfiguration.

Recovery procedure:

# 1. check the istiod status
kubectl get pods -n istio-system -l app=istiod
kubectl logs -n istio-system -l app=istiod --tail=200

# 2. restart istiod
kubectl rollout restart deployment istiod -n istio-system

# 3. check the certificate renewal status
istioctl proxy-status

# Note: even with istiod down, existing mTLS connections survive until the certificates expire
# the default certificate TTL is 24 hours, so istiod has to be recovered within 24 hours

# Preventive measure: an HA istiod configuration
helm upgrade istiod istio/istiod -n istio-system \
  --set pilot.replicaCount=3 \
  --set pilot.resources.requests.memory=512Mi \
  --set pilot.resources.requests.cpu=500m

References

• Istio Ambient Mesh official documentation - the Istio project's official Ambient Mesh overview and architecture description
• Istio Ambient Data Plane architecture - the detailed data plane architecture document for ztunnel and the waypoint proxy
• Sidecar or Ambient? - the official Istio comparison guide - the official comparison of sidecar mode and ambient mode
• Istio Ambient Mesh Beta announcement (v1.22) - the official blog announcing that ambient mode reached Beta
• Introducing the Rust-based ztunnel - the official Istio blog - the technical detail behind ztunnel's Rust-based implementation
• Traffic in Ambient Mesh: Ztunnel, eBPF, Waypoint Proxies - Solo.io - an analysis of traffic flow between ztunnel, eBPF and the waypoint proxy
• Network Cost Comparison: Sidecar vs Ambient - Tetrate - a comparative analysis of network cost between sidecar and ambient mode
• Guide to migrating from sidecars to Ambient Mesh - the guide to switching from sidecars to ambient mode
• Zero-downtime migration from sidecars to ambient - Solo.io - a zero-downtime migration strategy
• Istio Ambient L7 Flow Analysis - Jimmy Song - an analysis of the ztunnel-to-waypoint-proxy flow in L7 traffic management