Kubernetes Gateway API Production Guide: HTTPRoute, GRPCRoute, and Ingress Migration Strategy

• Why Gateway API Replaces Ingress
  • Architecture Overview
  • Role Separation Model
• Comparison: Gateway API vs Ingress vs Service Mesh
• Installing Gateway API CRDs and a Controller
  • Step 1: Install the Gateway API CRDs
  • Step 2: Deploy a Gateway Controller
• HTTPRoute: Core Concepts and Patterns
  • Basic HTTPRoute Configuration
  • Advanced: HTTP-to-HTTPS Redirect and Request Modification
• GRPCRoute: Native gRPC Traffic Management
  • GRPCRoute Configuration
  • When to Use GRPCRoute vs HTTPRoute for gRPC
• Step-by-Step Ingress Migration
  • Migration Architecture
  • Phase 1: Audit Existing Ingress Resources
  • Phase 2: Create Gateway Resources Alongside Ingress
  • Phase 3: Create HTTPRoutes Matching Ingress Rules
  • Phase 4: Gradual Traffic Shift
  • Phase 5: Delete Ingress Resources One at a Time
• Traffic Splitting and Canary Deployments
• TLS Termination Patterns
  • Gateway-Level TLS with cert-manager
• Production Troubleshooting
  • Common Failure Cases and Recovery
  • Operational Warnings
  • Health Check Script for Production
• Implementation Comparison: Envoy Gateway vs NGINX Gateway Fabric vs Istio
• Real-World Implementation Pattern: Multi-Team Platform
• Summary
• References
• Quiz

Why Gateway API Replaces Ingress

The Kubernetes Ingress resource served the community well for years, but its limitations became a bottleneck for modern workloads. Vendor-specific annotations, no native gRPC support, no role separation, and a single-resource-for-everything design all drove the creation of the Gateway API.

Gateway API (GA since v1.0, with GRPCRoute GA since v1.1.0) introduces a role-oriented, protocol-aware, and portable networking model. As of v1.2, the old v1alpha2 versions of GRPCRoute and ReferenceGrant have been removed from both Standard and Experimental channels, signaling maturity and production-readiness.

Architecture Overview

┌─────────────────────────────────────────────────────────────────┐
│                    Cluster Administrator                        │
│                                                                 │
│  ┌──────────────┐                                               │
│  │ GatewayClass │  Defines which controller handles Gateways    │
│  └──────┬───────┘                                               │
│         │                                                       │
│         ▼                                                       │
│  ┌──────────────┐     Platform / Infra Team                     │
│  │   Gateway    │  Configures listeners, TLS, allowed routes    │
│  └──┬───┬───┬───┘                                               │
│     │   │   │                                                   │
│     ▼   ▼   ▼      Application Developer                       │
│  ┌────┐┌─────────┐┌──────────┐                                  │
│  │HTTP││GRPCRoute││TLSRoute  │  Define per-service routing      │
│  │Route│└─────────┘└──────────┘                                 │
│  └─┬──┘                                                        │
│    │                                                            │
│    ▼                                                            │
│  ┌──────────────────────────┐                                   │
│  │  Backend Services / Pods │                                   │
│  └──────────────────────────┘                                   │
└─────────────────────────────────────────────────────────────────┘

Role Separation Model

This separation means developers never need to touch TLS certificates, and platform operators never need to understand application-level routing logic.

Comparison: Gateway API vs Ingress vs Service Mesh

Installing Gateway API CRDs and a Controller

Step 1: Install the Gateway API CRDs

# Install the Standard Channel CRDs (includes HTTPRoute, GRPCRoute GA resources)
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/standard-install.yaml

# Verify CRDs are installed
kubectl get crds | grep gateway
# Expected output:
# gatewayclasses.gateway.networking.k8s.io
# gateways.gateway.networking.k8s.io
# grpcroutes.gateway.networking.k8s.io
# httproutes.gateway.networking.k8s.io
# referencegrants.gateway.networking.k8s.io

# (Optional) Install Experimental Channel for TCPRoute, TLSRoute, UDPRoute
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.2.1/experimental-install.yaml

Step 2: Deploy a Gateway Controller

Choose an implementation that fits your needs. Popular options:

# Option A: Envoy Gateway
helm install eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.3.0 \
  -n envoy-gateway-system --create-namespace

# Option B: NGINX Gateway Fabric
helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric \
  --version 2.4.2 \
  -n nginx-gateway --create-namespace

# Option C: Istio (with Gateway API support)
istioctl install --set profile=minimal

HTTPRoute: Core Concepts and Patterns

HTTPRoute is the workhorse of Gateway API. It handles HTTP and terminated HTTPS connections with rich matching capabilities.

Basic HTTPRoute Configuration

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: envoy-gateway
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: production
  namespace: infra
spec:
  gatewayClassName: envoy-gateway
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      tls:
        mode: Terminate
        certificateRefs:
          - name: wildcard-tls
            namespace: cert-manager
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              gateway-access: 'true'
    - name: http
      protocol: HTTP
      port: 80
---
apiVersion: gateway.networking.k8s.io/v1
kind: ReferenceGrant
metadata:
  name: allow-cert-ref
  namespace: cert-manager
spec:
  from:
    - group: gateway.networking.k8s.io
      kind: Gateway
      namespace: infra
  to:
    - group: ''
      kind: Secret
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-routes
  namespace: app-team
spec:
  parentRefs:
    - name: production
      namespace: infra
  hostnames:
    - 'api.example.com'
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v2/users
          headers:
            - name: X-API-Version
              value: '2'
      backendRefs:
        - name: users-v2
          port: 8080
          weight: 90
        - name: users-v3
          port: 8080
          weight: 10
    - matches:
        - path:
            type: PathPrefix
            value: /v2/users
      backendRefs:
        - name: users-v2
          port: 8080
    - matches:
        - path:
            type: PathPrefix
            value: /healthz
      backendRefs:
        - name: health-service
          port: 8080

Advanced: HTTP-to-HTTPS Redirect and Request Modification

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: redirect-to-https
  namespace: app-team
spec:
  parentRefs:
    - name: production
      namespace: infra
      sectionName: http
  hostnames:
    - 'api.example.com'
  rules:
    - filters:
        - type: RequestRedirect
          requestRedirect:
            scheme: https
            statusCode: 301
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: api-with-modifications
  namespace: app-team
spec:
  parentRefs:
    - name: production
      namespace: infra
      sectionName: https
  hostnames:
    - 'api.example.com'
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /api
      filters:
        - type: RequestHeaderModifier
          requestHeaderModifier:
            add:
              - name: X-Request-ID
                value: 'generated-by-gateway'
            remove:
              - X-Internal-Debug
        - type: URLRewrite
          urlRewrite:
            path:
              type: ReplacePrefixMatch
              replacePrefixMatch: /v1
      backendRefs:
        - name: api-backend
          port: 8080

GRPCRoute: Native gRPC Traffic Management

GRPCRoute provides first-class support for gRPC traffic. Unlike wrapping gRPC in HTTPRoute with path matching, GRPCRoute understands gRPC service and method names directly.

GRPCRoute Configuration

apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
  name: grpc-user-service
  namespace: grpc-apps
spec:
  parentRefs:
    - name: production
      namespace: infra
  hostnames:
    - 'grpc.example.com'
  rules:
    # Route by gRPC service name
    - matches:
        - method:
            service: com.example.UserService
            method: GetUser
      backendRefs:
        - name: user-service
          port: 50051
    # Route by service name with header matching
    - matches:
        - method:
            service: com.example.UserService
          headers:
            - name: x-canary
              value: 'true'
      backendRefs:
        - name: user-service-canary
          port: 50051
    # Default route for all other gRPC methods
    - matches:
        - method:
            service: com.example.UserService
      backendRefs:
        - name: user-service-stable
          port: 50051
          weight: 95
        - name: user-service-canary
          port: 50051
          weight: 5

When to Use GRPCRoute vs HTTPRoute for gRPC

Step-by-Step Ingress Migration

Migrating from Ingress to Gateway API should be done incrementally. Never perform a hard cutover in production.

Migration Architecture

                    ┌──────────────────────┐
                    │      DNS / LB        │
                    │  api.example.com     │
                    └──────────┬───────────┘
                               │
                    ┌──────────▼───────────┐
                    │   Traffic Split      │
                    │  (DNS weight or LB)  │
                    └──┬───────────────┬───┘
                       │               │
              ┌────────▼─────┐  ┌──────▼────────┐
              │   Ingress    │  │   Gateway      │
              │  (existing)  │  │   (new)        │
              │              │  │                │
              │  nginx-ctrl  │  │  envoy-gw      │
              └──────┬───────┘  └──────┬─────────┘
                     │                 │
                     └────────┬────────┘
                              │
                     ┌────────▼────────┐
                     │  Same Backend   │
                     │  Services       │
                     └─────────────────┘

Phase 1: Audit Existing Ingress Resources

# List all Ingress resources across namespaces
kubectl get ingress -A -o wide

# Export all Ingress resources for analysis
kubectl get ingress -A -o yaml > ingress-backup.yaml

# Check for annotation dependencies
kubectl get ingress -A -o json | \
  jq -r '.items[] | .metadata.name + " -> " + (.metadata.annotations | keys | join(", "))'

# Use ingress2gateway for automated conversion
# Install: go install github.com/kubernetes-sigs/ingress2gateway@latest
ingress2gateway print --all-namespaces

Phase 2: Create Gateway Resources Alongside Ingress

# Deploy Gateway alongside existing Ingress
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: migration-gateway
  namespace: infra
  annotations:
    purpose: 'ingress-migration-phase2'
spec:
  gatewayClassName: envoy-gateway
  listeners:
    - name: https
      protocol: HTTPS
      port: 443
      tls:
        mode: Terminate
        certificateRefs:
          - name: wildcard-tls
      allowedRoutes:
        namespaces:
          from: All

Phase 3: Create HTTPRoutes Matching Ingress Rules

# Verify the Gateway is Programmed before proceeding
kubectl get gateway migration-gateway -n infra -o jsonpath='{.status.conditions}'

# Expected: type=Programmed, status=True
# WARNING: Do NOT delete Ingress resources until Gateway shows Programmed=True

Phase 4: Gradual Traffic Shift

# Validate each HTTPRoute individually
for route in $(kubectl get httproute -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}'); do
  ns=$(echo $route | cut -d/ -f1)
  name=$(echo $route | cut -d/ -f2)
  echo "--- Checking $ns/$name ---"
  kubectl get httproute $name -n $ns -o jsonpath='{.status.parents[*].conditions}' | jq .
done

# Test with curl before switching DNS
GATEWAY_IP=$(kubectl get gateway migration-gateway -n infra \
  -o jsonpath='{.status.addresses[0].value}')
curl -H "Host: api.example.com" https://$GATEWAY_IP/healthz --resolve "api.example.com:443:$GATEWAY_IP"

Phase 5: Delete Ingress Resources One at a Time

# Delete one Ingress at a time, verify, then proceed
kubectl delete ingress old-api-ingress -n app-team

# Immediately verify the service is reachable
curl -f https://api.example.com/healthz || echo "ROLLBACK NEEDED"

# If rollback is needed:
kubectl apply -f ingress-backup.yaml

Traffic Splitting and Canary Deployments

Gateway API makes canary deployments a first-class feature with weight-based traffic splitting.

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: canary-deployment
  namespace: app-team
spec:
  parentRefs:
    - name: production
      namespace: infra
  hostnames:
    - 'app.example.com'
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        # Stable: receives 95% of traffic
        - name: app-stable
          port: 8080
          weight: 95
        # Canary: receives 5% of traffic
        - name: app-canary
          port: 8080
          weight: 5

To gradually shift traffic:

# Increase canary weight incrementally
# 5% -> 10% -> 25% -> 50% -> 100%
# At each step, check error rates and latency

# Monitor canary metrics (Prometheus example)
# rate(http_requests_total{app="app-canary",code=~"5.."}[5m])
# / rate(http_requests_total{app="app-canary"}[5m])

TLS Termination Patterns

Gateway-Level TLS with cert-manager

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: secure-gateway
  namespace: infra
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  gatewayClassName: envoy-gateway
  listeners:
    - name: https-wildcard
      protocol: HTTPS
      port: 443
      hostname: '*.example.com'
      tls:
        mode: Terminate
        certificateRefs:
          - name: wildcard-example-com
    - name: https-specific
      protocol: HTTPS
      port: 443
      hostname: 'api.specific.com'
      tls:
        mode: Terminate
        certificateRefs:
          - name: api-specific-cert
    - name: tls-passthrough
      protocol: TLS
      port: 8443
      hostname: 'mtls.example.com'
      tls:
        mode: Passthrough

Production Troubleshooting

Common Failure Cases and Recovery

Problem 1: Gateway stuck in "not Programmed" state

# Check Gateway status conditions
kubectl describe gateway production -n infra

# Common causes:
# 1. GatewayClass controller not running
kubectl get pods -n envoy-gateway-system

# 2. Invalid TLS certificate reference
kubectl get secret wildcard-tls -n infra

# 3. Listener conflict (duplicate port/hostname)
kubectl get gateway -A -o jsonpath='{range .items[*]}{.metadata.name}: {range .spec.listeners[*]}{.port}/{.hostname} {end}{"\n"}{end}'

Problem 2: HTTPRoute not attaching to Gateway

# Check route status
kubectl get httproute api-routes -n app-team -o yaml | yq '.status'

# Common causes:
# 1. Missing ReferenceGrant for cross-namespace
kubectl get referencegrant -A

# 2. Namespace not allowed by Gateway listener
kubectl get ns app-team --show-labels
# Ensure the label matches the Gateway's allowedRoutes selector

# 3. Hostname mismatch between Gateway listener and HTTPRoute
# The HTTPRoute hostname must be a subdomain of or match the listener hostname

Problem 3: 503 errors after migration

# Check if backend endpoints are healthy
kubectl get endpoints users-v2 -n app-team

# Check if the service port matches the HTTPRoute backendRef port
kubectl get svc users-v2 -n app-team -o jsonpath='{.spec.ports}'

# Check gateway controller logs
kubectl logs -n envoy-gateway-system -l app=envoy-gateway --tail=100

# Verify the Envoy proxy config was generated correctly
# (Envoy Gateway specific)
kubectl get envoyproxy -A

Problem 4: gRPC requests failing with UNAVAILABLE

# Verify the backend supports HTTP/2
kubectl exec -it deploy/user-service -n grpc-apps -- \
  grpcurl -plaintext localhost:50051 list

# Check if the Gateway listener protocol is correct
# For gRPC, you need HTTPS (with TLS termination) or
# the controller must support HTTP/2 cleartext (h2c)

# Check GRPCRoute status
kubectl get grpcroute -A -o yaml | yq '.items[].status'

Operational Warnings

1. Never delete all Ingress resources at once. Migrate one service at a time and verify reachability after each deletion.

2. Always check Gateway Programmed status before routing traffic. A Gateway that is Accepted but not Programmed has not provisioned its data plane.

3. ReferenceGrant is required for cross-namespace references. Without it, a route in namespace A cannot reference a backend in namespace B or a secret in namespace C.

4. Gateway API v1.2 removed v1alpha2 GRPCRoute. Before upgrading, ensure your controller supports v1 GRPCRoute. Run kubectl get crds grpcroutes.gateway.networking.k8s.io -o jsonpath='{.spec.versions[*].name}' to check.

5. Weight-based splitting rounds to the nearest integer. If you set weights of 1 and 99, the actual split may vary slightly depending on the controller implementation.

6. TLS Passthrough listeners cannot inspect HTTP headers. If you need header-based routing, use TLS Terminate mode instead.

Health Check Script for Production

#!/bin/bash
# gateway-health-check.sh
# Run after any Gateway API change in production

set -euo pipefail

echo "=== Gateway Status ==="
kubectl get gateways -A -o custom-columns=\
'NAMESPACE:.metadata.namespace,NAME:.metadata.name,CLASS:.spec.gatewayClassName,PROGRAMMED:.status.conditions[?(@.type=="Programmed")].status,ACCEPTED:.status.conditions[?(@.type=="Accepted")].status'

echo ""
echo "=== HTTPRoute Status ==="
kubectl get httproutes -A -o custom-columns=\
'NAMESPACE:.metadata.namespace,NAME:.metadata.name,HOSTNAMES:.spec.hostnames[*],ACCEPTED:.status.parents[*].conditions[?(@.type=="Accepted")].status'

echo ""
echo "=== GRPCRoute Status ==="
kubectl get grpcroutes -A -o custom-columns=\
'NAMESPACE:.metadata.namespace,NAME:.metadata.name,ACCEPTED:.status.parents[*].conditions[?(@.type=="Accepted")].status' 2>/dev/null || echo "No GRPCRoutes found"

echo ""
echo "=== ReferenceGrants ==="
kubectl get referencegrants -A

echo ""
echo "=== Remaining Ingress Resources (should be empty post-migration) ==="
kubectl get ingress -A

Implementation Comparison: Envoy Gateway vs NGINX Gateway Fabric vs Istio

Real-World Implementation Pattern: Multi-Team Platform

# Platform team deploys shared Gateway
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: platform-gateway
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: shared-gateway
  namespace: platform
spec:
  gatewayClassName: platform-gateway
  listeners:
    - name: https-web
      protocol: HTTPS
      port: 443
      hostname: '*.web.example.com'
      tls:
        mode: Terminate
        certificateRefs:
          - name: web-wildcard
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              tier: web
    - name: https-api
      protocol: HTTPS
      port: 443
      hostname: '*.api.example.com'
      tls:
        mode: Terminate
        certificateRefs:
          - name: api-wildcard
      allowedRoutes:
        namespaces:
          from: Selector
          selector:
            matchLabels:
              tier: api
---
# Team A deploys their own routes (no TLS knowledge needed)
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: team-a-frontend
  namespace: team-a
spec:
  parentRefs:
    - name: shared-gateway
      namespace: platform
      sectionName: https-web
  hostnames:
    - 'app.web.example.com'
  rules:
    - backendRefs:
        - name: frontend
          port: 3000
---
# Team B deploys gRPC routes independently
apiVersion: gateway.networking.k8s.io/v1
kind: GRPCRoute
metadata:
  name: team-b-grpc
  namespace: team-b
spec:
  parentRefs:
    - name: shared-gateway
      namespace: platform
      sectionName: https-api
  hostnames:
    - 'payments.api.example.com'
  rules:
    - matches:
        - method:
            service: com.example.PaymentService
      backendRefs:
        - name: payment-grpc
          port: 50051

Summary

The Kubernetes Gateway API represents a fundamental shift in how we manage cluster networking. Key takeaways:

• HTTPRoute is the primary routing resource for HTTP/HTTPS traffic with rich matching, filtering, and traffic splitting capabilities.
• GRPCRoute provides native gRPC service/method-level routing without annotation workarounds.
• Migration from Ingress should be incremental: install Gateway API CRDs, deploy a controller, create routes alongside existing Ingress, verify, then delete Ingress resources one by one.
• Role separation (GatewayClass, Gateway, Route) enables platform teams and application teams to work independently.
• Always verify the Gateway status shows Programmed: True before relying on it for production traffic.

The ecosystem is mature: Envoy Gateway, NGINX Gateway Fabric, and Istio all provide robust Gateway API implementations. Choose based on your existing stack and whether you need east-west (mesh) capabilities.

References

• Kubernetes Gateway API Official Documentation
• KEP-2649: Gateway API Enhancement Proposal
• Envoy Gateway Documentation
• NGINX Gateway Fabric Documentation
• Istio Gateway API Support
• Migrating from Ingress - Gateway API Guide
• ingress2gateway Tool - Kubernetes Blog
• Gateway API v1.2 Release Blog

Quiz