Kubernetes Multi-Cluster Upgrade Runbook 2026

• Runbook Overview
  • Who This Applies To
  • Version Skew Policy Summary
• Preparation Checklist
  • D-7: Analyze the Changes
  • D-5: Upgrade the Staging Cluster
  • D-3: Staging Verification Complete
  • D-1: Final Preparation for the Production Upgrade
• Production Upgrade Procedure
  • Phase 1: Declare the Start of the Upgrade
  • Phase 2: Sequential Multi-Cluster Upgrade
    • Upgrade Commands per Managed Service
  • Phase 3: Per-Cluster Health Check
  • Phase 4: Traffic Shifting and Observation
• Rollback Procedure
  • Rollback Decision Criteria
  • Rolling Back a Managed Service
  • Multi-Cluster Traffic Rollback
• Troubleshooting: Real Failure Scenarios During a Multi-Cluster Upgrade
  • Scenario 1: drain Fails on a PDB Violation During a Node Group Upgrade
  • Scenario 2: Addon Incompatibility After the Upgrade
  • Scenario 3: Service Mesh Failure from a Version Mismatch Between Clusters
  • Scenario 4: Wrong EKS Addon Upgrade Order
• Post-Upgrade Observation Period
  • What to Observe and for How Long
  • Observation Period Definition
• Postmortem Template
• Automation: Upgrading from a CI/CD Pipeline
• References

Runbook Overview

This document is a step-by-step procedure (runbook) that a team operating two or more Kubernetes clusters should follow when carrying out a version upgrade. It is written so that judgement calls during an incident are kept to a minimum and the work follows a procedure decided in advance.

Who This Applies To

• Multi-region or multi-environment (dev/staging/prod) cluster operations
• Environments mixing managed services (EKS, GKE, AKS) with self-managed clusters (kubeadm, kOps)
• Target upgrade version: v1.32 → v1.33 (or a similar minor version upgrade)

Version Skew Policy Summary

Kubernetes does not allow skipping a minor version. Upgrades must be done sequentially.

Preparation Checklist

Complete the items below in order, starting at D-7 (one week before the upgrade).

D-7: Analyze the Changes

#!/bin/bash
# d7-changelog-review.sh
# Collect the changes in the target upgrade version

TARGET_VERSION="v1.33"

echo "=== ${TARGET_VERSION} change check ==="

echo ""
echo "[1] Deprecated API check"
echo "  - Check the Removed/Deprecated sections in the release notes"
echo "  - Whether an Endpoints API → EndpointSlice migration is needed"
echo "  - flowcontrol.apiserver.k8s.io/v1beta3 → v1 transition"

echo ""
echo "[2] Feature Gate change check"
echo "  - InPlacePodVerticalScaling: Beta (enabled by default)"
echo "  - UserNamespacesSupport: Stable"
echo "  - SidecarContainers: Stable"
echo "  - NFTablesProxyMode: Stable"

echo ""
echo "[3] Compatibility check for the addons in use"
# Check the compatibility matrix of each addon
ADDONS=(
  "calico"
  "cilium"
  "ingress-nginx"
  "cert-manager"
  "external-dns"
  "prometheus-operator"
  "argocd"
)

for addon in "${ADDONS[@]}"; do
  echo "  - $addon: [  ] compatibility check done"
done

D-5: Upgrade the Staging Cluster

Always upgrade staging before production. If there is no staging cluster, run the upgrade on the dev cluster first.

# List the clusters in the multi-cluster environment
kubectl config get-contexts

# Switch to the staging cluster context
kubectl config use-context staging-cluster

# Run the staging upgrade (managed service example: EKS)
aws eks update-cluster-version \
  --name staging-cluster \
  --kubernetes-version 1.33

# Monitor the upgrade status
aws eks describe-update \
  --name staging-cluster \
  --update-id <update-id>

D-3: Staging Verification Complete

After observing staging for at least 48 hours, check the items below.

- [ ] All nodes Ready
- [ ] All system Pods Running
- [ ] Application Pods behaving normally
- [ ] Ingress/Service traffic normal
- [ ] CronJob succeeded at least once
- [ ] HPA/VPA behaving normally
- [ ] Monitoring metric collection normal
- [ ] Log pipeline normal
- [ ] No deprecated API warnings
- [ ] Performance test passed (response time, throughput)

D-1: Final Preparation for the Production Upgrade

#!/bin/bash
# d1-final-prep.sh

echo "=== D-1 final preparation ==="

# 1. Change announcement
echo "[1] Change announcement sent: [ ]"
echo "  - Internal team Slack announcement"
echo "  - External status page update"
echo "  - Maintenance window shared"

# 2. On-call owners
echo "[2] On-call owners:"
echo "  - Primary: ___________"
echo "  - Secondary: ___________"
echo "  - Escalation path: Primary → Secondary → Tech Lead"

# 3. Rollback preparation
echo "[3] Rollback readiness:"
echo "  - [ ] etcd snapshot created"
echo "  - [ ] Previous version binaries kept"
echo "  - [ ] Rollback procedure document reviewed"
echo "  - [ ] Estimated rollback time: ___ minutes"

# 4. Maintenance window
echo "[4] Work window:"
echo "  - Start: ___:___ (KST)"
echo "  - Expected end: ___:___ (KST)"
echo "  - Hard limit: ___:___ (KST)"

Production Upgrade Procedure

Phase 1: Declare the Start of the Upgrade

# Announce the start of work through the Slack webhook
curl -X POST "$SLACK_WEBHOOK_URL" \
  -H 'Content-type: application/json' \
  -d '{
    "text": ":wrench: [MAINTENANCE] Kubernetes upgrade starting\nCluster: prod-cluster-01\nTarget version: v1.33\nOperator: @oncall\nExpected duration: 2 hours"
  }'

Phase 2: Sequential Multi-Cluster Upgrade

In a multi-cluster environment, upgrade sequentially starting from the cluster with the smallest share of traffic.

Upgrade order:
1. canary-cluster (5% of traffic) ← upgrade first, observe for 1 hour
2. prod-cluster-02 (30% of traffic) ← after canary is stable
3. prod-cluster-01 (65% of traffic) ← last

Upgrade Commands per Managed Service

EKS:

# Upgrade the Control Plane
aws eks update-cluster-version \
  --name prod-cluster-01 \
  --kubernetes-version 1.33

# Wait for the upgrade to finish
aws eks wait cluster-active --name prod-cluster-01

# Upgrade the Managed Node Group
aws eks update-nodegroup-version \
  --cluster-name prod-cluster-01 \
  --nodegroup-name workers-general \
  --kubernetes-version 1.33

# Upgrade the addons
for ADDON in vpc-cni coredns kube-proxy; do
  LATEST=$(aws eks describe-addon-versions \
    --addon-name $ADDON \
    --kubernetes-version 1.33 \
    --query 'addons[0].addonVersions[0].addonVersion' \
    --output text)

  aws eks update-addon \
    --cluster-name prod-cluster-01 \
    --addon-name $ADDON \
    --addon-version $LATEST
done

GKE:

# Upgrade the Control Plane
gcloud container clusters upgrade prod-cluster-01 \
  --master \
  --cluster-version 1.33.0-gke.100 \
  --zone asia-northeast3-a

# Upgrade the Node Pool
gcloud container clusters upgrade prod-cluster-01 \
  --node-pool workers-general \
  --cluster-version 1.33.0-gke.100 \
  --zone asia-northeast3-a

AKS:

# Check the versions available for upgrade
az aks get-upgrades \
  --resource-group myResourceGroup \
  --name prod-cluster-01 \
  --output table

# Upgrade the Control Plane and Node Pool together
az aks upgrade \
  --resource-group myResourceGroup \
  --name prod-cluster-01 \
  --kubernetes-version 1.33.0

Phase 3: Per-Cluster Health Check

Always run this after each cluster upgrade completes.

#!/bin/bash
# cluster-health-check.sh <cluster-context>
CONTEXT=$1

echo "=== Health Check: $CONTEXT ==="
kubectl --context "$CONTEXT" get nodes -o wide

# Check the version of every node
echo ""
echo "Node versions:"
kubectl --context "$CONTEXT" get nodes \
  -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.nodeInfo.kubeletVersion}{"\n"}{end}'

# System Pod status
echo ""
echo "System pods:"
FAILING=$(kubectl --context "$CONTEXT" -n kube-system get pods --no-headers | \
  grep -v "Running\|Completed" | wc -l)
echo "Failing system pods: $FAILING"

# Production workload status
echo ""
echo "Production workloads:"
kubectl --context "$CONTEXT" -n production get deployments -o wide

# Check for Pending Pods
echo ""
echo "Pending pods:"
kubectl --context "$CONTEXT" get pods -A --field-selector=status.phase=Pending

# Check events (last 10 minutes)
echo ""
echo "Recent warnings:"
kubectl --context "$CONTEXT" get events -A --sort-by='.lastTimestamp' \
  --field-selector type=Warning | tail -20

Phase 4: Traffic Shifting and Observation

How you control traffic across clusters depends on your infrastructure.

# Example of AWS Route53 weight-based routing
# Shift traffic to the canary cluster gradually

# Step 1: 5% of traffic to canary
# prod-cluster-01: weight 65
# prod-cluster-02: weight 30
# canary-cluster: weight 5

# Step 2: confirm canary is stable, then wait 30 minutes

# Step 3: 20% of traffic to canary
# prod-cluster-01: weight 50
# prod-cluster-02: weight 30
# canary-cluster: weight 20

# Update the Route53 weights
aws route53 change-resource-record-sets \
  --hosted-zone-id Z1234567890 \
  --change-batch '{
    "Changes": [{
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "api.example.com",
        "Type": "A",
        "SetIdentifier": "canary",
        "Weight": 20,
        "AliasTarget": {
          "HostedZoneId": "Z9876543210",
          "DNSName": "canary-alb.ap-northeast-2.elb.amazonaws.com",
          "EvaluateTargetHealth": true
        }
      }
    }]
  }'

Rollback Procedure

If a problem appears after the upgrade, decide whether to roll back using the criteria below.

Rollback Decision Criteria

Rolling Back a Managed Service

Managed services often do not support downgrading the Control Plane. Instead, replace the node group with one on the previous version.

# EKS: create a node group on the previous version
aws eks create-nodegroup \
  --cluster-name prod-cluster-01 \
  --nodegroup-name workers-rollback \
  --kubernetes-version 1.32 \
  --node-role arn:aws:iam::123456789:role/eks-node-role \
  --subnets subnet-abc subnet-def \
  --instance-types m6i.xlarge \
  --scaling-config minSize=3,maxSize=10,desiredSize=5

# Move the Pods from the old node group to the new one
kubectl cordon -l eks.amazonaws.com/nodegroup=workers-general
kubectl drain -l eks.amazonaws.com/nodegroup=workers-general \
  --ignore-daemonsets --delete-emptydir-data

# Delete the old node group
aws eks delete-nodegroup \
  --cluster-name prod-cluster-01 \
  --nodegroup-name workers-general

Multi-Cluster Traffic Rollback

Remove traffic from the cluster where the problem occurred.

# Set the traffic weight of the affected cluster to 0
aws route53 change-resource-record-sets \
  --hosted-zone-id Z1234567890 \
  --change-batch '{
    "Changes": [{
      "Action": "UPSERT",
      "ResourceRecordSet": {
        "Name": "api.example.com",
        "Type": "A",
        "SetIdentifier": "canary",
        "Weight": 0,
        "AliasTarget": {
          "HostedZoneId": "Z9876543210",
          "DNSName": "canary-alb.ap-northeast-2.elb.amazonaws.com",
          "EvaluateTargetHealth": true
        }
      }
    }]
  }'

Troubleshooting: Real Failure Scenarios During a Multi-Cluster Upgrade

Scenario 1: drain Fails on a PDB Violation During a Node Group Upgrade

$ kubectl drain node-xyz --ignore-daemonsets
error: Cannot evict pod as it would violate the pod's disruption budget.

Cause: the PDB's minAvailable equals the current number of Running Pods, so not even one can be evicted.

Fix:

# Check the PDB status
kubectl get pdb -A -o wide

# Example output:
# NAMESPACE   NAME       MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS
# production  web-pdb    3               N/A               0
#                                                          ↑ 0 means drain is impossible

# Option 1: raise the replica count first to create headroom
kubectl -n production scale deployment web --replicas=4
# Retry drain after 1-2 minutes

# Option 2: relax the PDB temporarily (caution: availability drops)
kubectl -n production patch pdb web-pdb \
  --type=merge -p '{"spec":{"minAvailable":2}}'
# Restore it after the drain finishes
kubectl -n production patch pdb web-pdb \
  --type=merge -p '{"spec":{"minAvailable":3}}'

Scenario 2: Addon Incompatibility After the Upgrade

$ kubectl -n kube-system logs ingress-nginx-controller-xyz
Error: the server could not find the requested resource (get ingresses.networking.k8s.io)

Cause: the Ingress Controller version is not compatible with the new Kubernetes API version.

Fix:

# Check compatible addon versions and upgrade
helm repo update

# Check the compatible ingress-nginx versions
helm search repo ingress-nginx/ingress-nginx --versions | head -10

# Upgrade to a compatible version
helm upgrade ingress-nginx ingress-nginx/ingress-nginx \
  --namespace kube-system \
  --version 4.12.0 \
  --reuse-values

Scenario 3: Service Mesh Failure from a Version Mismatch Between Clusters

# Cluster A: v1.33, cluster B: v1.32
# Some traffic routing fails in the Istio multi-cluster setup
$ istioctl proxy-status
NAME                   CLUSTER      CDS   LDS   EDS   RDS
web-v1-xyz.production  cluster-a    SYNCED SYNCED SYNCED SYNCED
web-v1-abc.production  cluster-b    STALE  STALE  STALE  STALE

Cause: you need to confirm that the Istio version supports the Kubernetes versions on both clusters.

Fix:

# Check Istio compatibility
istioctl version

# Standardize on an Istio version supported by both clusters
# Istio 1.24+ supports Kubernetes v1.31~v1.33
istioctl upgrade --set revision=1-24-0

Scenario 4: Wrong EKS Addon Upgrade Order

$ aws eks update-addon --addon-name vpc-cni ...
An error occurred (InvalidParameterException): Addon version v1.19.0 is not
compatible with cluster version 1.33. Available versions: v1.19.2, v1.20.0

Fix:

# Query the latest compatible version
aws eks describe-addon-versions \
  --addon-name vpc-cni \
  --kubernetes-version 1.33 \
  --query 'addons[0].addonVersions[:5].{version:addonVersion,default:compatibilities[0].defaultVersion}' \
  --output table

# Update to a compatible version
aws eks update-addon \
  --cluster-name prod-cluster-01 \
  --addon-name vpc-cni \
  --addon-version v1.20.0 \
  --resolve-conflicts OVERWRITE

Post-Upgrade Observation Period

What to Observe and for How Long

Observation Period Definition

0-2 hours after the upgrade: intensive observation (check every 5 minutes)
2-24 hours after the upgrade: normal observation (every 30 minutes)
24-72 hours after the upgrade: alert-driven observation (respond to anomaly alerts)
72 hours onward: return to normal operations, write the postmortem

Postmortem Template

Write the postmortem within 72 hours of finishing the upgrade. Write it whether or not there was an incident, and feed it into the next upgrade.

# Cluster Upgrade Postmortem

## Basic Information

- Target cluster: \_\_\_
- Previous version: v1.32.x → current version: v1.33.x
- Work window: YYYY-MM-DD HH:MM ~ HH:MM (total \_\_\_ hours)
- Operator: \_\_\_

## Progress Summary

- [ ] Staging upgrade: no problems / problems occurred (details: \_\_\_)
- [ ] Production upgrade: no problems / problems occurred (details: \_\_\_)
- [ ] Rollback executed: Yes / No

## Issues Found

| Issue  | Severity | Resolution | Time taken |
| ------ | -------- | ---------- | ---------- |
| \_\_\_ | P1~P4    | \_\_\_     | \_\_\_ min |

## Improvements (to apply at the next upgrade)

1. ***
2. ***
3. ***

## Documents That Need Updating

- [ ] This runbook
- [ ] On-call guide
- [ ] Monitoring dashboards

Automation: Upgrading from a CI/CD Pipeline

An example of automating the recurring multi-cluster upgrade with GitHub Actions.

# .github/workflows/cluster-upgrade.yml
name: Kubernetes Cluster Upgrade

on:
  workflow_dispatch:
    inputs:
      target_version:
        description: 'Target Kubernetes version'
        required: true
        default: '1.33'
      cluster:
        description: 'Cluster to upgrade'
        required: true
        type: choice
        options:
          - canary-cluster
          - prod-cluster-02
          - prod-cluster-01
      dry_run:
        description: 'Dry run mode'
        required: true
        type: boolean
        default: true

jobs:
  pre-check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: ${{ secrets.AWS_ROLE_ARN }}
          aws-region: ap-northeast-2

      - name: Pre-upgrade checks
        run: |
          aws eks describe-cluster --name ${{ inputs.cluster }} \
            --query 'cluster.{version:version,status:status}' \
            --output table

          # Check the current version
          CURRENT=$(aws eks describe-cluster --name ${{ inputs.cluster }} \
            --query 'cluster.version' --output text)
          echo "Current version: $CURRENT"
          echo "Target version: ${{ inputs.target_version }}"

          # Check addon compatibility
          for ADDON in vpc-cni coredns kube-proxy; do
            echo "Checking $ADDON compatibility..."
            aws eks describe-addon-versions \
              --addon-name $ADDON \
              --kubernetes-version ${{ inputs.target_version }} \
              --query 'addons[0].addonVersions[0].addonVersion' \
              --output text
          done

  upgrade:
    needs: pre-check
    runs-on: ubuntu-latest
    if: ${{ !inputs.dry_run }}
    environment: production # manual approval required
    steps:
      - name: Upgrade Control Plane
        run: |
          aws eks update-cluster-version \
            --name ${{ inputs.cluster }} \
            --kubernetes-version ${{ inputs.target_version }}

          echo "Waiting for control plane upgrade..."
          aws eks wait cluster-active --name ${{ inputs.cluster }}

      - name: Upgrade Node Groups
        run: |
          NODEGROUPS=$(aws eks list-nodegroups \
            --cluster-name ${{ inputs.cluster }} \
            --query 'nodegroups[]' --output text)

          for NG in $NODEGROUPS; do
            echo "Upgrading node group: $NG"
            aws eks update-nodegroup-version \
              --cluster-name ${{ inputs.cluster }} \
              --nodegroup-name $NG \
              --kubernetes-version ${{ inputs.target_version }}
          done

      - name: Upgrade Addons
        run: |
          for ADDON in vpc-cni coredns kube-proxy; do
            LATEST=$(aws eks describe-addon-versions \
              --addon-name $ADDON \
              --kubernetes-version ${{ inputs.target_version }} \
              --query 'addons[0].addonVersions[0].addonVersion' \
              --output text)

            aws eks update-addon \
              --cluster-name ${{ inputs.cluster }} \
              --addon-name $ADDON \
              --addon-version $LATEST \
              --resolve-conflicts OVERWRITE
          done

  post-verify:
    needs: upgrade
    runs-on: ubuntu-latest
    steps:
      - name: Health Check
        run: |
          aws eks update-kubeconfig --name ${{ inputs.cluster }}

          echo "Node status:"
          kubectl get nodes -o wide

          echo "System pods:"
          kubectl -n kube-system get pods

          echo "Failing pods:"
          kubectl get pods -A --field-selector=status.phase!=Running,status.phase!=Succeeded

      - name: Notify
        run: |
          curl -X POST "${{ secrets.SLACK_WEBHOOK }}" \
            -H 'Content-type: application/json' \
            -d "{
              \"text\": \":white_check_mark: Cluster upgrade complete\nCluster: ${{ inputs.cluster }}\nVersion: ${{ inputs.target_version }}\"
            }"

References

• Kubernetes v1.33 Octarine release
• Official Kubernetes cluster upgrade documentation
• kubeadm upgrade guide
• AWS EKS Upgrade Best Practices
• GKE Cluster Upgrade Best Practices
• Kubernetes release information