- Introduction — What It Means to Hand Postgres Over to an Operator
- Part 1 — Installation: A Single Manifest
- Part 2 — Building a 3-Instance Cluster
- Part 3 — Verifying Replication
- Part 4 — Killing the Primary: 23.1-Second Failover
- Part 5 — Honest Numbers and Pitfalls
- Closing
- References
Introduction — What It Means to Hand Postgres Over to an Operator
Running a stateful database on Kubernetes was long treated almost as a taboo. CloudNativePG (CNPG) is an operator that breaks that taboo head-on — declare Postgres's high availability, failover, backups, and rolling upgrades with a single CRD, and the operator handles primary election and replica management for you. This article is not an introduction but a measurement: I installed CNPG on a real 8-node cluster, brought up a 3-instance Postgres, and actually killed the primary to measure how many seconds failover takes. It is the counterpart to the Rust GPU operator piece where I wrote an operator myself — this is the story of the "using a well-built operator" side.
Part 1 — Installation: A Single Manifest
Installing CNPG is minimal. Apply a single release manifest and you're done.
kubectl apply --server-side -f \
https://raw.githubusercontent.com/cloudnative-pg/cloudnative-pg/release-1.30/releases/cnpg-1.30.0.yaml
This one line installs everything — several CRDs (clusters, poolers, scheduledbackups, publications, subscriptions, and more), the controller Deployment in the cnpg-system namespace, RBAC, and webhooks. The controller took about 20 seconds to come up:
$ kubectl -n cnpg-system rollout status deploy/cnpg-controller-manager
deployment "cnpg-controller-manager" successfully rolled out
NAME READY UP-TO-DATE AVAILABLE AGE
cnpg-controller-manager 1/1 1 1 16s
Part 2 — Building a 3-Instance Cluster
A Postgres cluster is declared with a Cluster CR. I set it to 1 primary + 2 replicas, i.e., 3 instances.
apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
name: pg-test
namespace: cnpg-test
spec:
instances: 3
storage:
size: 1Gi
storageClass: nfs-client # NFS in the homelab — the pitfall is in Part 5
bootstrap:
initdb:
database: appdb
owner: appuser
Once applied, the operator begins bootstrapping. I watched the state transitions in real time:
Setting up primary
→ Waiting for the instances to become active
→ ready=1 Creating a new replica ← replica cloning starts once the primary is up
→ ready=2 Creating a new replica
→ ready=3 Cluster in healthy state ← 3/3 healthy in about 2 minutes
The three instances were automatically spread across different nodes (cubi02, cubi03, cubi04) — the operator uses anti-affinity to keep them from piling onto a single node. CNPG also creates three connection services for you:
Service Role
──────────── ─────────────────────────────
pg-test-rw read/write → always routed to the current primary
pg-test-ro read-only → load-balanced across replicas
pg-test-r any instance (reads)
The -rw service is the key — if the application looks at just this one name, then even when the primary changes due to failover, it automatically connects to the new primary.
Part 3 — Verifying Replication
I inserted 1000 rows into the primary and checked whether all three instances matched.
$ INSERT 1000 rows into the primary (pg-test-1)
INSERT 0 1000
$ Row count per instance
pg-test-1 (primary): 1000 rows
pg-test-2 (replica): 1000 rows ← replicated
pg-test-3 (replica): 1000 rows ← replicated
Streaming replication synchronized all three nodes instantly. Now for the real experiment.
Part 4 — Killing the Primary: 23.1-Second Failover
The thing I was most curious about — if the primary suddenly disappears, how many seconds until recovery? I instantly killed the primary pod with --grace-period=0 --force, then measured the time by polling at 0.5-second intervals until a new primary appeared.
=== FAILOVER TEST: killing primary pg-test-1 ===
deleted at t0; polling for new primary...
=== NEW PRIMARY: pg-test-2 (was pg-test-1) ===
failover time: 23.1 s
rows after failover: 1000 ← zero data loss
In 23.1 seconds, pg-test-2 was promoted to the new primary, and all 1000 rows were still there. And the pg-test-1 that had died is not thrown away — the operator automatically brings it back and re-enrolls it as a replica:
=== Final roles after self-healing ===
pg-test-1: replica ← died and came back, demoted to replica
pg-test-2: primary ← the newly promoted primary
pg-test-3: replica
$ Additional writes to the new primary → OK, 1500 rows total
The new primary immediately accepted writes (1000→1500 rows), and the cluster returned to 3/3 healthy. Zero human intervention. This is what an operator is worth.
Part 5 — Honest Numbers and Pitfalls
In keeping with the principle that this blog writes only what has been verified, I leave the limitations of this experiment intact, too.
- Is 23 seconds fast? It depends on the situation. CNPG's failover time is the sum of detecting the primary's death (the health-check interval), promoting a replica, and updating the
-rwservice endpoint. In production, node failures are more common than pod deletions, and in that case the node-detection time (node-monitor-grace-period, etc.) is added on, which can make it longer. Conversely, with tuning it gets shorter. The "23 seconds" is a measured value for this homelab and this configuration, not a universal constant. - The NFS storage pitfall. I used
nfs-client(NFS provisioner) storage, but putting Postgres on NFS is not recommended in production — because of fsync guarantees and file-locking issues. It ran fine as a homelab test, but for a real service you should use local SSD or block storage (Ceph RBD, etc.). - Synchronous vs. asynchronous replication. This test used the default (asynchronous) replication. With asynchronous replication, in theory a tiny number of unreplicated transactions right before the primary's death can be lost. If you need zero loss, you have to turn on CNPG's synchronous replication (
minSyncReplicas), and in exchange write latency increases.
Closing
CNPG refutes, with real measurements, the conventional wisdom that "databases on Kubernetes are dangerous" — a single manifest brings up a 3-node HA Postgres, it recovers on its own within 23 seconds even when you kill the primary outright, and the dead node comes back as a replica. Of course the real homework of storage, replication mode, and failover tuning remains, but that is not a question of "can you hand a DB to an operator" — it is a question of "how do you hand it over well." Next, I plan to verify backups (ScheduledBackup) and point-in-time recovery (PITR) the same way, by killing things.