Tag: #kubernetes
Writing on GPUs, LLMs, MLOps, Kubernetes — and mindset · 247 posts
Fixing Kubernetes Pod Pending — How to Read the Rejection Reason the Scheduler Left Behind
When a Pod sits in Pending for minutes on end, the scheduler has already recorded the rejection reason in an event. We start with how to decode precisely that sentence beginning with 0/5 nodes are available, then rule ou
2026-07-26 · 14 min read #kubernetes#scheduler#pending#troubleshooting#autoscalingConnection refused vs. timeout — narrowing the cause at the TCP level
When a connection fails, the terminal hands back one of two messages. Connection refused means the peer returned an RST; timeout means nothing at all answered the SYN you sent. That one-line difference almost entirely de
2026-07-26 · 12 min read #network#tcp#troubleshooting#linux#kubernetesping works but only large requests hang — dissecting MTU, MSS and the PMTUD blackhole
When small requests go through fine but only large responses stall halfway, it is almost always a path MTU problem. TCP advertises MSS based on the interface MTU, and when a narrower segment sits somewhere along the path
2026-07-26 · 14 min read #network#mtu#tcp#vpn#kubernetesCPU steal time and throttling — telling apart the st column in top, burstable credits, and CFS quota
CPU utilization reads 40%, yet p99 latency spikes and the st column in top shows 20%. Three completely different causes hide behind these look-alike symptoms: steal time, where the hypervisor does not hand a physical CPU
2026-07-26 · 17 min read #linux#cpu#cgroups#kubernetes#cloudWhen the OOM Killer Kills a Process — From Decoding the dmesg Report to Telling cgroup OOM Apart
A process vanished without leaving a single log line and the exit code is 137. This post covers how to read the dmesg report left behind by the kernel OOM Killer one line at a time, how the badness score is computed and
2026-07-26 · 18 min read #linux#memory#oom#cgroups#kubernetesdig works but the application fails — start by checking the DNS resolution order
A situation where dig answers normally but only the application cannot find the name is not a bug, it is structure. dig and nslookup bypass /etc/hosts and nsswitch.conf entirely and query the resolver directly. This post
2026-07-26 · 14 min read #network#dns#linux#kubernetes#troubleshootingDiagnosing Kubernetes CrashLoopBackOff by Cause — What to Look At When the Logs Are Empty
A start-to-finish walkthrough of the situation where a Pod is stuck in CrashLoopBackOff but kubectl logs prints nothing. It starts with the precise meaning of BackOff — not a cause, but a restart delay that grows from 10
2026-07-26 · 14 min read #kubernetes#crashloopbackoff#troubleshooting#kubectl#sreKubernetes ImagePullBackOff and ErrImagePull Fully Dissected — Ending It With One Cause String
Breaks down, cause by cause, the situation where a pod passes through ErrImagePull and then sits in ImagePullBackOff. It starts from two points: that the two states are different stages of the same event, and that the an
2026-07-26 · 12 min read #kubernetes#imagepullbackoff#container-registry#troubleshooting#containerdUsing the Three Kubernetes Probes Properly — The Exact Boundaries of liveness, readiness, and startup
Three incidents — a pod with nothing wrong with it restarting on a schedule, a flood of 502s only right after a deploy, and a slow-starting application that never reaches Ready — all come out of probe design. After separ
2026-07-26 · 17 min read #kubernetes#liveness-probe#readiness-probe#graceful-shutdown#sreKubernetes OOMKilled(137) Memory Troubleshooting — What to Check Before You Raise the Limit
This post covers the situation where a container dies with Exit Code 137 while the application log leaves behind no exception at all. It starts from the fact that 137 tells you only one thing, 128 plus 9, that is SIGKILL
2026-07-26 · 14 min read #kubernetes#oomkilled#memory#jvm#cgroupTalos Linux 1.13 — How a Shell-less Immutable OS Ended Up Shipping a Debug Shell
Talos Linux, the Kubernetes-only immutable OS, shipped 1.13 on April 27, 2026. The release brings a kernel built with Clang/ThinLTO, reproducible disk images, machine-wide container image signature verification, and its
2026-07-17 · 13 min read #linux#kubernetes#talos#immutable-infrastructure#devopsFlux 2.9 and Two Years After Weaveworks — How a GitOps Project Survives Losing Its Sponsor
When Weaveworks, the company that created Flux, shut down in early 2024, a GitHub discussion appeared asking: is the project future at risk? Two and a half years later, on 2026-06-30, Flux shipped 2.9.0. This post recons
2026-07-17 · 12 min read #devops#gitops#fluxcd#kubernetes#open-sourceOTel's Kubernetes Attributes Are Stable Now — What to Do Before k8sattributes Flips Its Default
OpenTelemetry's Kubernetes semantic conventions were promoted to stable in semconv v1.42.0 on June 12, 2026. But the Collector's k8sattributes processor still defaults to the old schema (v0), so most people haven't notic
2026-07-16 · 12 min read #opentelemetry#observability#kubernetes#semantic-conventions#telemetry-pipelineMove to Ambient and EnvoyFilter Goes Silently Ignored — The Gap Istio 1.30's TrafficExtension Closes, and What It Doesn't
The real obstacle blocking migration to ambient mode isn't resources — it's extensibility. Istio's official migration doc states outright that EnvoyFilter is not supported on waypoints, that it will be "silently ignored"
2026-07-16 · 15 min read #kubernetes#istio#service-mesh#envoy#ambient-meshThe Workload/PodGroup API in Kubernetes v1.36 — Gang Scheduling Is Moving Into kube-scheduler
Gang scheduling for AI training and batch workloads has so far been the job of external schedulers like Volcano or Kueue, but Kubernetes has started pulling this capability into its core. v1.35 shipped the Workload API a
2026-07-16 · 19 min read #kubernetes#scheduling#gang-scheduling#distributed-training#draWhy KubeVirt GPU Passthrough VMs Could Not Be Scheduled for 112 Days — A Real Cluster Postmortem
After a Rust operator delivered the diagnosis that "all 4 GPU nodes are NotReady," I traced the cause of that death all the way to the end on a real 8-node cluster (GPU Operator v25.3.0, KubeVirt v1.7.0). The real reason
2026-07-11 · 5 min read #kubevirt#gpu#kubernetes#nvidia#devopsGood tools are invisible: gingerBill on friction as a feature, and where "invisible" hides failure
A reflection on gingerBill's essay 'Good Tools Are Invisible,' which argues that a good tool disappears into the background and that reselling friction as a fun puzzle is a rationalization. I agree with his feeling-versu
2026-07-11 · 7 min read #tools#developer-experience#kubernetes#build-systems#editorsSpinning Up and Killing Postgres on Kubernetes with CloudNativePG — Failover Measured at 23 Seconds
On a real 8-node Kubernetes cluster, I installed CloudNativePG (CNPG) v1.30.0, brought up a 3-instance Postgres cluster, and then actually killed the primary. From bootstrap through replication checks, to failover after
2026-07-11 · 5 min read #cloudnativepg#postgresql#kubernetes#operator#databaseBuilding a Kubernetes GPU Operator in Rust — Diagnosing a Real Cluster with kube-rs
Against a production 8-node homelab cluster (k8s v1.32.5), I used kube-rs to build and run a GPU operator in Rust myself. I defined a GpuInventory custom resource and launched two controllers (node scan → record CR statu
2026-07-11 · 6 min read #rust#kubernetes#operator#gpu#kube-rsGPU Operator × KubeVirt Complete Guide — Components, Configuration, Versions, Partial MIG, and Manual MIG
The two pillars of GPU infrastructure on Kubernetes, all on one page. Covers the GPU Operator operands, ClusterPolicy configuration, and versioning scheme, plus a custom config that applies MIG to only some of the GPUs o
2026-07-09 · 9 min read #kubernetes#gpu#kubevirt#mig#nvidia