Tag: #gpu

한국어English日本語中文

← All posts

• A Three-Line Config File Killed Four GPUs for a Week — How containerd Drop-in Merging Really Works

  The nodes stopped advertising GPUs. The nvidia runtime was written out cleanly in the config file, but it was nowhere to be found in containerd config dump. The culprit was a three-line registry config I had added a week

  2026-08-26 · 12 min read #kubernetes#containerd#gpu#nvidia#troubleshooting

• vLLM Metrics — What to Chart and What to Alert On

  The series vLLM exposes answer questions GPU metrics cannot: how many requests are running versus waiting right now, how full the KV cache is, how long until the first token. This post reads the official vLLM documentati

  2026-08-12 · 7 min read #gpu#kubernetes#vllm#prometheus#observability

• MIG and Time-Slicing — Two Ways to Share One GPU

  There are broadly two ways to put multiple workloads on one GPU: time-slicing, which divides time, and MIG, which divides hardware. Despite sounding similar, their isolation guarantees are nothing alike. This post works

  2026-08-12 · 6 min read #gpu#kubernetes#mig#time-slicing#nvidia

• DCGM Exporter — GPU Utilization Is Not What You Think It Is

  DCGM Exporter is the standard path for exposing GPU telemetry in Prometheus format, but the utilization-style metric that ends up on nearly every dashboard does not measure what people expect. This post reads the default

  2026-08-12 · 9 min read #gpu#kubernetes#dcgm#prometheus#observability

• A GPU Troubleshooting Playbook — Fix the Layers, Then Walk Down

  The biggest waste in diagnosing GPU problems on Kubernetes is poking around without an order. A pod that will not schedule, a pod that runs but cannot see the GPU, a driver and toolkit version mismatch, memory exhaustion

  2026-08-12 · 7 min read #gpu#kubernetes#troubleshooting#nvidia#gpu-operator

• NVIDIA GPU Operator — The Six Pieces You Used to Install by Hand

  Running GPUs on Kubernetes used to mean matching six pieces on every node by hand: the driver, the NVIDIA Container Toolkit, the device plugin, DCGM, GPU Feature Discovery, and Node Feature Discovery. The NVIDIA GPU Oper

  2026-08-12 · 6 min read #gpu#kubernetes#gpu-operator#nvidia#dcgm

• GPU Serving SLOs and Alert Design — What to Promise and What to Page On

  Putting an SLO on a GPU inference service starts with deciding which metric represents the user experience. Time to first token and throughput eat each other, so setting a target on only one guarantees the other collapse

  2026-08-12 · 7 min read #gpu#kubernetes#slo#alerting#prometheus

• Device Plugins and GPU Scheduling — Where nvidia.com/gpu Comes From

  Kubernetes does not know what a GPU is. What makes a node advertise GPUs as a resource is a device plugin registered with the kubelet, and the name that falls out of it is the extended resource nvidia.com/gpu. This post

  2026-08-12 · 6 min read #gpu#kubernetes#device-plugin#scheduling#nvidia

• Inside vLLM (7) — Deployment Tuning, Common Pitfalls, and OOM Triage

  A practical order for tuning a vLLM deployment. Covers what gpumemoryutilization actually sets, when to use tensor parallelism versus pipeline parallelism, how to choose quantization and a KV cache data type, and a diagn

  2026-08-12 · 10 min read #vllm#gpu#quantization#tensor-parallel#llm

• Inside vLLM (2) — Why PagedAttention Splits the KV Cache Into Pages

  Explains why PagedAttention splits the KV cache into fixed-size blocks, verified against the original paper (arXiv:2309.06180) and the official vLLM design documentation. Covers the internal and external fragmentation th

  2026-08-12 · 7 min read #vllm#paged-attention#kv-cache#llm#gpu

• Inside vLLM (1) — The Full Path From One Request to One Token

  Follows the full path a single request takes through vLLM, from arrival to the first token out. Covers what the API server, scheduler, KV cache manager, worker, and sampler each do, and how the process layout splits afte

  2026-08-12 · 6 min read #vllm#llm#inference#gpu#ai-platform

• GPU Compiler and Framework Landscape — One Problem, Turning a Graph into a Kernel, a Different Answer at Every Layer

  This post puts NVCC and PTX, LLVM, MLIR, Triton, torch.compile, XLA, IREE, and TVM on one map. Different names, different owners, but they all solve the same problem: turning a computation graph into an executable kernel

  2026-08-02 · 21 min read #gpu#compiler#mlir#triton#pytorch

• Label Locality Scheduling in Ray 2.56 — Placement Groups Start Seeing NVLink Racks, Not Nodes

  Ray 2.56.0, released on June 29, 2026, adds a domain-level scheduling layer to placement groups as an alpha feature. Until now every placement strategy — PACK, STRICTPACK and friends — operated strictly at node granulari

  2026-07-17 · 11 min read #ray#gpu#scheduling#distributed-systems

• How Much VRAM to Run an LLM Locally — Compute It From Formulas, Not Tables

  The answer to "how many GB does an 8B model need?" is not a table but two formulas. Weights are parameter count times bpw divided by 8; the KV cache is 2 times layers times KV heads times headdim times bytes times tokens

  2026-07-17 · 24 min read #llm#quantization#kv-cache#local-llm#gpu

• Triton Gluon: The Language That Hand-Writes the Layouts the Compiler Used to Hide

  Gluon is a low-level GPU language built on the same compiler stack as Triton — it hands kernel authors the layouts, shared memory, and warp specialization that Triton normally keeps hidden. Its reason for existing is cle

  2026-07-16 · 19 min read #gpu#triton#kernel#compiler#performance

• Rust 1.97 Cut Off Pre-Volta GPUs — Inside the nvptx64 Baseline Bump

  Rust 1.97 (July 9, 2026) raised the minimum requirements for the nvptx64-nvidia-cuda target to PTX ISA 7.0 (CUDA 11 driver or newer) and SM 7.0 (Volta or newer). Maxwell- and Pascal-generation GPUs, and CUDA 10 and older

  2026-07-16 · 13 min read #rust#cuda#gpu#compiler#nvidia

• Running Small Models Hands-On with a Single RTX 5090 — microGPT, OCR, Music Generation

  I SSHed into a single RTX 5090 (Blackwell, 32GB) and ran a trio of small models by hand. I trained a char-level GPT from scratch in 28 seconds (10.75M parameters, 1.17M tokens/s), pitted a dedicated OCR model (TrOCR) aga

  2026-07-11 · 8 min read #pytorch#gpu#llm#ocr#hands-on

• Why KubeVirt GPU Passthrough VMs Could Not Be Scheduled for 112 Days — A Real Cluster Postmortem

  After a Rust operator delivered the diagnosis that "all 4 GPU nodes are NotReady," I traced the cause of that death all the way to the end on a real 8-node cluster (GPU Operator v25.3.0, KubeVirt v1.7.0). The real reason

  2026-07-11 · 5 min read #kubevirt#gpu#kubernetes#nvidia#devops

• Diffusion LLMs That Write CUDA Kernels — DICE and Why Parallel Generation Might Help

  DICE, a February 2026 preprint, claims that diffusion large language models beat autoregressive models of the same size at generating CUDA kernels, setting a new state of the art. The core idea: instead of writing one to

  2026-07-11 · 6 min read #ai#llm#diffusion#cuda#gpu

• Building a Kubernetes GPU Operator in Rust — Diagnosing a Real Cluster with kube-rs

  Against a production 8-node homelab cluster (k8s v1.32.5), I used kube-rs to build and run a GPU operator in Rust myself. I defined a GpuInventory custom resource and launched two controllers (node scan → record CR statu

  2026-07-11 · 6 min read #rust#kubernetes#operator#gpu#kube-rs