Tag: #llm-inference

한국어English日本語中文

← All posts

• vLLM Metrics — What to Chart and What to Alert On

  The series vLLM exposes answer questions GPU metrics cannot: how many requests are running versus waiting right now, how full the KV cache is, how long until the first token. This post reads the official vLLM documentati

  2026-08-12 · 7 min read #gpu#kubernetes#vllm#prometheus#observability

• Making vLLM Fast — Configuration, Internals, and Where to Actually Touch the Code

  A step-by-step walk through improving vLLM performance, starting from everything you can fix without touching code. Covers batching-related arguments, prefix caching, chunked prefill, and quantization choice first, then

  2026-08-02 · 21 min read #vllm#llm-inference#paged-attention#benchmark#scheduler

• llama.cpp Comes to the Browser — The Ceiling a WebGPU Backend Measured Across 16 Devices

  LlamaWeb, the WebGPU backend for llama.cpp released by a UC Santa Cruz team in May 2026, left behind the widest dataset to date measuring browser LLM inference — 16 devices across 8 vendors. The results cut both ways. It

  2026-07-16 · 16 min read #webgpu#llm-inference#llama-cpp#on-device-ai#browser

• AI Inference Engines 2026 - vLLM · SGLang · llama.cpp · TGI · TensorRT-LLM · MLX · mistral.rs · DeepSpeed-MII · Aphrodite Deep Dive

  In 2026, LLM engineering is no longer about which model — it is about which inference engine. We dissect vLLM V1, SGLang 0.4, TensorRT-LLM, TGI 3.x, llama.cpp, MLX-LM, mistral.rs, DeepSpeed-MII, Aphrodite, CTranslate2, E

  2026-05-16 · 20 min read #llm-inference#vllm#sglang#llama-cpp#tgi

• Complete LLM Inference Optimization Guide 2025: vLLM, TensorRT-LLM, KV Cache, Speculative Decoding

  Everything about LLM inference optimization! vLLM (PagedAttention), TensorRT-LLM (FP8/INT4), KV Cache management, Speculative Decoding, Continuous Batching, FlashAttention, Quantization (GPTQ/AWQ/GGUF), Model Serving (Tr

  2026-04-14 · 25 min read #llm-inference#vllm#tensorrt-llm#kv-cache#speculative-decoding

• NPU Deep Dive: How Transformer Architecture Runs Directly on Silicon

  A complete technical breakdown of how NPUs differ from CPUs and GPUs, how every transformer operation maps to hardware, and why LLM inference is memory-bound — not compute-bound. From Apple ANE to Qualcomm Hexagon, Groq

  2026-03-18 · 22 min read #npu#transformer#ai-hardware#quantization#kv-cache

• The New Throne Beyond vLLM: 5 Reasons SGLang Is Changing the LLM Inference Landscape

  From SGLang's RadixAttention-based KV cache management and 29% throughput advantage through hyper-specialized design, to its 4,000-line Python zero-overhead scheduler, Prefill-Decode Disaggregation, and Compressed FSM-ba

  2026-03-01 · 35 min read #sglang#llm-inference#vllm#radix-attention#model-serving