Tag: #inference

한국어English日本語中文

← All posts

• On-Device and Edge AI — When AI Moves Inside the Device

  AI inference, once confined to the cloud, is moving into smartphones, PCs, and embedded devices. This post examines the three drivers (latency, privacy, cost), the rise of NPUs and on-device LLMs, and the investment and

  2026-06-18 · 19 min read #ai#edge-ai#on-device#npu#llm

• Making Inference Fast — Quantization, Sparsity, and Dataflow from a Hardware Lens

  We break down the cost structure of inference through the memory-wall lens, then connect quantization (INT8/FP8/FP4), structured sparsity (2:4), dataflow architectures, operator fusion, batching and KV caching into one p

  2026-06-16 · 18 min read #inference#quantization#sparsity#dataflow#gpu

• GPU vs TPU vs ASIC — The 2026 Inference War

  A comparison of the GPU, TPU, and ASIC competition over 2026 inference workloads. We cover Google TPU v6 Trillium and Ironwood, the fast-growing cloud in-house inference ASICs, the throughput/latency/cost/power trade-off

  2026-06-16 · 23 min read #gpu#tpu#asic#inference#ai-hardware

• Cerebras Wafer-Scale Deep Dive — A Whole Model on a Single Chip

  A close look at the design of the Cerebras WSE-3, a single chip carved from an entire wafer. We cover the on-chip SRAM-centric structure that routes around the memory wall, the fault-tolerant design, real-time inference

  2026-06-16 · 20 min read #cerebras#wafer-scale#ai-hardware#memory-wall#inference

• The Memory Wall and HBM — The Real Bottleneck That Divides AI Performance

  In an era where compute is cheap and data movement is expensive, the real bottleneck of AI performance is memory. From the memory-wall concept to HBM generations, the roofline model and arithmetic intensity, the KV cache

  2026-06-16 · 21 min read #memory-wall#hbm#bandwidth#roofline#inference

• Groq and SambaNova — Chips That Went All In on Inference

  A deep look at the working principles of two chips that bet everything on inference rather than training: the Groq LPU and the SambaNova RDU. We cover how deterministic execution and compiler scheduling, plus reconfigura

  2026-06-16 · 19 min read #groq#sambanova#inference#ai-hardware#lpu

• Edge AI and the NPU — On-Device Inference Accelerators

  We lay out why edge AI runs inference on the device instead of the cloud (latency, privacy, cost) and the concept of the NPU that makes it possible. From Apple Neural Engine, Qualcomm, Edge TPU, and ARM Ethos to model co

  2026-06-16 · 22 min read #edge-ai#npu#on-device#inference#quantization

• The 2026 AI Accelerator Landscape — From Blackwell to Vera Rubin

  A developer-oriented map of the 2026 AI accelerator market. We cover NVIDIA Blackwell and the next-generation Vera Rubin, AMD MI350X, the moment inference capex first overtook training, and how to choose a chip per workl

  2026-06-16 · 21 min read #ai-hardware#nvidia-blackwell#vera-rubin#inference#gpu

• Local LLM Inference Optimization — From Quantization to Breaking the VRAM Ceiling

  Privacy concerns, cost pressure, and big-tech fatigue are driving a local LLM revival. We map the entire landscape of local inference optimization: VRAM-first hardware thinking, GGUF and AWQ quantization, llama.cpp vs vL

  2026-06-12 · 16 min read #llm#inference#quantization#llama-cpp#vllm

• The Rise of Diffusion LMs — Can They Become an Alternative to Autoregression

  In June 2026 Google released DiffusionGemma, and text diffusion models are lighting up GeekNews and Hacker News. We critically analyze the structural limits of autoregressive generation, the principles of mask-based deno

  2026-06-12 · 17 min read #llm#diffusion#text-generation#gemma#inference

• LLM Serving & Local Inference in 2026 — vLLM / llama.cpp / MLX / Ollama / LM Studio / SGLang / TGI Deep Dive

  A map of the 2026 LLM serving and inference landscape. Datacenter camp (vLLM, SGLang, TGI, Triton, TensorRT-LLM), local camp (llama.cpp, MLX, llamafile, Ollama, LM Studio, GPT4All), emerging camp (KTransformers, MLC LLM,

  2026-05-16 · 25 min read #llm#model-serving#inference#vllm#llama-cpp

• AI Inference Engines 2026 - vLLM · SGLang · llama.cpp · TGI · TensorRT-LLM · MLX · mistral.rs · DeepSpeed-MII · Aphrodite Deep Dive

  In 2026, LLM engineering is no longer about which model — it is about which inference engine. We dissect vLLM V1, SGLang 0.4, TensorRT-LLM, TGI 3.x, llama.cpp, MLX-LM, mistral.rs, DeepSpeed-MII, Aphrodite, CTranslate2, E

  2026-05-16 · 20 min read #llm-inference#vllm#sglang#llama-cpp#tgi

• Edge AI Complete Guide 2025: On-Device Inference, Model Optimization, TensorRT/ONNX/CoreML

  Everything about Edge AI! On-device inference (TensorRT/ONNX Runtime/CoreML/TFLite), model optimization (quantization/pruning/knowledge distillation), hardware (NVIDIA Jetson/Apple Neural Engine/Qualcomm NPU), Federated

  2026-04-13 · 21 min read #edge-ai#on-device#inference#tensorrt#onnx

• AI Model Serving and Inference Optimization Complete Guide: vLLM, TensorRT, Triton, Ollama

  The complete guide to efficiently serving AI models in production. Master vLLM, TensorRT, NVIDIA Triton Inference Server, Ollama, quantization (INT8/INT4), batch processing, and latency optimization with real-world examp

  2026-03-17 · 19 min read #mlops#model-serving#vllm#tensorrt#triton

• LLM Inference Optimization Complete Guide: KV Cache, Speculative Decoding, Continuous Batching

  A complete guide to pushing LLM inference to its limits. Covers KV Cache, Speculative Decoding, Continuous Batching, PagedAttention, FlashInfer, multi-GPU inference, and DeepSeek MLA in depth.

  2026-03-17 · 25 min read #llm#inference#optimization#kv-cache#speculative-decoding

• BitNet 1-bit LLM Inference Framework: Running Large Language Models on CPU

  A guide to BitNet 1-bit LLM inference framework for running large language models on CPU hardware.

  2026-03-15 · 25 min read #llm#bitnet#1-bit-llm#inference#cpu-deployment

• LLM Inference Serving Framework Comparison: TensorRT-LLM vs vLLM vs SGLang Production Deployment Strategy

  A comparison of LLM inference serving frameworks — TensorRT-LLM, vLLM, and SGLang — with production deployment strategies.

  2026-03-12 · 21 min read #llm#inference#tensorrt-llm#vllm#sglang

• The Complete vLLM Guide — From PagedAttention to Production Optimization

  Covering everything about LLM inference optimization — from vLLM core PagedAttention mechanism to Continuous Batching, Tensor/Pipeline Parallelism, and Prefix Caching.

  2026-03-03 · 9 min read #llm#vllm#inference#paged-attention#model-serving

• Accelerating LLM Inference 2-3x with Speculative Decoding: From Theory to Production

  A deep dive into the mathematical foundations of Speculative Decoding, the Draft-Verify pipeline, acceptance probability analysis, practical implementation in vLLM/TensorRT-LLM, and Apple's Mirror Speculative Decoding.

  2026-03-02 · 8 min read #llm#speculative-decoding#inference#optimization#vllm

• LLM Inference Optimization: In-Depth Analysis of vLLM and TensorRT-LLM

  Analyzing core LLM inference optimization techniques including PagedAttention, Continuous Batching, and quantization, based on the official documentation of vLLM and TensorRT-LLM.

  2026-03-01 · 22 min read #llm#inference#vllm#tensorrt-llm#optimization