LabHub

Blog

LLM Serving & Local Inference in 2026 — vLLM / llama.cpp / MLX / Ollama / LM Studio / SGLang / TGI Deep Dive

2026-05-16 · 25 min read · #llm#model-serving#inference#vllm#llama-cpp#mlx#llamafile#ollama#lm-studio#gpt4all#sglang#tgi#ktransformers#tensorrt-llm#modular-max#quantization#deep-dive#english

한국어English日本語

Prologue — "Where you run the model" matters again

In 2023, everyone used the OpenAI API. In 2024, Anthropic, Google, and Mistral joined in, and the question was "which API." From 2025 onward, the question shifted back to "where do you run it." Model sizes shrank to usable tiers (8B, 30B, 70B). Quantization got good enough that Q4_K_M is genuinely usable. Laptops started shipping with 200GB of unified memory (M4 Max 128GB, M3 Ultra 192GB). Cloud token prices fluctuate by the week. As a result, whether you run the same model on an API, a dedicated server, or locally now determines your cost, latency, and governance posture.

The problem is there are too many choices. vLLM, SGLang, TGI, llama.cpp, MLX, llamafile, Ollama, LM Studio, GPT4All, KTransformers, MLC LLM, Triton, TensorRT-LLM, Modular MAX — each has a different texture, suits different workloads, and overlaps with others. On the cloud side, Together, Fireworks, Groq, Cerebras, SambaNova, and Lepton (acquired by NVIDIA in 2025) all pitch "just call the API and we'll run it well."

This post draws the map as of May 2026. Who does what well, which camp suits which workload, and how this all meets the Korean and Japanese model ecosystems.

Chapter 1 · The 2026 LLM serving map — three camps

Big picture first. The 2026 LLM serving and inference market splits into roughly three camps.

Camp A · Datacenter / high-throughput serving

Production serving for enterprises, SaaS, and research labs. Tens to thousands of RPS, multiple users, GPU clusters.

Camp B · Local / single-user

Laptops, workstations, home servers. Sub-1 RPS, single user, privacy-first.

Camp C · Cloud serving SaaS

"Pick a model, call an API" — no GPU ops burden.

Each camp was built under different assumptions. Datacenter: "GPUs are expensive, squeeze utilization." Local: "make it run on my laptop." Cloud SaaS: "forget GPU ops, just watch the token unit price." So even when they look like they're solving the same problem, who solves it well differs.

Chapter 2 · vLLM — the PagedAttention standard

vLLM started in 2023 at UC Berkeley (Sky Computing Lab) as an OSS inference server. It became the vLLM Project as an independent OSS organization in 2024, and in 2025 joined the PyTorch Foundation under the Linux Foundation. By 2026 it is the de facto standard for OSS LLM serving.

The core invention is PagedAttention. By managing the KV cache in block units like an OS page table, vLLM eliminates memory fragmentation and lets many concurrent requests' KV caches share the same GPU efficiently. The result is 2–4x throughput over naive batching as a typical baseline.

Where vLLM stands in 2026:

A typical vLLM serve:

pip install vllm

vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --tensor-parallel-size 4 \
  --max-model-len 32768 \
  --enable-prefix-caching \
  --enable-chunked-prefill

This launches an OpenAI-compatible HTTP server on port 8000, and clients use the OpenAI SDK as-is.

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
    model="meta-llama/Llama-3.3-70B-Instruct",
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

When vLLM: multi-user production serving, OSS models, NVIDIA (or AMD ROCm, Intel Gaudi, TPU) GPUs available, throughput and cost efficiency as priorities. For a single-user laptop, it is overkill.

Chapter 3 · llama.cpp — gguf, CPU/GPU dual-wield

llama.cpp is the C/C++ inference engine Georgi Gerganov started in 2023. Built on the ambition of "run an LLM with one binary and no external dependencies," by 2026 it is the base layer of local and embedded LLM inference. Almost every user-friendly local tool — Ollama, LM Studio, llamafile, GPT4All — uses llama.cpp internally.

Core texture:

You can spin up an OpenAI-compatible server with llama-server.

./llama-server \
  --model models/Llama-3.3-70B-Q4_K_M.gguf \
  --n-gpu-layers 99 \
  --ctx-size 8192 \
  --host 0.0.0.0 --port 8080

Strengths and weaknesses in 2026:

When llama.cpp: personal or small-user local, embedded/edge, leveraging GGUF quantization, "I want it to end in one binary."

Chapter 4 · MLX (Apple) — Apple Silicon inference

MLX is the OSS array and ML framework Apple ML Research published in December 2023. It has an API similar to NumPy plus PyTorch, and treats Apple Silicon's unified memory as a first-class citizen. On M1/M2/M3/M4 chips, CPU and GPU share the same memory, and MLX flows computation across it without data copies.

MLX in 2026:

Typical MLX use:

pip install mlx mlx-lm

mlx_lm.generate \
  --model mlx-community/Llama-3.3-70B-Instruct-4bit \
  --prompt "Why run LLMs on Apple Silicon" \
  --max-tokens 200

Directly from Python:

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/Llama-3.3-70B-Instruct-4bit")
text = generate(model, tokenizer, prompt="Hello", max_tokens=200)
print(text)

Why MLX matters more and more:

Weakness: Apple Silicon only. Multi-user serving is not as polished as vLLM.

When MLX: local LLM development, fine-tuning, prototyping on Mac, workloads tied to M-series laptops or desktops.

Chapter 5 · llamafile (Mozilla) — single binary

llamafile is the project Mozilla (Mozilla Innovation Group) published in November 2023. The core idea: bundle model weights + inference engine + tokenizer into a single executable that runs by double-click on any OS and CPU architecture.

The technical trick is Actually Portable Executable (APE), a format authored by Justine Tunney (same person behind Cosmopolitan libc). One file runs as ELF on Linux, Mach-O on macOS, and PE on Windows simultaneously — and FreeBSD/OpenBSD/NetBSD on top. The llama.cpp inference engine and GGUF weights are bundled in.

Typical use:

curl -L -o llava-v1.5.llamafile \
  https://huggingface.co/Mozilla/llava-v1.5-7B-llamafile/resolve/main/llava-v1.5-7b-q4.llamafile
chmod +x llava-v1.5.llamafile
./llava-v1.5.llamafile

This boots an OpenAI-compatible HTTP server, and the same file can be copied to Mac/Linux/Windows and run as-is. In 2026, Mozilla continues publishing llamafile variants of major OSS models (LLaMA, Mistral, Phi, Gemma, Qwen) on Hugging Face.

Why llamafile matters:

Weakness: single-file size limits (previously 4GB, worked around with ZipAlign and external weights since 2024); production serving isn't its texture.

When llamafile: handing an LLM to non-technical users, offline distribution, classrooms and workshops, archival.

Chapter 6 · Ollama — the easiest local LLM

Ollama is a local LLM runtime that started in June 2023. Internally it uses llama.cpp, but layers on Docker-clean UX — pull, run, push for models, and a Modelfile for customization — and in 2026 has become the default entry point for local LLMs.

# Install (Mac/Linux/Windows)
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model and chat immediately
ollama run llama3.3

# Backend server mode
ollama serve

# Call the API from another terminal
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.3",
  "prompt": "Hello",
  "stream": false
}'

Where Ollama stands in 2026:

The draw is simplicity. "Install on Mac and chat with a quantized 70B in five minutes" genuinely works. As a result, almost every OSS LLM tool — LangChain, LlamaIndex, n8n, VS Code extensions — supports "Ollama-compatible" out of the box.

Weakness: multi-user throughput and advanced batching trail vLLM/SGLang. The Modelfile DSL is thin. The model library trends toward Ollama Hub, though arbitrary GGUF import is possible.

When Ollama: personal laptop, home server, "start in five minutes," instant connection to OSS tools.

Chapter 7 · LM Studio / GPT4All — GUI options

For users who prefer GUI to CLI.

LM Studio

A desktop app (Mac/Windows/Linux) that started in 2023. Model search, download, chat, and server mode all in one .app. You can search Hugging Face Hub for GGUF models, filter and download them, and either chat in the UI or launch an OpenAI-compatible local server.

LM Studio in 2026:

After its license change made commercial use free (personal and corporate alike), it's often used for in-house demos and POCs.

GPT4All

An OSS project Nomic AI started in early 2023. Desktop app + Python SDK + model library. With "an LLM that runs on any laptop" as its motto, it's strong on CPU inference and small models (3B, 7B, 13B). Features like LocalDocs (local document retrieval) put RAG directly into the desktop app.

GPT4All in 2026:

LM Studio vs GPT4All in one line: LM Studio is "GUI strong on exploring Hugging Face models," GPT4All is "GUI strong on desktop chat plus LocalDocs RAG."

When GUI: non-technical users, demos, education, internal tools, "I want to finish everything with a mouse."

Chapter 8 · SGLang — structured generation

SGLang is an inference system from a joint LMSYS, UCB, and Stanford collaboration, published in early 2024. It belongs to the same "high-throughput serving" camp as vLLM, but enters with two distinguishing cards — structured generation and RadixAttention.

RadixAttention

If PagedAttention solved "memory fragmentation," RadixAttention shares prefixes across requests in a radix tree. With 1000 requests sharing the same system prompt, the KV cache for that prefix is computed once and shared automatically. The effect is large on agentic workloads (repeated tool specs and few-shots).

Frontend DSL — structured generation

SGLang writes LLM calls in a Python DSL where branching, looping, parallelism, and constraints are first-class.

import sglang as sgl

@sgl.function
def multi_turn_question(s, question):
    s += sgl.system("You are a helpful assistant.")
    s += sgl.user(question)
    s += sgl.assistant(sgl.gen("answer", max_tokens=256))
    s += sgl.user("Was that answer correct?")
    s += sgl.assistant(sgl.gen("verification", choices=["yes", "no"]))

state = multi_turn_question.run(question="What is the capital of France?")
print(state["answer"], state["verification"])

sgl.gen(..., choices=...) and JSON-schema constraints are enforced by constrained decoding in the backend. That is, the model can't emit anything other than "yes"/"no."

SGLang strengths in 2026:

When SGLang: structured output and tool use heavy workloads, agentic serving with frequent prefix sharing, evaluating alongside vLLM with your own workload as the benchmark.

Chapter 9 · TGI (Hugging Face) — ops-friendly

TGI (Text Generation Inference) is the inference server Hugging Face built. Started in late 2022, by 2026 it draws less academic spotlight than vLLM/SGLang but has settled in as the ops-friendly choice.

docker run --gpus all -p 8080:80 \
  -v $PWD/models:/data \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3.3-70B-Instruct \
  --quantize bitsandbytes-nf4

The license briefly went to HFOIL in 2024 before returning to Apache 2.0. As of 2026 it's free OSS again.

When TGI: teams already heavy on Hugging Face Hub, mode parity with Hugging Face Endpoints, preference for "ops metrics and standard docker container" texture.

vLLM/SGLang vs TGI in one line: academic accel innovation lands first on vLLM/SGLang; TGI's strength is "operationalization."

Chapter 10 · KTransformers (Tsinghua) — long-context optimized

KTransformers is an inference framework published in 2024 by the MADSys lab at Tsinghua University. Its focus is clear — run a giant MoE model on one consumer GPU.

Key tricks:

# DeepSeek-V3 on RTX 4090 plus large system RAM
git clone https://github.com/kvcache-ai/ktransformers
cd ktransformers
pip install -e .
python -m ktransformers.local_chat \
  --model_path /path/to/DeepSeek-V3-GGUF \
  --gguf_path /path/to/DeepSeek-V3-GGUF \
  --cpu_infer 32

Why KTransformers matters:

Weakness: multi-user serving and throughput are still better on vLLM. There's an install and tuning learning curve.

When KTransformers: you want to run a giant MoE model as a single user without access to a GPU cluster; long-context workloads.

Chapter 11 · NVIDIA Triton + TensorRT-LLM — the datacenter standard

NVIDIA-camp full-stack. The two are often used as a pair.

TensorRT-LLM

NVIDIA's OSS LLM inference library. It compiles PyTorch models into TensorRT engines (NVIDIA-only IR) for acceleration in NVIDIA-pushed precisions like FP16, INT8, INT4, FP8, and NVFP4. Its strength is quickly exploiting new instructions on Hopper (H100/H200) and Blackwell (B100/B200, GB200).

Triton Inference Server

NVIDIA's general inference server. Beyond LLMs, it serves vision, tabular, and custom models multi-framework in one server. You can use the TensorRT-LLM backend for LLM mode, or directly run TensorRT-LLM's OpenAI-compatible server (trtllm-serve).

# Build the model into a TensorRT-LLM engine
trtllm-build --checkpoint_dir ./llama3-70b \
  --output_dir ./engines/llama3-70b \
  --gemm_plugin auto \
  --max_batch_size 32

# Serve with trtllm-serve (OpenAI compatible)
trtllm-serve ./engines/llama3-70b --port 8000

vLLM vs TensorRT-LLM in one line: vLLM is OSS, spans many GPUs (NVIDIA, AMD, Intel, TPU), and is easier to approach. TensorRT-LLM is NVIDIA-only but squeezes the last drop out of NVIDIA GPUs (especially FP8/FP4). Many production teams keep both and benchmark with their own workload.

When TensorRT-LLM/Triton: NVIDIA Hopper/Blackwell GPU clusters, absolute throughput and latency in production, an ops team familiar with the NVIDIA ecosystem.

Chapter 12 · Modular MAX (Mojo team) — the new entry

Modular is the company Chris Lattner (LLVM, Swift, Clang) founded in 2022. They build Mojo (a high-performance systems language aiming for Python compatibility) and MAX (Modular Accelerated Xecution, an AI inference platform). The OSS transition for MAX started in 2024, and by 2026 the pitch is one graph compiler for inference across NVIDIA, AMD, Intel, and Apple.

Core idea:

pip install modular
max serve --model-path meta-llama/Llama-3.3-70B-Instruct

Modular in 2026:

Weakness: Mojo's OSS transition is phased, so the community is still small. Model coverage and feature breadth are wider on vLLM.

When MAX: you want a faster path than vLLM's ROCm backend on AMD GPUs; you want to be an early adopter of the Mojo ecosystem; you want to avoid NVIDIA-only lock-in.

Chapter 13 · Quantization — GGUF / AWQ / GPTQ / FP8

You can't pick a serving framework without also picking a quantization format. Five mainstream choices in 2026.

GGUF (llama.cpp family)

AWQ (Activation-aware Weight Quantization)

GPTQ

FP8

NVFP4 / MXFP4

One-line recommendations:

Chapter 14 · Cloud serving SaaS — Together / Fireworks / Groq / Cerebras / Lepton

The camp for teams who don't want to operate GPUs themselves. The texture is "call OSS models with one API."

Together AI

The big sibling of OSS model hosting. 200+ models like Llama, Mistral, Qwen, DeepSeek via OpenAI-compatible APIs. Fine-tune, dedicated endpoints, dedicated deployments. Academic collaboration (joint training of models like RedPajama and Stripedhyena) is another strength.

Fireworks AI

Evaluated for fast throughput and quality function calling and structured output. Strong self-hosted training and fine-tune tooling. They also sell an inference stack optimized for Mixture-of-Experts and long-context.

Groq

The LPU (Language Processing Unit) custom chip. Massive SRAM and deterministic execution serve the same model at overwhelmingly low latency (hundreds of tokens per second). OSS Llama and Mixtral focus; model breadth is limited but on speed they're almost always a top contender.

Cerebras

The wafer-scale engine — one wafer is one chip (WSE-3 has ~4 trillion transistors). A giant single chip means less model partitioning and communication overhead than an H100 cluster. They advertise fast inference throughput in their cloud, and some are used for huge training jobs by governments and research labs.

SambaNova

The custom RDU (Reconfigurable Dataflow Unit) chip. Dataflow architecture optimized for dense and sparse LLM workloads. Enterprise and government markets primarily.

Lepton AI (NVIDIA acquisition, 2025)

A cloud inference platform founded by Jia Yangqing (original Caffe author, Meta, Alibaba) and others. NVIDIA acquired it in 2025, and it now sits as the inference layer of NVIDIA's cloud ecosystem. Integrated with NVIDIA DGX Cloud and NIM (NVIDIA Inference Microservices), it's settling in as "OSS model inference run by NVIDIA directly."

One-line comparison

Chapter 15 · Korea / Japan — Upstage Solar / KT Mi:dm / Sakana / NTT Tsuzumi / ELYZA

You miss half the story if you only watch English models. The Korean and Japanese OSS and semi-OSS camps cemented their position in 2026.

Korea

These models often run on vLLM, SGLang, llama.cpp forks and branches with patches for Korean tokenizers and specific kernels. OSS compatibility varies per model, as do licenses (research-only, restricted commercial, fully OSS) — license review before adoption is mandatory.

Japan

Operationally, Korean and Japanese models typically ship both local serving (llama.cpp, Ollama, vLLM) and OSS hosting (Together, Fireworks, plus local clouds) compatible GGUF and safetensors variants. For regulated industries like government and banking, on-prem vLLM/TGI is essentially the default choice.

Chapter 16 · Who should pick what

Wrapping the 12 camps into situational recommendations.

Individual / hobbyist

Startup (10–50 people)

Enterprise (regulated industries, finance, government)

Datacenter / hyperscale

Avoiding hardware lock-in

Epilogue — Models became commoditized; infra is the differentiator

In 2023, "which model" was the question. In 2026, "where and how do you run it" is the question. Models became plentiful (OSS 70B, even giant MoE), prices change by the hour, quantization quality has reached a level different from a year ago, and one GPU now plugs into a laptop.

The conclusion of this post is simple — benchmark against your own workload. Throughput, latency, cost, and operational complexity all come out different per workload, so measurement always beats recommendation. And the real message of this post might be that all 12 camps for that measurement are OSS in 2026.

What we have to do now isn't make GPUs run faster — it's decide which camp fits our problem within a week.

References

Comments

No comments yet.

Sign in to leave a comment