LLM Quantization Practical Guide: GPTQ, AWQ, GGUF Format Comparison and Precision-Performance Trade-offs

• 1. Introduction: Why Quantize
• 2. Quantization Fundamentals
  • 2.1 Numeric Representations and Precision
  • 2.2 The Math Behind Quantization
  • 2.3 PTQ vs QAT
  • 2.4 Group Quantization
• 3. GPTQ: The Early Breakthrough
  • 3.1 How the GPTQ Algorithm Works
  • 3.2 Quantizing with AutoGPTQ
  • 3.3 Inference with a GPTQ Model
• 4. AWQ: Activation-Aware Quantization
  • 4.1 How the AWQ Algorithm Works
  • 4.2 AWQ vs GPTQ: The Key Differences
  • 4.3 Quantizing with AutoAWQ
  • 4.4 AWQ GEMM vs GEMV Kernels
  • 4.5 Advanced AWQ Configuration
• 5. GGUF: The Edge Device Standard
  • 5.1 GGUF Format Overview
  • 5.2 Comparing GGUF Quantization Types
  • 5.3 Converting to GGUF with llama.cpp
  • 5.4 Inference with a GGUF Model
• 6. bitsandbytes NF4: The Heart of QLoRA
  • 6.1 NormalFloat4 (NF4) Theory
  • 6.2 Loading in 4-bit with bitsandbytes
  • 6.3 bitsandbytes + QLoRA Fine-Tuning
• 7. Overall Comparison and Benchmarks
  • 7.1 Memory Footprint Comparison
  • 7.2 Perplexity Comparison (WikiText-2)
  • 7.3 Inference Speed Comparison (tokens/sec, A100 80GB)
  • 7.4 Calculating Memory Savings
• 8. Serving Quantized Models on vLLM
  • 8.1 Serving an AWQ Model
  • 8.2 Serving a GPTQ Model
  • 8.3 Performance Tuning for Quantized Models on vLLM
• 9. Choosing a Quantization Format
  • 9.1 Decision Flow
  • 9.2 Recommendations by Scenario
• 10. Troubleshooting
  • 10.1 Common Problems and Fixes
  • 10.2 Performance Optimization Checklist
• 11. A Practical Benchmark Script
• 12. Recent Trends and Outlook
  • 12.1 Quantization Technology Trends in 2026
  • 12.2 FP8 Quantization (the Next Standard)
• 13. References

1. Introduction: Why Quantize

The biggest bottleneck when deploying an LLM to production is GPU memory and inference cost. A Llama 3 70B model needs roughly 140GB of GPU memory at FP16, which makes 2 A100 80GB GPUs the minimum requirement. Quantization is the technique of lowering the precision of model weights to cut memory usage and compute sharply.

As of 2025-2026, quantization has moved past being a simple cost-cutting measure and settled in as a requirement for production deployment. AWQ has proven its edge over GPTQ in both accuracy and speed, and llama.cpp's GGUF format has become the de facto standard for edge device deployment.

This article compares the major techniques, from the basic theory of quantization through GPTQ, AWQ, GGUF, and bitsandbytes NF4, and lays out an optimal quantization strategy backed by working code and benchmarks.

2. Quantization Fundamentals

2.1 Numeric Representations and Precision

Understanding the kinds of numeric representation deep learning models use, and how they behave, is the starting point for quantization.

2.2 The Math Behind Quantization

Quantization is the process of mapping continuous floating-point values onto discrete integer values.

import numpy as np

def symmetric_quantize(weights, n_bits=8):
    """Symmetric quantization: the zero point is 0"""
    qmax = 2**(n_bits - 1) - 1
    qmin = -2**(n_bits - 1)

    # Compute the scale
    abs_max = np.max(np.abs(weights))
    scale = abs_max / qmax

    # Quantize
    quantized = np.clip(np.round(weights / scale), qmin, qmax).astype(np.int8)

    return quantized, scale

def asymmetric_quantize(weights, n_bits=8):
    """Asymmetric quantization: uses a zero point"""
    qmax = 2**n_bits - 1
    qmin = 0

    w_min = np.min(weights)
    w_max = np.max(weights)

    scale = (w_max - w_min) / (qmax - qmin)
    zero_point = int(np.round(-w_min / scale))

    quantized = np.clip(
        np.round(weights / scale) + zero_point, qmin, qmax
    ).astype(np.uint8)

    return quantized, scale, zero_point

def dequantize(quantized, scale, zero_point=0):
    """Dequantization: integers back to floating point"""
    return scale * (quantized.astype(np.float32) - zero_point)

# Example
weights = np.random.randn(1000).astype(np.float32)
q_sym, scale_sym = symmetric_quantize(weights, n_bits=4)
q_asym, scale_asym, zp = asymmetric_quantize(weights, n_bits=4)

print(f"Original memory: {weights.nbytes} bytes")
print(f"INT4 memory: {q_sym.nbytes // 2} bytes (theoretical)")
print(f"Compression ratio: {weights.nbytes / (q_sym.nbytes // 2):.1f}x")

2.3 PTQ vs QAT

Quantization techniques split broadly into two families.

Post-Training Quantization (PTQ)

• Applies quantization to an already-trained model
• Needs no extra training, so it is fast and simple
• GPTQ, AWQ, and GGUF all take this approach
• Only calibration data is required (typically 128 to 512 samples)

Quantization-Aware Training (QAT)

• Simulates the effect of quantization during training
• The model adapts to quantization noise, giving higher accuracy
• Costs a full training run
• Mostly used on smaller models

# PTQ vs QAT comparison pipeline
class QuantizationPipeline:
    """Quantization pipeline comparison"""

    @staticmethod
    def ptq_pipeline(model, calibration_data, method="awq"):
        """PTQ pipeline"""
        # 1. Collect statistics from the calibration data
        # 2. Determine the quantization parameters (scale, zero_point)
        # 3. Quantize the weights
        # 4. Save the quantized model
        steps = [
            "Load pretrained model",
            "Run calibration (128 samples)",
            "Compute quantization parameters",
            "Quantize weights",
            "Save quantized model",
        ]
        return steps

    @staticmethod
    def qat_pipeline(model, training_data):
        """QAT pipeline"""
        # 1. Insert fake quantization nodes
        # 2. Retrain fully or partially
        # 3. Apply the real quantization
        # 4. Save the quantized model
        steps = [
            "Insert fake quantization nodes",
            "Fine-tune with quantization simulation",
            "Convert to actual quantized model",
            "Evaluate and save",
        ]
        return steps

2.4 Group Quantization

Group quantization sits at the heart of modern quantization techniques. Instead of one scale/zero_point for the whole tensor, it keeps a separate set of quantization parameters for each fixed-size group.

def group_quantize(weights, n_bits=4, group_size=128):
    """Group-wise quantization"""
    qmax = 2**(n_bits - 1) - 1
    qmin = -2**(n_bits - 1)

    # Split into groups
    num_groups = weights.shape[-1] // group_size
    w_grouped = weights.reshape(-1, num_groups, group_size)

    scales = []
    zeros = []
    quantized_groups = []

    for i in range(num_groups):
        group = w_grouped[:, i, :]
        abs_max = np.max(np.abs(group), axis=-1, keepdims=True)
        scale = abs_max / qmax

        q = np.clip(np.round(group / (scale + 1e-10)), qmin, qmax)
        quantized_groups.append(q)
        scales.append(scale)

    return quantized_groups, scales

# A smaller group_size gives better precision but more overhead
# group_size=128 is generally the best balance

3. GPTQ: The Early Breakthrough

3.1 How the GPTQ Algorithm Works

GPTQ (Generative Pre-trained Transformer Quantization) is a PTQ algorithm proposed by Frantar et al. in 2022, an improvement on Optimal Brain Quantization (OBQ). The core idea is to use Hessian information to minimize quantization error.

The main characteristics of GPTQ are as follows.

• Quantizes sequentially, column by column
• Compensates for one column's quantization error through the weight updates of later columns
• Efficient Hessian inverse computation using Cholesky decomposition
• 128 calibration samples are enough

3.2 Quantizing with AutoGPTQ

from transformers import AutoTokenizer
from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig
import torch

# 1. Load the model and tokenizer
model_name = "meta-llama/Llama-3.1-8B"
tokenizer = AutoTokenizer.from_pretrained(model_name)

# 2. Quantization settings
quantize_config = BaseQuantizeConfig(
    bits=4,                  # 4-bit quantization
    group_size=128,          # Group size
    desc_act=True,           # Activation order (improves accuracy)
    damp_percent=0.01,       # Damping ratio
    sym=True,                # Symmetric quantization
    true_sequential=True,    # Sequential quantization (improves accuracy)
    model_seqlen=4096,       # Sequence length
)

# 3. Load the model
model = AutoGPTQForCausalLM.from_pretrained(
    model_name,
    quantize_config=quantize_config,
    torch_dtype=torch.float16,
)

# 4. Prepare the calibration data
from datasets import load_dataset

dataset = load_dataset("wikitext", "wikitext-2-raw-v1", split="train")
calibration_texts = [text for text in dataset["text"] if len(text) > 100][:128]

calibration_data = []
for text in calibration_texts:
    tokens = tokenizer(
        text,
        return_tensors="pt",
        max_length=2048,
        truncation=True,
        padding=False,
    )
    calibration_data.append(tokens.input_ids)

# 5. Run quantization (roughly 15-30 min for a 7B model)
model.quantize(calibration_data)

# 6. Save the quantized model
output_dir = "./llama3-8b-gptq-4bit"
model.save_quantized(output_dir)
tokenizer.save_pretrained(output_dir)

print(f"Quantization complete! Saved to: {output_dir}")

3.3 Inference with a GPTQ Model

from auto_gptq import AutoGPTQForCausalLM
from transformers import AutoTokenizer

# Load the quantized model
model = AutoGPTQForCausalLM.from_quantized(
    "./llama3-8b-gptq-4bit",
    device_map="auto",
    use_safetensors=True,
    inject_fused_attention=True,   # Use fused attention
    inject_fused_mlp=True,         # Use fused MLP
    use_triton=False,              # Use the CUDA kernel
    disable_exllama=False,         # Enable the ExLlama v2 kernel
    exllama_config={"version": 2}, # Use ExLlama v2
)

tokenizer = AutoTokenizer.from_pretrained("./llama3-8b-gptq-4bit")

# Inference
prompt = "Explain quantum computing in simple terms:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=256,
        temperature=0.7,
        do_sample=True,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

4. AWQ: Activation-Aware Quantization

4.1 How the AWQ Algorithm Works

AWQ (Activation-aware Weight Quantization) is a quantization technique proposed by MIT Han Lab in 2023 that works around the limits of GPTQ. The key insight is that "not all weights are equally important".

The core principles of AWQ are as follows.

1. Activation awareness: weights in channels whose input activations are large matter more
2. Per-channel scaling: scaling up the weights of important channels reduces quantization error
3. Equivalent transformation: the scaling factor is absorbed into the next layer, so there is no added compute cost
4. Weight-only quantization: activations stay in FP16

4.2 AWQ vs GPTQ: The Key Differences

4.3 Quantizing with AutoAWQ

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

# 1. Load the model
model_name = "meta-llama/Llama-3.1-8B"
model = AutoAWQForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

# 2. Quantization settings
quant_config = {
    "zero_point": True,        # Asymmetric quantization (improves accuracy)
    "q_group_size": 128,       # Group size
    "w_bit": 4,                # 4-bit quantization
    "version": "GEMM",        # Use the GEMM kernel (best for batched inference)
}

# 3. Run quantization (roughly 5-10 min for a 7B model)
model.quantize(tokenizer, quant_config=quant_config)

# 4. Save
output_dir = "./llama3-8b-awq-4bit"
model.save_quantized(output_dir)
tokenizer.save_pretrained(output_dir)

print(f"AWQ quantization complete! Saved to: {output_dir}")

4.4 AWQ GEMM vs GEMV Kernels

AWQ ships two kernel implementations.

# GEMM kernel: optimized for batched inference
quant_config_gemm = {
    "zero_point": True,
    "q_group_size": 128,
    "w_bit": 4,
    "version": "GEMM",   # Best when batch size > 1
}

# GEMV kernel: optimized for single-request inference
quant_config_gemv = {
    "zero_point": True,
    "q_group_size": 128,
    "w_bit": 4,
    "version": "GEMV",   # Best when batch size = 1
}

# Selection guide:
# - Serving (many users at once) -> GEMM
# - Local inference (single user) -> GEMV
# - vLLM integration -> GEMM (required)

4.5 Advanced AWQ Configuration

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer

model_name = "meta-llama/Llama-3.1-70B"

# Quantization settings for a large model
model = AutoAWQForCausalLM.from_pretrained(
    model_name,
    safetensors=True,
    device_map="auto",     # Spread automatically across GPUs
)

tokenizer = AutoTokenizer.from_pretrained(model_name)

# Optimal settings for a 70B model
quant_config = {
    "zero_point": True,
    "q_group_size": 128,
    "w_bit": 4,
    "version": "GEMM",
}

# Use custom calibration data
from datasets import load_dataset

# Domain-specific calibration data
calib_data = load_dataset(
    "HuggingFaceH4/ultrachat_200k",
    split="train_sft",
)

# Run quantization (roughly 1-2 hours for a 70B model, A100 80GB x2)
model.quantize(
    tokenizer,
    quant_config=quant_config,
    calib_data=calib_data,
    n_samples=128,
    seqlen=2048,
)

model.save_quantized("./llama3-70b-awq-4bit")
tokenizer.save_pretrained("./llama3-70b-awq-4bit")

5. GGUF: The Edge Device Standard

5.1 GGUF Format Overview

GGUF (GPT-Generated Unified Format) is a model format developed by the llama.cpp project, optimized for inference on CPUs and edge devices. It replaces the earlier GGML format and has the following characteristics.

• Model weights and metadata live in a single file
• Supports a range of quantization levels (Q2_K to Q8_0)
• Supports many backends: CPU, Metal (Apple), CUDA, Vulkan, and more
• Fast loading based on memory mapping (mmap)
• Used by llama.cpp, ollama, LM Studio, and others

5.2 Comparing GGUF Quantization Types

5.3 Converting to GGUF with llama.cpp

# 1. Build llama.cpp
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j$(nproc) LLAMA_CUDA=1  # Build with CUDA support

# 2. Convert a HuggingFace model to GGUF
python convert_hf_to_gguf.py \
    ../meta-llama/Llama-3.1-8B \
    --outtype f16 \
    --outfile llama3-8b-f16.gguf

# 3. Run the quantization
./llama-quantize \
    llama3-8b-f16.gguf \
    llama3-8b-Q4_K_M.gguf \
    Q4_K_M

# 4. Convert at several quantization levels
for quant in Q3_K_M Q4_K_S Q4_K_M Q5_K_M Q6_K Q8_0; do
    ./llama-quantize \
        llama3-8b-f16.gguf \
        "llama3-8b-${quant}.gguf" \
        "$quant"
    echo "Completed: $quant"
done

# 5. High-quality quantization with an importance matrix
./llama-imatrix \
    -m llama3-8b-f16.gguf \
    -f calibration_data.txt \
    --output-frequency 10 \
    -o imatrix.dat

./llama-quantize \
    --imatrix imatrix.dat \
    llama3-8b-f16.gguf \
    llama3-8b-IQ4_XS.gguf \
    IQ4_XS

5.4 Inference with a GGUF Model

# Inference through the llama.cpp CLI
./llama-cli \
    -m llama3-8b-Q4_K_M.gguf \
    -p "Explain the theory of relativity:" \
    -n 256 \
    --temp 0.7 \
    --top-p 0.9 \
    -ngl 35    # Number of layers to offload to the GPU

# Run in server mode
./llama-server \
    -m llama3-8b-Q4_K_M.gguf \
    --host 0.0.0.0 \
    --port 8080 \
    -ngl 35 \
    --ctx-size 4096 \
    --parallel 4   # Number of concurrent requests

# Using llama-cpp-python from Python
from llama_cpp import Llama

# Load the model
llm = Llama(
    model_path="./llama3-8b-Q4_K_M.gguf",
    n_gpu_layers=35,     # GPU offload
    n_ctx=4096,          # Context size
    n_batch=512,         # Batch size
    verbose=False,
)

# Inference
output = llm(
    "Explain quantum computing in simple terms:",
    max_tokens=256,
    temperature=0.7,
    top_p=0.9,
    stop=["###", "\n\n\n"],
)

print(output["choices"][0]["text"])

# Using the OpenAI-compatible API
output = llm.create_chat_completion(
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is quantum computing?"},
    ],
    max_tokens=256,
    temperature=0.7,
)

print(output["choices"][0]["message"]["content"])

6. bitsandbytes NF4: The Heart of QLoRA

6.1 NormalFloat4 (NF4) Theory

NF4 (NormalFloat4) in bitsandbytes is a data type introduced in the QLoRA paper: 4-bit quantization optimized for weights that follow a normal distribution.

Its key characteristics are as follows.

• Picks the optimal quantization levels under the assumption that the weight distribution is normal
• Information-theoretically optimal compared with plain INT4
• Double quantization also saves memory on the quantization constants
• Used mainly for fine-tuning (QLoRA) rather than for inference

6.2 Loading in 4-bit with bitsandbytes

import torch
from transformers import (
    AutoModelForCausalLM,
    AutoTokenizer,
    BitsAndBytesConfig,
)

# NF4 quantization settings
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,                    # Load in 4-bit
    bnb_4bit_quant_type="nf4",            # NF4 type
    bnb_4bit_compute_dtype=torch.bfloat16, # Compute in BF16
    bnb_4bit_use_double_quant=True,        # Double quantization
    bnb_4bit_quant_storage=torch.uint8,    # Storage type
)

# Load the model
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    quantization_config=bnb_config,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")

# Check memory usage
print(f"Model memory: {model.get_memory_footprint() / 1024**3:.2f} GB")

# INT8 quantization is also available
bnb_config_8bit = BitsAndBytesConfig(
    load_in_8bit=True,
    llm_int8_threshold=6.0,     # Outlier threshold
    llm_int8_has_fp16_weight=False,
)

6.3 bitsandbytes + QLoRA Fine-Tuning

from peft import LoraConfig, get_peft_model, prepare_model_for_kbit_training
from transformers import TrainingArguments, Trainer

# Prepare the 4-bit model for fine-tuning
model = prepare_model_for_kbit_training(model)

# LoRA settings
lora_config = LoraConfig(
    r=16,                          # LoRA rank
    lora_alpha=32,                 # Scaling factor
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj",
    ],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
)

model = get_peft_model(model, lora_config)

# Check the trainable parameters
trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
total_params = sum(p.numel() for p in model.parameters())
print(f"Trainable: {trainable_params:,} / Total: {total_params:,}")
print(f"Ratio: {100 * trainable_params / total_params:.2f}%")

7. Overall Comparison and Benchmarks

7.1 Memory Footprint Comparison

7.2 Perplexity Comparison (WikiText-2)

7.3 Inference Speed Comparison (tokens/sec, A100 80GB)

7.4 Calculating Memory Savings

def calculate_memory_savings(
    model_params_billions: float,
    original_bits: int = 16,
    target_bits: int = 4,
    group_size: int = 128,
):
    """Quantization memory savings calculator"""
    # Original memory
    original_bytes = model_params_billions * 1e9 * (original_bits / 8)
    original_gb = original_bytes / (1024**3)

    # Weight memory after quantization
    quantized_bytes = model_params_billions * 1e9 * (target_bits / 8)

    # Quantization metadata (scale + zero_point per group)
    num_groups = model_params_billions * 1e9 / group_size
    # Per group: scale(FP16=2bytes) + zero_point(FP16=2bytes)
    metadata_bytes = num_groups * 4

    total_quantized_bytes = quantized_bytes + metadata_bytes
    quantized_gb = total_quantized_bytes / (1024**3)

    # KV cache memory (FP16, separate)
    # Rough estimate: 2 * num_layers * 2 * hidden_dim * seq_len * 2bytes
    # This part is unrelated to quantization

    savings_pct = (1 - quantized_gb / original_gb) * 100

    return {
        "original_gb": round(original_gb, 2),
        "quantized_gb": round(quantized_gb, 2),
        "savings_gb": round(original_gb - quantized_gb, 2),
        "savings_pct": round(savings_pct, 1),
        "compression_ratio": round(original_gb / quantized_gb, 2),
    }

# Savings by model size
for params in [7, 13, 34, 70]:
    result = calculate_memory_savings(params)
    print(f"\n{params}B model:")
    print(f"  FP16: {result['original_gb']} GB")
    print(f"  INT4: {result['quantized_gb']} GB")
    print(f"  Savings: {result['savings_gb']} GB ({result['savings_pct']}%)")
    print(f"  Compression: {result['compression_ratio']}x")

8. Serving Quantized Models on vLLM

8.1 Serving an AWQ Model

from vllm import LLM, SamplingParams

# Load the AWQ model (auto-detected)
llm = LLM(
    model="./llama3-8b-awq-4bit",
    quantization="awq",
    dtype="half",
    gpu_memory_utilization=0.90,
    max_model_len=8192,
    tensor_parallel_size=1,
)

# Serving
sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=256,
)

prompts = [
    "What is machine learning?",
    "Explain neural networks:",
    "How does gradient descent work?",
]

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    prompt = output.prompt
    generated = output.outputs[0].text
    print(f"Prompt: {prompt}")
    print(f"Output: {generated}\n")

8.2 Serving a GPTQ Model

from vllm import LLM, SamplingParams

# Serving a GPTQ model
llm = LLM(
    model="./llama3-8b-gptq-4bit",
    quantization="gptq",
    dtype="half",
    gpu_memory_utilization=0.90,
    max_model_len=8192,
)

# Run as a vLLM OpenAI-compatible server
# python -m vllm.entrypoints.openai.api_server \
#     --model ./llama3-8b-gptq-4bit \
#     --quantization gptq \
#     --dtype half \
#     --port 8000 \
#     --gpu-memory-utilization 0.9

8.3 Performance Tuning for Quantized Models on vLLM

# Run the vLLM server (production settings)
python -m vllm.entrypoints.openai.api_server \
    --model ./llama3-70b-awq-4bit \
    --quantization awq \
    --dtype half \
    --tensor-parallel-size 2 \
    --gpu-memory-utilization 0.92 \
    --max-model-len 8192 \
    --max-num-batched-tokens 32768 \
    --max-num-seqs 256 \
    --enable-chunked-prefill \
    --port 8000

# Run the benchmark
python -m vllm.entrypoints.openai.api_server \
    --model ./llama3-8b-awq-4bit \
    --quantization awq &

# Load test
python benchmarks/benchmark_serving.py \
    --model ./llama3-8b-awq-4bit \
    --num-prompts 1000 \
    --request-rate 10 \
    --endpoint /v1/completions

9. Choosing a Quantization Format

9.1 Decision Flow

Quantization format decision flow:

1. What is the goal?
   - Fine-tuning -> bitsandbytes NF4 + QLoRA
   - Production serving -> go to 2
   - Local/edge deployment -> GGUF

2. Serving framework?
   - vLLM -> AWQ (optimal) or GPTQ
   - TensorRT-LLM -> AWQ (recommended)
   - Custom implementation -> AWQ

3. Hardware?
   - NVIDIA GPU -> AWQ/GPTQ
   - Apple Silicon -> GGUF (Metal)
   - CPU only -> GGUF
   - AMD GPU -> GGUF (ROCm/Vulkan)

4. Required quality level?
   - Maximum quality -> Q5_K_M or Q6_K (GGUF)
   - Balanced -> Q4_K_M (GGUF) or AWQ 4bit
   - Minimum memory -> Q3_K_M or Q2_K (GGUF)

9.2 Recommendations by Scenario

10. Troubleshooting

10.1 Common Problems and Fixes

CUDA OOM (Out of Memory)

# Problem: GPU runs out of memory during quantization
# Fix 1: set max_memory
model = AutoAWQForCausalLM.from_pretrained(
    model_name,
    max_memory={0: "20GiB", 1: "20GiB", "cpu": "30GiB"},
)

# Fix 2: CPU offload
model = AutoAWQForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    offload_folder="./offload",
)

Quality drops after quantization

# Problem: output quality degrades sharply after quantization
# Fix 1: shrink the group size
quant_config = {"q_group_size": 64, "w_bit": 4}  # 128 -> 64

# Fix 2: use 5 bits
quant_config = {"q_group_size": 128, "w_bit": 5}  # 4bit -> 5bit (GPTQ only)

# Fix 3: use domain-specific calibration data
# For specialized domains such as medicine or law, calibrate on text from that domain

vLLM fails to load the quantized model

# Problem: vLLM does not recognize the quantized model
# Fix: check quantization_config in config.json
cat model_dir/config.json | python -m json.tool | grep -A 10 quant

# GPTQ config example (must be present in config.json):
# "quantization_config": {
#     "bits": 4,
#     "group_size": 128,
#     "quant_method": "gptq"
# }

# AWQ config example:
# "quantization_config": {
#     "bits": 4,
#     "group_size": 128,
#     "quant_method": "awq",
#     "version": "gemm",
#     "zero_point": true
# }

10.2 Performance Optimization Checklist

• Use the AWQ GEMM kernel (for batched serving)
• Enable the ExLlama v2 kernel (when using GPTQ)
• Tune the number of GPU-offloaded layers (GGUF)
• Confirm the calibration data matches the target domain
• Use group_size 128 (the default recommendation)
• Set vLLM gpu_memory_utilization to 0.9 or higher
• Shard large models with tensor parallelism
• Check and adjust the KV cache size
• Pick the kernel by batch size (GEMM vs GEMV)
• Validate quality by measuring perplexity after quantization

11. A Practical Benchmark Script

import time
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

def benchmark_model(model, tokenizer, prompts, max_tokens=128):
    """Model inference benchmark"""
    results = []

    for prompt in prompts:
        inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

        # Warmup
        with torch.no_grad():
            model.generate(**inputs, max_new_tokens=1)

        # Measure
        torch.cuda.synchronize()
        start = time.perf_counter()

        with torch.no_grad():
            outputs = model.generate(
                **inputs,
                max_new_tokens=max_tokens,
                do_sample=False,
            )

        torch.cuda.synchronize()
        elapsed = time.perf_counter() - start

        num_tokens = outputs.shape[1] - inputs.input_ids.shape[1]
        tokens_per_sec = num_tokens / elapsed

        results.append({
            "prompt_len": inputs.input_ids.shape[1],
            "output_len": num_tokens,
            "time_sec": round(elapsed, 3),
            "tokens_per_sec": round(tokens_per_sec, 1),
        })

    # GPU memory usage
    memory_gb = torch.cuda.max_memory_allocated() / (1024**3)

    avg_tps = sum(r["tokens_per_sec"] for r in results) / len(results)

    return {
        "avg_tokens_per_sec": round(avg_tps, 1),
        "gpu_memory_gb": round(memory_gb, 2),
        "results": results,
    }

# Benchmark prompts
test_prompts = [
    "Explain the concept of quantum entanglement in detail:",
    "Write a Python function to implement binary search:",
    "Describe the architecture of a modern web application:",
    "What are the key differences between SQL and NoSQL databases?",
    "Explain how neural networks learn from data:",
]

# FP16 benchmark
print("=== FP16 Benchmark ===")
model_fp16 = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B",
    torch_dtype=torch.float16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-3.1-8B")
result_fp16 = benchmark_model(model_fp16, tokenizer, test_prompts)
print(f"FP16 - Speed: {result_fp16['avg_tokens_per_sec']} tok/s, "
      f"Memory: {result_fp16['gpu_memory_gb']} GB")

del model_fp16
torch.cuda.empty_cache()

# AWQ 4bit benchmark
print("\n=== AWQ 4bit Benchmark ===")
from awq import AutoAWQForCausalLM
model_awq = AutoAWQForCausalLM.from_quantized(
    "./llama3-8b-awq-4bit",
    fuse_layers=True,
)
result_awq = benchmark_model(model_awq.model, tokenizer, test_prompts)
print(f"AWQ  - Speed: {result_awq['avg_tokens_per_sec']} tok/s, "
      f"Memory: {result_awq['gpu_memory_gb']} GB")

12. Recent Trends and Outlook

12.1 Quantization Technology Trends in 2026

1. AWQ becomes the standard: major serving frameworks such as vLLM and TensorRT-LLM adopt AWQ as their default quantization format
2. FP8 quantization: native FP8 support on H100/H200 GPUs makes 8-bit quantization possible without a performance loss
3. A widening GGUF ecosystem: rapid growth of local inference tools such as llama.cpp, ollama, and LM Studio
4. Mixed-precision quantization: applying a different bit width per layer to optimize the accuracy-efficiency balance
5. 1-bit LLM (BitNet): Microsoft's BitNet b1.58 research demonstrates that extreme quantization is possible

12.2 FP8 Quantization (the Next Standard)

# FP8 quantization in vLLM (H100/H200 only)
from vllm import LLM

llm = LLM(
    model="meta-llama/Llama-3.1-70B",
    quantization="fp8",
    dtype="auto",
    tensor_parallel_size=2,
    gpu_memory_utilization=0.92,
)

# FP8 compared with INT4:
# - Almost lossless (perplexity difference under 0.1%)
# - 50% memory savings (versus FP16)
# - High throughput from the H100 FP8 tensor cores
# - No calibration needed (dynamic quantization)

13. References

1. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers - Frantar et al., 2022
2. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration - Lin et al., 2023
3. QLoRA: Efficient Finetuning of Quantized LLMs - Dettmers et al., 2023
4. llama.cpp GitHub Repository - GGUF format and quantization implementation
5. AutoGPTQ GitHub Repository - Automated GPTQ quantization tool
6. AutoAWQ GitHub Repository - Automated AWQ quantization tool
7. vLLM Quantization Documentation - vLLM quantization guide
8. bitsandbytes GitHub Repository - NF4 and INT8 quantization
9. The Era of 1-bit LLMs (BitNet) - Ma et al., 2024
10. SmoothQuant: Accurate and Efficient Post-Training Quantization - Xiao et al., 2022