LabHub

Blog

Complete Guide to Open Source LLMs: Llama 3, Mistral, DeepSeek, Qwen, and Gemma

2026-03-17 · 14 min read · #llm#llama#mistral#deepseek#qwen#gemma#open-source#2026-03

한국어English日本語

Complete Guide to Open Source LLMs (2024-2026)

2024 and 2025 were the most turbulent years in the history of open source LLMs. Meta's Llama 3 pushed past GPT-3.5, and DeepSeek shook the AI industry by releasing GPT-4-class performance as open source. This guide pulls together the major open source LLMs — architecture, performance, licensing, and how to use them.

1. Open Source LLM Ecosystem Overview

Until early 2023, closed source models such as GPT-4 and Claude were overwhelmingly ahead on performance. The open source camp caught up quickly, however, and in the second half of 2024 several open source models reached GPT-4 level.

Advantages of open source LLMs:

Limits of open source LLMs:

Key Development Timeline

License Categories

LicenseRepresentative modelsCommercial useModified redistribution
Apache 2.0Mistral, Gemma, PhiAllowedAllowed
Meta Llama LicenseLlama seriesConditionalConditional
MITSome Phi modelsAllowedAllowed
Custom licenseDeepSeekConditionalRestricted

The main restriction in the Llama license: a service with 700 million or more monthly active users needs a separate agreement.

2. The Meta Llama Series

How Llama Evolved

Llama 1 (2023.02): 7B, 13B, 33B, 65B parameters. Released for academic research, but it leaked quickly and spread through the community.

Llama 2 (2023.07): 7B, 13B, 70B. Commercial use allowed. Includes Chat and Code versions.

Llama 3 (2024.04): 8B, 70B. 128K vocabulary, GQA applied. A large performance gain over the previous generation.

Llama 3.1 (2024.07): 8B, 70B, 405B. 128K context window, stronger multilingual support, native function calling.

Llama 3.2 (2024.09): 1B, 3B (small), 11B, 90B (multimodal). Vision models included.

Llama 3.3 (2024.12): 70B. Reaches 405B-level performance at 70B.

Llama 3 Architecture Innovations

# Key features of the Llama 3 architecture
architecture_features = {
    "Vocabulary size": "128,256 tokens (4x that of Llama 2)",
    "Positional encoding": "RoPE (Rotary Position Embedding)",
    "Attention mechanism": "GQA (Grouped Query Attention)",
    "Activation function": "SwiGLU",
    "Context window": "8K (Llama 3) / 128K (Llama 3.1+)",
    "Training data": "15T+ tokens",
}

RoPE (Rotary Position Embedding)

Instead of absolute position encoding, relative position is expressed through a rotation matrix. This helps generalization to long contexts.

GQA (Grouped Query Attention)

A variant of Multi-Head Attention (MHA) in which several query heads share a single key/value head. Inference speed and memory efficiency improve substantially.

# Multi-Head Attention vs GQA comparison
mha_params = {
    "query_heads": 32,
    "key_heads": 32,     # same as query
    "value_heads": 32,   # same as query
}

gqa_params = {
    "query_heads": 32,
    "key_heads": 8,      # 1 per group
    "value_heads": 8,    # 1 per group
    "Memory savings": "4x"
}

Llama 3 in Practice

# Using Llama 3 through HuggingFace
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
import torch

model_id = "meta-llama/Meta-Llama-3-8B-Instruct"

# 4-bit quantization (saves memory)
from transformers import BitsAndBytesConfig

quantization_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    quantization_config=quantization_config,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Build the pipeline
text_generator = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
)

messages = [
    {"role": "system", "content": "You are a helpful AI assistant."},
    {"role": "user", "content": "Explain the difference between machine learning and deep learning."},
]

output = text_generator(
    messages,
    max_new_tokens=512,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
)
print(output[0]['generated_text'][-1]['content'])

3. Mistral AI

Mistral 7B

A 7B model released in September 2023 that recorded the strongest performance among models of the same size at the time.

Core technical innovations:

Sliding Window Attention (SWA): each token attends only to the previous W tokens. A fixed-size cache makes processing of unlimited length possible.

Rolling Buffer Cache: implements the KV cache as a circular buffer to save memory.

Pre-fill and Chunking: splits long prompts into chunks for processing.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "mistralai/Mistral-7B-Instruct-v0.3"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.float16,
    device_map="auto"
)

# Mistral instruction format
messages = [
    {"role": "user", "content": "Implement quicksort in Python."}
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=1024,
    do_sample=True,
    temperature=0.7
)

response = tokenizer.decode(generated_ids[0][len(model_inputs.input_ids[0]):], skip_special_tokens=True)
print(response)

Mixtral 8x7B (MoE)

Mixtral uses a Mixture of Experts (MoE) architecture. It has 46.7B parameters in total, but only 12.9B are activated at inference time.

# The MoE architecture explained
moe_params = {
    "Total experts": 8,
    "Experts activated": 2,
    "Total parameters": "46.7B",
    "Active parameters": "12.9B",
    "Inference speed": "similar to a 12.9B model",
    "Performance": "competes with 70B models",
}

4. The DeepSeek Series

DeepSeek V3

A 671B MoE model released in December 2024 by DeepSeek of China. Putting performance that competes with GPT-4o and Claude 3.5 Sonnet into open source sent a shock through the industry.

Multi-Head Latent Attention (MLA): the core innovation, compressing the KV cache into a low-dimensional space.

# The core idea behind MLA (conceptual code)
class MultiHeadLatentAttention:
    """
    Conventional MHA: the K, V cache grows in proportion to token count
    MLA: K and V are compressed into low-dimensional latent vectors for storage
    - Memory usage: 5-13x reduction
    - Performance: on par with MHA
    """
    def __init__(self, hidden_dim=7168, num_heads=128, kv_lora_rank=512):
        self.hidden_dim = hidden_dim
        self.num_heads = num_heads
        self.kv_lora_rank = kv_lora_rank  # compressed KV dimension
        # Conventional KV cache: num_heads * head_dim * 2
        # MLA KV cache: kv_lora_rank (far smaller)

DeepSeek MoE innovations:

# Using the DeepSeek V3 API (OpenAI compatible)
from openai import OpenAI

client = OpenAI(
    api_key="<DEEPSEEK_API_KEY>",
    base_url="https://api.deepseek.com"
)

response = client.chat.completions.create(
    model="deepseek-chat",
    messages=[
        {"role": "system", "content": "You are a helpful assistant"},
        {"role": "user", "content": "Implement a binary search algorithm in Python."}
    ],
    stream=False
)

print(response.choices[0].message.content)

DeepSeek R1

A reasoning-specialized model released in January 2025. It showed math and coding performance that competes with OpenAI o1.

Core characteristics:

# DeepSeek R1 usage example
client = OpenAI(
    api_key="<DEEPSEEK_API_KEY>",
    base_url="https://api.deepseek.com"
)

# R1 is a reasoning model, so reasoning_content is included
response = client.chat.completions.create(
    model="deepseek-reasoner",
    messages=[
        {"role": "user", "content": "Find the sum of the prime numbers from 1 to 100."}
    ]
)

# The reasoning process
reasoning = response.choices[0].message.reasoning_content
# The final answer
answer = response.choices[0].message.content

print("Reasoning process (summary):", reasoning[:200], "...")
print("Final answer:", answer)

What Korean Companies Should Weigh

DeepSeek was developed by a Chinese company, so a company adopting it should consider the following:

  1. Data sovereignty: when the API is used, data is sent to servers in China
  2. Open source local execution: deploying the model weights on your own servers means no data leaves
  3. License: commercial use is possible, but some restrictions need checking
  4. Regulatory risk: in certain industries (finance, healthcare, defense) legal review is needed before use

5. The Alibaba Qwen Series

Qwen2 / Qwen2.5

A model series developed by Alibaba Cloud. It shows strength in Asian languages (Chinese, Korean, Japanese).

# Qwen2.5 usage example
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "Qwen/Qwen2.5-7B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

messages = [
    {"role": "system", "content": "You are Qwen, a helpful assistant."},
    {"role": "user", "content": "Explain current AI trends in Korean."}
]

text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)

model_inputs = tokenizer([text], return_tensors="pt").to(model.device)

generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=1024,
    temperature=0.7,
    top_p=0.8,
    repetition_penalty=1.05,
)

response_ids = [
    output_ids[len(input_ids):]
    for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]

response = tokenizer.batch_decode(response_ids, skip_special_tokens=True)[0]
print(response)

Qwen2-VL (Multimodal)

The multimodal version, able to process images, video, and documents.

from transformers import Qwen2VLForConditionalGeneration, AutoProcessor
from qwen_vl_utils import process_vision_info
import torch

model = Qwen2VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2-VL-7B-Instruct",
    torch_dtype=torch.bfloat16,
    device_map="auto"
)
processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct")

# Image analysis
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "https://example.com/chart.png"},
            {"type": "text", "text": "Explain the trend visible in this chart, in Korean."},
        ],
    }
]

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
image_inputs, video_inputs = process_vision_info(messages)

inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    return_tensors="pt",
).to(model.device)

generated_ids = model.generate(**inputs, max_new_tokens=512)
response = processor.batch_decode(
    [out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)],
    skip_special_tokens=True
)[0]
print(response)

Qwen Model Lineup Comparison

ModelParametersSpecialtyContext
Qwen2.5-0.5B0.5BLightweight128K
Qwen2.5-7B7BGeneral purpose128K
Qwen2.5-72B72BHigh performance128K
Qwen2.5-Coder-7B7BCoding128K
Qwen2.5-Math-7B7BMath4K
Qwen2-VL-7B7BMultimodal128K

6. Google Gemma

Gemma 2 Architecture

A small open source model developed by Google DeepMind, built on Gemini technology.

Core techniques:

# Using Gemma 2
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-9b-it")
model = AutoModelForCausalLM.from_pretrained(
    "google/gemma-2-9b-it",
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

# Gemma chat format
chat = [
    {"role": "user", "content": "Explain the difference between generators and iterators in Python."},
]

prompt = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
inputs = tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt")

outputs = model.generate(
    inputs.to(model.device),
    max_new_tokens=1024,
    do_sample=True,
    temperature=1,
    top_k=50,
    top_p=0.95,
)

response = tokenizer.decode(outputs[0][len(inputs[0]):], skip_special_tokens=True)
print(response)

Mobile/Edge Deployment

Gemma's small models (2B, 9B) perform well, which makes them a good fit for mobile deployment.

# On-device execution through MediaPipe (conceptual)
# Using the Google AI Edge SDK

# Running Gemma on Android/iOS
from mediapipe.tasks.python.genai import bundler, llm_inference

# Bundling the model
bundler.bundle_model(
    model_path="gemma-2-2b-it-q4_k_m.gguf",
    tokenizer_path="tokenizer.model",
    start_token="<start_of_turn>",
    stop_tokens=["<end_of_turn>"],
    output_path="gemma_bundled.task",
)

7. The Microsoft Phi Series

Phi-3 / Phi-4

A small, high-performance model series developed by Microsoft Research. Its defining trait is training on high-quality synthetic data at a "textbook level".

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "microsoft/Phi-4"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    torch_dtype=torch.bfloat16,
    trust_remote_code=True
)

messages = [
    {"role": "system", "content": "You are a helpful AI assistant."},
    {"role": "user", "content": "Explain quantum computing in simple terms."}
]

pipe = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
)

output = pipe(messages, max_new_tokens=512, return_full_text=False)
print(output[0]['generated_text'])

Small but Strong: The Answer Is Data Quality

What sits at the core of the Phi series is data quality.

Training data composition (Phi-3-mini):
- "Textbook level" synthetic data: about 1T tokens
- Textbooks, Wikipedia, code: high-quality filtering
- Crawled data: only what passes the quality filter is used

The "Less is More" philosophy:
- 7B-model performance from 3.8B parameters
- Data efficiency: trained on 3.3T tokens (1/4 of Llama 3)

8. Open Source LLM Selection Guide

The Best Model per Task (as of 2025)

TaskRecommended modelWhy
General conversation (Korean)EXAONE 3.5 7.8BKorean-specialized
Code generationQwen2.5-Coder-32BBest code performance
Math/reasoningDeepSeek-R1 (distilled)Reasoning-specialized
MultimodalQwen2-VL / LLaVAImage understanding
Edge/mobileGemma 2 2BSmall and strong
General high performanceLlama 3.3 70BBalanced performance
Cost efficiencyMistral 7BFast inference

Recommendations by Size

3B and under (runs without a GPU):

7B-8B (8GB VRAM):

13B-14B (16GB VRAM):

30B-34B (24GB VRAM):

70B+ (80GB VRAM or multi-GPU):

Korean Language Support Comparison

ModelKorean comprehensionKorean generationKorean cultural understanding
EXAONE 3.5★★★★★★★★★★★★★★★
HyperCLOVA X★★★★★★★★★★★★★★★
Qwen2.5-72B★★★★★★★★★★★
Llama 3.3 70B★★★★★★★★
DeepSeek V3★★★★★★★★
Mistral 7B★★★★★

9. Running Locally with Ollama

Installing and Using Ollama

Ollama is a tool that makes it easy to run open source LLMs locally.

# macOS/Linux install
curl -fsSL https://ollama.com/install.sh | sh

# Download and run a model
ollama run llama3.2

# Other model examples
ollama run mistral
ollama run qwen2.5:7b
ollama run deepseek-r1:8b
ollama run gemma2:9b

Using Ollama from Python

import ollama

# A simple chat
response = ollama.chat(model='llama3.2', messages=[
    {
        'role': 'user',
        'content': 'Tell me 5 traditional Korean dishes.',
    },
])
print(response['message']['content'])

Streaming Responses

import ollama

def stream_response(model: str, prompt: str):
    print(f"Model: {model}")
    print(f"Prompt: {prompt}")
    print("Response: ", end="", flush=True)

    stream = ollama.chat(
        model=model,
        messages=[{'role': 'user', 'content': prompt}],
        stream=True,
    )

    full_response = ""
    for chunk in stream:
        text = chunk['message']['content']
        print(text, end='', flush=True)
        full_response += text

    print()
    return full_response

response = stream_response('mistral', 'Implement the Fibonacci sequence in Python.')

FastAPI + Ollama Server

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import ollama
import json

app = FastAPI()

@app.post("/chat")
async def chat(request: dict):
    model = request.get("model", "llama3.2")
    message = request.get("message", "")

    async def generate():
        stream = ollama.chat(
            model=model,
            messages=[{"role": "user", "content": message}],
            stream=True,
        )
        for chunk in stream:
            data = {
                "content": chunk["message"]["content"],
                "done": chunk.get("done", False)
            }
            yield f"data: {json.dumps(data, ensure_ascii=False)}\n\n"

    return StreamingResponse(generate(), media_type="text/event-stream")

@app.get("/models")
async def list_models():
    models = ollama.list()
    return {"models": [m["name"] for m in models["models"]]}

Performance vs Memory Requirements by Ollama Model

# Memory required per model (FP16 basis)
model_requirements = {
    "llama3.2:3b":    {"vram_gb": 2,  "speed": "very fast",  "quality": "fair"},
    "llama3.2:8b":    {"vram_gb": 5,  "speed": "fast",       "quality": "good"},
    "mistral:7b":     {"vram_gb": 5,  "speed": "fast",       "quality": "good"},
    "qwen2.5:7b":     {"vram_gb": 5,  "speed": "fast",       "quality": "good"},
    "gemma2:9b":      {"vram_gb": 6,  "speed": "moderate",   "quality": "good"},
    "llama3.3:70b":   {"vram_gb": 40, "speed": "slow",       "quality": "very good"},
    "qwen2.5:72b":    {"vram_gb": 43, "speed": "slow",       "quality": "very good"},
    "deepseek-r1:8b": {"vram_gb": 5,  "speed": "moderate",   "quality": "reasoning-specialized"},
}

# Quantized versions (Q4 basis)
quantized_requirements = {
    "llama3.2:8b-q4":    {"vram_gb": 3,   "speed": "very fast"},
    "llama3.3:70b-q4":   {"vram_gb": 20,  "speed": "moderate"},
    "qwen2.5:72b-q4":    {"vram_gb": 22,  "speed": "moderate"},
}

10. Open Source LLM Serving Stack

vLLM (High-Performance Inference Server)

pip install vllm

from vllm import LLM, SamplingParams

# Load the model
llm = LLM(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    tensor_parallel_size=2,   # use 2 GPUs
    gpu_memory_utilization=0.9,
    max_model_len=8192,
)

# Batch inference
prompts = [
    "Explain the characteristics of Korean NLP.",
    "What is the difference between deep learning and machine learning?",
    "What is the Transformer architecture?",
]

sampling_params = SamplingParams(
    temperature=0.7,
    top_p=0.9,
    max_tokens=512,
)

outputs = llm.generate(prompts, sampling_params)

for output in outputs:
    print(f"Prompt: {output.prompt[:50]}...")
    print(f"Response: {output.outputs[0].text[:200]}")
    print()

vLLM OpenAI-Compatible Server

# Run the server
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3-8B-Instruct \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 2

# Using the vLLM server with the OpenAI client
from openai import OpenAI

client = OpenAI(
    api_key="EMPTY",
    base_url="http://localhost:8000/v1"
)

response = client.chat.completions.create(
    model="meta-llama/Meta-Llama-3-8B-Instruct",
    messages=[{"role": "user", "content": "Hello!"}],
    max_tokens=256,
)
print(response.choices[0].message.content)

Wrapping Up

The open source LLM ecosystem developed at a remarkable pace through 2024-2026. The key points:

Performance:

Korean:

Deployment:

Choosing the right model means weighing task, level of Korean support, hardware availability, and licensing together. If you are building a Korean-language service, look at EXAONE or HyperCLOVA X first; if you need general-purpose capability, consider Llama 3.3 70B or Qwen2.5 72B.

Comments

No comments yet.

Sign in to leave a comment