LabHub

Blog

LLM, Tool Calling & Embedding Benchmarks Deep Dive: What Each Benchmark Actually Measures

2026-03-17 · 24 min read · #llm#benchmarks#mmlu#mteb#bfcl#embedding#tool-calling#document-parsing

한국어English日本語中文

LLM, Tool Calling & Embedding Benchmarks Deep Dive

When evaluating AI models, benchmark names appear everywhere. MMLU 85%, HumanEval 90%, MTEB #1 — let's fully understand what these numbers actually mean, how each benchmark works, and which ones matter for which use cases.

1. LLM General Benchmarks

MMLU (Massive Multitask Language Understanding)

Published by UC Berkeley in 2020, MMLU measures the breadth of an LLM's knowledge and comprehension across diverse academic fields.

How it works:

Example question:
Subject: High School Chemistry

Example 1: What is the element with atomic number 6?
(A) Nitrogen  (B) Oxygen  (C) Carbon  (D) Neon
Answer: (C)

...5 examples provided...

Test: What is required for an ionic bond to form?
(A) Between two non-metal atoms
(B) Between a metal and non-metal atom
(C) Between two metal atoms
(D) Between a noble metal and non-metal
Answer: ?

Score interpretation:

Limitations:

HellaSwag

Published in 2019, HellaSwag measures "commonsense reasoning" and "sentence completion." The name stands for Harder Endings, Longer contexts, and Low-shot Activities For Situations With Adversarial Generations.

How it works:

Example:
Situation: "A man is grilling hot dogs. He flips the hot dogs with tongs."

What comes next?
(A) He throws the hot dogs into the fire.
(B) He places the finished hot dogs in buns.
(C) He takes ingredients out of the refrigerator.
(D) He opens a recipe book.

Answer: (B)

Score interpretation:

Limitations:

ARC (AI2 Reasoning Challenge)

Created by the Allen Institute for AI, this benchmark measures reasoning ability using elementary-to-middle school science questions.

Two sets:

Easy Set:

Challenge Set:

Challenge Set example:
"What is the primary role of stomata in plant leaves?"

(A) Absorbing light for photosynthesis
(B) Regulating gas exchange and water transpiration
(C) Transporting water from roots to leaves
(D) Moving sugars to other plant parts

→ Requires understanding of function, not just surface retrieval

Score interpretation:

TruthfulQA

Published in 2021, TruthfulQA measures a model's tendency to produce hallucinations and false beliefs.

How it works:

Example questions:
- "Is it true that we only use 10% of our brains?"
  → Misconception. Correct: "No, this is a myth."

- "What is a rectangle with right-angle diagonals called?"
  → Trick question. Rectangle diagonals are not right-angle.

- "Was Napoleon very short?"
  → Actually average height for his era.

Score interpretation:

Key point: TruthfulQA is designed to be difficult to score high on. A low-scoring model is particularly good at producing believable false information.

WinoGrande

Published in 2019, WinoGrande uses 44,000 commonsense reasoning problems to measure pronoun disambiguation ability.

How it works:

Example:
"The trophy didn't fit in the brown suitcase because ___ was too big."
(A) it [trophy]
(B) it [suitcase]
→ Requires understanding that the trophy was too big

"At the library, Sarah read more books than Amy. ___ enjoyed reading."
(A) Sarah
(B) Amy
→ Commonsense judgment required

Score interpretation:

BIG-Bench (Beyond the Imitation Game Benchmark)

A large-scale benchmark containing 204 diverse tasks that evaluates capabilities difficult to assess with existing benchmarks.

BIG-Bench Hard (BBH):

BBH example tasks:
- Boolean Expressions: Evaluate "(True and False) or (not True and True)"
- Causal Judgment: Determine direction of causation
- Formal Fallacies: Identify logical errors
- Movie Recommendation: Preference-based recommendations
- Object Counting: Count objects from textual descriptions
- Temporal Sequences: Sort events chronologically
- Word Sorting: Sort by alphabet or given condition

Chain-of-Thought effect:

GPQA (Graduate-Level Google-Proof Q&A)

Published in 2023, GPQA requires PhD-level scientific expertise and is designed so that even Google searches cannot easily find the answer.

How it works:

Score interpretation:

Example (Physics):
"What is the primary advantage of topological qubits in quantum computers?"

(A) Can only operate at absolute zero temperature
(B) Topologically protected, resistant to environmental noise
(C) Faster gate speeds than traditional transistors
(D) Support unlimited qubit count

→ Requires deep understanding of quantum error correction

LiveBench

A dynamic benchmark that adds new questions monthly to prevent data contamination.

How it works:

Why it matters:

2. Coding Benchmarks

HumanEval

Published by OpenAI in 2021, HumanEval is the most widely used coding benchmark for measuring Python programming ability.

How it works:

# Example problem
def has_close_elements(numbers: List[float], threshold: float) -> bool:
    """
    Check whether any two numbers in the list are closer
    to each other than the given threshold.

    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
    False
    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
    True
    """
    # Model must implement this

pass@k metric:

Score interpretation:

Limitations:

MBPP (Mostly Basic Python Problems)

A collection of 374 crowd-sourced Python problems published by Google Research.

Differences from HumanEval:

# MBPP example
"""
Write a function to find the maximum product subarray.
assert max_product_subarray([6, -3, -10, 0, 2]) == 180
assert max_product_subarray([-1, -3, -10, 0, 60]) == 60
"""

SWE-bench

Published in 2023, SWE-bench measures the ability to resolve real GitHub issues and bugs.

How it works:

Example issue:
Repository: scikit-learn
Issue: "KNeighborsClassifier.predict() returns incorrect
        results when given sparse matrix input"

What the model must do:
1. Understand the issue description
2. Locate relevant source code
3. Generate a bug fix patch
4. Ensure existing tests pass

SWE-bench Lite:

Score interpretation:

Why it matters:

LiveCodeBench

A dynamic coding benchmark that continuously adds new problems from LeetCode, AtCoder, and CodeForces to prevent data contamination.

Features:

3. Reasoning & Math Benchmarks

GSM8K (Grade School Math)

A benchmark of 8,500 elementary school math problems published by OpenAI in 2021.

Features:

Example problem:
"Janet's ducks lay 16 eggs per day. Every morning
she eats 3 for breakfast and uses 4 for muffins for
her friends. She sells the remainder at $2 per egg.
How much does she earn per day?"

Chain-of-Thought reasoning:
1. Daily eggs: 16
2. Eaten: 3
3. Used for muffins: 4
4. Eggs to sell: 16 - 3 - 4 = 9
5. Earnings: 9 * 2 = $18

Answer: $18

Score interpretation:

MATH

A collection of 12,500 competition-level math problems published in 2021.

7 subject areas:

5 difficulty levels:

Level 5 example:
"Factor x^4 + 4x^3 - 2x^2 - 12x + 9"

Answer: (x^2 + 2x - 3)^2 = (x+3)^2(x-1)^2
→ Requires advanced algebraic manipulation

Score interpretation:

AIME (American Invitational Mathematics Examination)

Real problems from the American math olympiad qualifying exam.

Features:

Score interpretation:

4. Tool Calling / Function Calling Benchmarks

BFCL (Berkeley Function Calling Leaderboard)

The most comprehensive function calling benchmark, published by UC Berkeley in 2024.

2,000+ function calling scenarios:

Categories:

  1. Simple Function Calling — single function, clear parameters
  2. Multiple Functions — select the right function from several options
  3. Parallel Functions — invoke multiple functions simultaneously
  4. Nested Functions — call functions within other functions
  5. REST API — call real HTTP API endpoints

Evaluation criteria:

AST validation approach:

# Ground truth function call
get_weather(
    location="Seoul, Korea",
    unit="celsius",
    forecast_days=3
)

# Model-generated call
get_weather(
    location="Seoul",  # Partial match — allowed?
    unit="C",          # Type/format error
    days=3             # Parameter name error!
)

AST (Abstract Syntax Tree) parsing verifies structural correctness

Supported languages/environments:

Score interpretation (2024):

tau-bench (τ-bench)

A benchmark measuring real agent task completion, going beyond simple function call accuracy to measure end-to-end task success rates.

How it works:

Example scenario:
"Find a one-way flight from New York to Paris on March 20,
book the cheapest option, and send a confirmation email."

Required steps:
1. search_flights(origin="NYC", destination="Paris", date="2026-03-20")
2. select_flight(flight_id="AF001", criteria="cheapest")
3. book_flight(flight_id="AF001", passenger_info=...)
4. send_confirmation_email(booking_id=..., email=...)

→ Measures accuracy of each step AND overall completion

ToolBench / ToolEval

Published in 2023, this benchmark evaluates tool-use ability with 16,000 real REST APIs.

How it works:

Solvable Pass Rate (SoPR) metric:

Evaluation criteria:

AgentBench

Published in 2023, this benchmark measures autonomous LLM agent ability across 8 different environments.

8 environments:

  1. OS — Operating system tasks (file manipulation, command execution)
  2. DB — Database queries and manipulation
  3. Knowledge Graph — Knowledge graph traversal
  4. Digital Card Game — Strategic card game
  5. Lateral Thinking Puzzles — Creative problem solving
  6. House Holding — Home management in a virtual environment
  7. Web Shopping — Online shopping tasks
  8. Web Browsing — Web navigation and information gathering
OS environment example:
"Find all .py files in the current directory created in 2023
and move them to a 'python_files' folder."

→ Requires combining find, mkdir, mv commands
→ Measures multi-step decision-making and error recovery

Score interpretation:

5. Embedding Benchmarks

MTEB (Massive Text Embedding Benchmark)

Published in 2022, MTEB is the most comprehensive benchmark for evaluating text embedding models.

56 datasets, 8 task types:

1. Retrieval

Example: "How to sort a list in Python"
→ Rank relevant Stack Overflow answers and documentation

2. Classification

3. Clustering

4. Semantic Textual Similarity (STS)

Example:
Sentence 1: "A dog is running in the park"
Sentence 2: "A canine is sprinting outdoors"
→ High similarity (~4.0/5.0)

Sentence 1: "The weather is sunny today"
Sentence 2: "I love eating pizza"
→ Low similarity (~0.5/5.0)

5. Reranking

6. Summarization

7. Pair Classification

Example:
- Duplicate detection: "How to sort a Python list" vs "Sort list in Python"
  → Duplicate (True)
- "Apples are fruit" vs "I like swimming"
  → Unrelated (False)

8. Bitext Mining

Example:
English: "The weather is nice today"
Korean: "오늘 날씨가 좋다"
→ Parallel pair detection

MTEB Leaderboard (HuggingFace):

BEIR (Benchmarking Information Retrieval)

Published in 2021, BEIR measures information retrieval performance across 18 diverse domains.

18 datasets:

nDCG@10 metric:

nDCG@10 = Normalized Discounted Cumulative Gain of top 10 results

Relevance scores:
- Highly relevant: 3 points
- Relevant: 2 points
- Marginally relevant: 1 point
- Not relevant: 0 points

Higher-ranked results receive more weight

Zero-shot evaluation:

6. RAG & Document Parsing Benchmarks

RAGAS (Retrieval Augmented Generation Assessment)

A comprehensive framework for measuring the quality of RAG systems.

5 core metrics:

1. Faithfulness

Context: "Python was created by Guido van Rossum in 1991."
Question: "When was Python created and by whom?"

High Faithfulness answer:
"Python was created by Guido van Rossum in 1991."

Low Faithfulness answer (hallucination):
"Python was created by Guido van Rossum in 1989,
 in Amsterdam, the Netherlands..."
→ Date and location not in context are fabricated

2. Answer Relevance

3. Context Precision

4. Context Recall

5. Context Entity Recall

RULER (Retrieval Under Long-context Evaluation Regime)

A benchmark measuring long-context LLM ability, going beyond simple Needle-in-a-Haystack to evaluate complex long-document understanding.

Task types:

  1. NIAH (Needle-in-a-Haystack): Find specific information in a long document
  2. Multi-key NIAH: Find multiple pieces of information simultaneously
  3. Multi-value NIAH: Extract multiple values for a single key
  4. Multi-hop Tracing: Reason through multiple steps following information chains
  5. Aggregation: Aggregate information across the full document
  6. QA: Question answering based on long context
Multi-hop Tracing example (in a 128K token document):
"Alice's manager is Bob. Bob's birthday is March 15th.
... (tens of thousands of tokens of unrelated content) ...
What is Alice's manager's birthday?"

→ Measures the ability to connect Alice → Bob → March 15th

DocVQA

Measures visual question answering ability on real document images.

How it works:

Example:
[Invoice image]
Question: "What is the total tax amount?"
→ Locate the tax line item and extract the value

[Medical form]
Question: "What is the patient's date of birth?"
→ Identify the specific field location and extract value

ANLS (Average Normalized Levenshtein Similarity) metric:

FinanceBench

A Q&A benchmark based on financial documents (10-K annual reports, 10-Q quarterly reports).

How it works:

Example:
[Apple Inc. 2023 Annual Report]
Question: "What was the year-over-year revenue growth rate
          of the Services segment in 2023?"

Required capabilities:
1. Find 2023 Services revenue
2. Find 2022 Services revenue
3. Calculate growth rate: (2023-2022)/2022 * 100

7. Multimodal Benchmarks

MMBench / MMMU

MMBench:

MMMU (Massive Multi-discipline Multimodal Understanding):

MMMU example:
[Chemical bonding diagram image]
Question: "What is the bond angle in this molecular structure?"
→ Requires visual interpretation of chemical structures

DocBench / OCRBench

OCRBench:

DocBench:

8. Benchmark Selection Guide

Reference benchmarks by use case:

Use CasePrimary BenchmarksSecondary Benchmarks
Chatbot / QA systemsMMLU, TruthfulQAHellaSwag, WinoGrande
Code generation toolsHumanEval, SWE-benchMBPP, LiveCodeBench
Agents / AutomationBFCL, AgentBenchτ-bench, ToolBench
RAG systemsMTEB Retrieval, BEIRRAGAS, RULER
Document processingDocVQA, OCRBenchFinanceBench
Math / ScienceMATH, GSM8KGPQA, AIME
Embedding model selectionFull MTEBBEIR by domain
MultimodalMMMU, MMBenchDocVQA

9. Limitations and Caveats

Data Contamination

The problem:

Mitigations:

Score Variation from Prompt Engineering

Same model, different prompts:
GSM8K standard prompting: 70%
GSM8K CoT prompting: 92%

→ Scores without stated prompting method are meaningless

The Gap Between Benchmark Scores and Real-World Usability

Language Bias

Benchmark Saturation

Quiz: Test Your Benchmark Understanding

Quiz 1: What does 5-shot learning in MMLU mean?

Answer: Before each test question, 5 example questions with their correct answers are provided in the prompt.

Explanation: In 5-shot learning, the prompt includes 5 example problems and their answers from the relevant subject before the actual test question. This guides the model to understand the question format and produce answers in the expected style. 0-shot means no examples, 1-shot means one example, and few-shot means a small number of examples.

Quiz 2: Why does GPT-4 score lower than humans on TruthfulQA?

Answer: TruthfulQA is deliberately designed to test misconceptions and false beliefs that humans commonly hold. AI models also learn incorrect information from training data and tend to generate plausible-sounding misinformation.

Explanation: The core purpose of TruthfulQA is to measure a model's tendency to produce "plausible but wrong" answers (hallucination). Humans can say "I'm not sure," but LLMs often confidently generate incorrect information. The benchmark is intentionally designed to be hard to score high on — differences between models are more meaningful than the absolute score itself.

Quiz 3: Why is pass@10 always higher than pass@1 in HumanEval?

Answer: pass@10 only requires at least 1 success out of 10 attempts, so it has a higher or equal probability of success compared to a single attempt (pass@1).

Explanation: pass@k is the probability of at least one success in k attempts. The formula is approximately 1 - (probability of failure)^k. As k increases, the probability of success increases, so pass@100 >= pass@10 >= pass@1 always holds. This metric is also used to assess the diversity and creativity of a model's code generation.

Quiz 4: Why does BFCL use AST validation?

Answer: To verify the structural meaning of code rather than doing text matching. AST parses code into a syntax tree to accurately check function names, parameter names, types, and values.

Explanation: Simple text comparison might treat get_weather(city='Seoul') and get_weather(city = 'Seoul') as different. AST parsing ignores surface differences like whitespace and quote style to verify actual semantic equivalence. It also recognizes the same call regardless of parameter order, enabling more accurate evaluation.

Quiz 5: Why does MTEB use nDCG@10 for Retrieval tasks?

Answer: nDCG@10 measures the quality of the top 10 search results while assigning more weight to higher-ranked results. This reflects real user behavior since users typically only look at the top results.

Explanation: nDCG (Normalized Discounted Cumulative Gain) discounts relevance scores (0~3) with a log function so that higher-ranked results are weighted more heavily. The @10 means only the top 10 results are evaluated. For example, a relevant document in position 1 receives a much higher score than the same document in position 10.

Quiz 6: What is the difference between Faithfulness and Answer Relevance in RAGAS?

Answer: Faithfulness measures whether the answer is grounded in the retrieved context (does not fabricate), while Answer Relevance measures whether the answer actually addresses the core of the question.

Explanation: The two metrics catch different failure modes. Low Faithfulness means the model is making up content not in the context (hallucination). Low Answer Relevance means the model is faithful to the context but answering something other than what was asked. A good RAG system needs both metrics to be high.

Quiz 7: Why is SWE-bench harder and more realistic than HumanEval?

Answer: SWE-bench uses real GitHub issues and codebases. Unlike writing a single function, it requires understanding thousands of lines of existing code, diagnosing the root cause of a bug, making minimal targeted changes, and passing an existing test suite.

Explanation: HumanEval involves writing clean function implementations, but SWE-bench simulates real software development. The model must (1) understand the issue description, (2) navigate the codebase, (3) diagnose the bug, (4) decide how to fix it, (5) generate a patch, and (6) verify it passes existing tests. This closely mirrors the everyday work of a real developer.

Quiz 8: What are the main solutions to the data contamination problem?

Answer: Dynamic benchmarks (LiveBench, LiveCodeBench), private test sets, continuous addition of new problems, and generative evaluation are the main solutions.

Explanation: Data contamination occurs when test questions are included in training data, producing artificially high scores. LiveBench continuously adds new problems from recent arxiv papers and competitive programming sites so models cannot preview them. Some approaches also require model submitters to declare whether the test set was included in training data.

Quiz 9: Why is zero-shot evaluation important in BEIR?

Answer: To measure the true generalization ability of embedding models. A model that works well across diverse domains without domain-specific fine-tuning is far more practical.

Explanation: When building real RAG systems, you often need to handle documents from diverse domains like medicine, law, and finance. Training separate models for each domain is costly, so embedding models that work well across domains in zero-shot settings are much more practical. BEIR evaluates this generalization ability across 18 domains.

Conclusion: Using Benchmarks Wisely

Benchmark scores show only one facet of model capability. It is essential to choose benchmarks that match your actual use case and consider multiple benchmarks holistically rather than relying on any single one.

Core principles:

  1. Choose benchmarks aligned with your goal: For code generation, HumanEval is more relevant than MMLU
  2. Consider multiple benchmarks together: Ranking #1 on a single benchmark does not mean best in all areas
  3. Check the prompting method: Verify whether results used CoT vs standard prompting
  4. Be aware of data contamination: Cross-check with dynamic benchmarks
  5. Test directly: Ultimately, evaluate on your actual use case

Benchmarks are maps, not the territory itself. Use multiple good maps to choose the optimal model for your needs.

Read the original: https://www.youngju.dev/blog/ai/2026-03-17-llm-tool-calling-embedding-benchmark-deep-dive.en

Comments

No comments yet.

Sign in to leave a comment