LabHub

Blog

Beyond OCR — OCR-free Document Understanding and Unified Models

2026-06-26 · 20 min read · #ai-papers#ocr-free#document-understanding#multimodal#donut#vision-language-model

한국어English日本語中文

Introduction: Why Go Beyond OCR

Pulling the total from a receipt, extracting line items from a contract, turning a chart into a table. Such document-understanding tasks have long been built on OCR (optical character recognition). But OCR is only good at reading characters; it does not tell you what those characters mean or which cell they belong to.

So traditional document processing chains several stages together like links. It detects where the characters are, recognizes what those characters are, analyzes the page layout, and finally interprets the meaning. The more stages, the more an error in one place propagates to the next, the accumulated-error problem.

Recently an OCR-free approach has risen, where a single model replaces the whole chain. It takes an image as input and directly produces the answer we want (text, key-value, table). In this post we organize the limits of the traditional pipeline, OCR-free end-to-end models, and the move toward unified models that bundle detection, recognition, and parsing into one.

OCR and Document Understanding Are Different

First, let us clarify terms. OCR is transcribing characters from an image into text. Document understanding is grasping what structure and meaning that text has. They are different stages, and traditional systems handled them separately.

OCR:                  image -> "total 32,500"  (characters to text)
document understanding: "total 32,500" + position/context -> {field: total, value: 32500}  (assigning meaning)

The core claim of the OCR-free approach is that there is no need to separate the two. When one model grasps characters and meaning at once while looking at the image, it can preserve information lost passing through intermediate text (position, font, layout context). It is similar to how a person looking at a receipt does not read each character and then separately attach meaning.

Core Principles: The Traditional OCR Pipeline and Its Limits

A traditional document-understanding pipeline usually consists of the following stages.

[document image]
   |
   v
[1. text detection] --> find boxes of character/word regions
   |
   v
[2. text recognition] --> turn characters in each box into a string
   |
   v
[3. layout analysis] --> grasp title/body/table/cell structure
   |
   v
[4. information extraction] --> key-value, table, meaning interpretation

Each stage is usually a separate model, trained and tuned separately. This structure has clear limits.

These limits motivate the OCR-free approach. Fewer stages means less accumulated error, and a single model that sees characters and meaning together enables context correction.

Deep Dive 1: OCR-free End-to-End Document Understanding

The Donut Family: Structured Output Directly From Image Without OCR

Donut (Document Understanding Transformer) proposed an approach that removes the OCR stage entirely and generates structured output (e.g., JSON) directly from a document image. A vision encoder reads the image and a text decoder autoregressively generates the result we want.

[document image]
   |
   v
[vision encoder] --> sequence of image features
   |
   v
[text decoder] --> structured output (e.g., JSON key-value)

The strength is clear. There is no separate OCR box detection or recognition model, and the model learns character shape and meaning at once. Make answers in the format of the information you want to extract and train on them, and the model produces answers in that format directly. Accumulated error disappears and the pipeline simplifies.

There is a trade-off. With no explicit OCR, it depends heavily on the answer format and sufficient data. And when the decoder must generate long output, speed can be an issue.

VLM-Based Document Understanding

The vision-language model (VLM) covered in the earlier post inherently has OCR-free document understanding ability. It turns images into visual tokens and an LLM reads them together with text, so it can take a document image and answer questions or extract tables.

[document image] --> [VLM] --> answer questions like "what's the total?" in natural language
                          --> structure tables into markdown/JSON
                          --> even point to the location (coordinates) of a specific field

The advantage of a VLM is generality. One model handles captioning, VQA, OCR, and grounding, and you can flexibly get the output you want with natural-language instructions. Thanks to strong language ability it tends to infer blurry or partially occluded characters from context.

Deep Dive 2: High Resolution, Layout, Tables, Formulas

Documents differ from natural images. They are dense with small text and full of structures like tables and formulas. Handling them well requires a few devices.

High-Resolution Handling

Reading small text in a document requires sufficient resolution. Force-resizing the image small smears the characters. So OCR-free models use strategies to handle high-resolution input.

high-resolution document strategies
method A: tiling             large page -> several tiles -> process each -> merge
method B: dynamic resolution  keep original ratio -> token count proportional to size

The higher the resolution the better the detail, but token count and cost grow. This balance is central to document model design.

Layout, Tables, Formulas

Since the format of the answer data determines the model output format for such structural elements, preparing consistent structured answers is important.

Deep Dive 3: Multilingual OCR and Unified Models

Multilingual OCR

Documents are written in many languages. Traditional OCR often kept a separate recognition model or dictionary per language, but an OCR-free VLM can handle several languages with one model by training on image-text data across languages. Efforts continue to broadly handle not only Latin scripts but also logographic scripts like Chinese-Japanese-Korean and right-to-left scripts. That said, accuracy can still drop on low-resource languages or rare fonts, so data augmentation matched to the target language distribution matters.

Unified Models: Detection-Recognition-Parsing as One

A recent trend is the unified model, which consolidates detection, recognition, layout analysis, and information extraction into a single model. Where the traditional pipeline kept a model per stage, a single encoder-decoder handles them all at once.

traditional pipeline:  detection model -> recognition model -> layout model -> extraction model
unified model:         single model -> (detection + recognition + layout + parsing) simultaneously

Deep Dive 3.5: The Difficulty of Reading Order and Layout

A frequently overlooked difficulty in understanding documents is reading order. A person looking at a multi-column newspaper naturally knows to read the left column top to bottom then move to the right column. But to a model this is not obvious.

multi-column layout example
  +------------------+------------------+
  | col1 first para  | col2 first para  |
  | col1 second para | col2 second para |
  +------------------+------------------+

  correct reading order: all of col1 -> all of col2
  wrong order:           alternate left-right per row (meaning breaks)

The traditional pipeline computed this order explicitly in the layout-analysis stage. An OCR-free model must learn this through training, so if not trained enough on data with diverse layouts, the reading order can scramble. Mix in tables, footnotes, sidebars, and captions and the difficulty rises further.

This problem is partly eased by output-format design. For example, training to output structured by region makes the model more conscious of region boundaries and better at ordering. The key is to verify with validation data whether the model emits a consistent reading order.

Deep Dive 3.6: Concrete Ways to Structure Tables

Tables are one of the trickiest elements in document understanding, because you must preserve row-column relationships, merged cells, and header hierarchy all at once. In what format does an OCR-free model learn to output a table?

table output format choices
  markdown table   simple, fits simple tables, limited merged-cell expression
  HTML table       can express merged cells (rowspan/colspan), verbose
  structured JSON  specifies per-cell coordinates/content, easy to parse, most flexible

Each format has trade-offs. Markdown is human-readable but struggles to express complex merged cells. HTML tables can express merges but the output gets long. Structured JSON is most flexible but you must prepare the answer data consistently in that format.

The key is to pick the format that fits the task and unify the answer data in that format. Since the model imitates the answer format, if the table output format is uneven, the model also produces inconsistent tables. And when evaluating table accuracy, comparing per cell and per structure, not simple string match, captures the real quality.

Deep Dive 4: Benchmarks and Evaluation Methods

How do you evaluate an OCR-free model? Metrics differ by task.

evaluation perspective summary
  what did it read      text recognition accuracy (edit distance)
  what did it extract   key-value precision/recall
  did it keep structure table cell/structure match
  did it understand     document VQA accuracy

The important point is that a single score does not represent all of a model's abilities. One model reads plain text well but is weak on table structure; another is strong in English but drops on multilingual. So building your own evaluation set matched to your actual document distribution is safest of all. Public benchmark scores are only a starting point and do not reflect the noise and diversity of real operational data as is.

Deep Dive 5: Designing the Inference Pipeline

The inference pipeline for putting an OCR-free model into a real service does not end with a single model call. It is better seen as three stages: preprocessing, model inference, and post-processing/verification.

[original document]
   |
   v
[preprocessing]   resolution normalization, rotation correction, denoising
   |
   v
[model inference]  OCR-free model -> generate structured output
   |
   v
[post-processing]  format validation (JSON schema), confidence judgment, fallback
   |
   v
[result]    passes validation -> auto process / fails -> human review

Each stage has a clear role.

The key in this structure is to see the model as one stage of the pipeline, not the endpoint of trust. Designing verification and fallback on the premise that the model can be wrong is what makes it operate stably in real work.

Deep Dive 6: Hard Cases Like Handwriting, Low Quality, Rotation

Real documents are not only clean print. Here is what goes wrong on tricky input.

These hard cases are hard to solve perfectly with the model alone. Preprocessing that raises input quality and post-processing that filters uncertain output must go together to reach an accuracy that holds up in practice.

Comparison Table: Traditional OCR vs OCR-free

ItemTraditional OCR pipelineOCR-free / unified model
Structuremulti-stage detect-recognize-layout-extractsingle-model end to end
Accumulated errorpiles up at each stagegreatly reduced
Context correctionhard (recognition sees only shape)possible (uses language ability)
Tables/formulasneeds separate rules/modelsstructured output via training
Multilingualtends to need per-language model/dictionarymultilingual with one model
Operational complexityhigh (manage several models)low (single model)
Data dependenceper-stage intermediate labelsdepends on final answer format
Output accuracy guaranteeper-stage verification easyverification stage recommended

Deep Dive 9.5: Controlling Output via Prompt Design

A VLM-based OCR-free model produces different output depending on the prompt (instruction) even for the same image. This is powerful flexibility and at the same time a source of variability.

same receipt, different instructions
  "read this receipt"               -> free natural-language summary
  "answer only the total as a number" -> 32500
  "extract items as a JSON array"     -> structured list
  "store name, date, total as key-value" -> clear field mapping

To get stable results in practice, designing prompts concretely and consistently is important.

The prompt is the cheapest knob to adjust output without retraining the model. But accuracy problems that the prompt alone cannot solve must ultimately be handled with data, fine-tuning, and post-processing. The prompt is a starting point, not a universal solution.

Practical Application: Receipts, Forms, Documents

Points to consider when applying an OCR-free approach to real work, organized by task.

In common, rather than trusting model output as is, keeping safeguards like format validation, confidence thresholds, and human review is important in practice. The model can confidently produce wrong values, so the more important the field, the more you should strengthen verification.

Deep Dive 7: Hybrid of OCR-free and Traditional OCR

Since OCR-free is not a cure-all, in practice a hybrid design mixing traditional OCR and OCR-free often appears. The idea is to use the strengths of each.

hybrid design (example)
  1. quickly extract text and positions with traditional OCR
   |
   v
  2. OCR-free/VLM interprets meaning seeing that text and the image together
   |
   v
  3. cross-check the two paths' results to estimate confidence

The lesson of the hybrid is clear. Rather than new technology fully replacing the old, combining the two to cover weaknesses is often more robust in practice.

Deep Dive 8: Getting Structured Output Reliably

When you take structured output (JSON, tables) from an OCR-free model, the model does not always emit a perfect format. You need devices for stable parsing.

structured-output stabilization flow
  [model output string]
   |
   v
  [attempt format parse]  JSON parse, etc.
   |
   +-- success --> [schema validation]  check field presence/type
   |                  |
   |                  +-- pass --> accept result
   |                  +-- fail --> fallback/retry
   |
   +-- fail --> [recovery attempt]  partial extraction or regenerate

Only with such post-processing can you safely connect model output to a real system. Design on the premise that the model occasionally going out of format is normal.

Deep Dive 9: A Cost and Latency View

OCR-free models, especially VLM-based ones, are heavier than traditional OCR. You need a view that manages cost and latency.

cost/latency comparison (intuition)
  traditional OCR    light, fast, cheap
  small OCR-free     medium
  large VLM          heavy, slow, expensive, but flexible/accurate

Chase only accuracy and cost becomes unbearable; chase only cost and quality drops. Balancing between the task's accuracy requirement and budget is the core of practical design.

Deep Dive 12: Additional Challenges of Multilingual Documents

Multilingual documents carry challenges beyond simply recognizing several languages.

stages of multilingual handling
  recognition    various writing systems to text
   |
   v
  structure understanding   grasp layout/tables regardless of language
   |
   v
  normalization  unify region-specific number/date/currency formats

The key is that you must consider not only recognition accuracy but also normalization after recognition to get a usable result in practice. Clarifying the target languages and regions, and preparing data and post-processing rules matched to that distribution, is the starting point of multilingual document handling.

Pitfalls and Limits

Deep Dive 10: Considerations When Applying by Domain

When applying an OCR-free model to a specific industry, each domain has different constraints and demands.

priorities by domain
  finance   accuracy > speed, strong verification
  healthcare accuracy > cost, domain fine-tuning
  legal     structure/order preservation, long-document handling
  logistics handling form diversity, throughput

The common lesson is to tune verification strength in proportion to the domain's cost of error. The more fatal an error is in a domain, the more you must not trust model output as is and stack safeguards thickly.

Deep Dive 11: Directions Ahead

Document-understanding models are advancing fast. Let us note a few currents.

These directions all aim at one thing: maintaining accuracy while reducing human intervention. For now, though, rather than full automation, a design combining the model with verification and human review is most robust in practice.

Closing

Document AI has shifted its center of gravity from a chain of character-reading stages to a single model that takes an image and produces meaning directly. The OCR-free approach reduces accumulated error, corrects context with language ability, and simplifies operations by unifying detection, recognition, and parsing into one.

Of course it is not a cure-all. Limits remain like hallucination, rare domains, and high-resolution cost, and you need verification to guarantee output accuracy. Still, the direction of one model seeing characters and meaning together is a powerful current in document understanding, and its scope of application keeps widening with the advance of VLMs. The key is to understand both the strengths and limits of the tool and use it with appropriate safeguards matched to the task's accuracy requirements.

References

Read the original: https://www.youngju.dev/blog/ai-papers/2026-06-26-ocr-free-document-understanding.en

Comments

No comments yet.

Sign in to leave a comment