LabHub

Blog

Vision LLM Architecture — How an Image Becomes Language

2026-06-26 · 20 min read · #llm#vision-language-model#multimodal#vit#qwen2-vl#architecture

한국어English日本語

Introduction: Why Vision-Language Models

Once text-only LLMs began understanding images, a whole set of domains opened up at once: document analysis, chart reading, UI automation, robotics. But an LLM is fundamentally a model that consumes a token sequence and predicts the next token. How can image data, a two-dimensional grid of pixels, be placed on the same plane as text tokens?

The core idea is simple. Cut the image into small pieces, turn each piece into a vector, then align those vectors into the embedding space the LLM operates in. Once you do that, each image patch behaves like a single word token. In this post we examine that transformation in depth through three components: the vision encoder, the projector, and the LLM decoder.

Understand this structure well and it becomes obvious why some models excel at high-resolution documents while others blow up their token budget, and why arbitrary-resolution handling is such a hard problem.

What Differs From a Text Model

A pure text LLM already has tokens as input. The tokenizer turns a string into a sequence of integer IDs, the embedding table maps each ID to a vector, and that is it. But an image has no tokenizer. There is no table to map pixels directly to integer IDs.

So a vision LLM places a neural network called the vision encoder instead of a tokenizer. This is the core difference. Text gets tokens by lookup table, while an image is passed through a learned encoder to get continuous feature vectors. These vectors play the role of visual tokens.

text path:   string -> tokenizer (lookup) -> token IDs -> embedding -> vector
image path:  pixels -> vision encoder (neural net) -> continuous features -> projector -> vector

The ends of the two paths are the same. Both converge to a sequence of vectors of the same dimension the LLM can handle. Only the starting point differs. Once you hold this view, you see that a vision LLM is in the end a text LLM with one more image input path attached.

Core Principles: A Vision LLM in Three Parts

Most modern vision-language models (VLMs) consist of three components.

  1. Vision Encoder: usually a ViT family model. It takes an image and converts it into a sequence of visual feature vectors.
  2. Vision-Language Projector / Adapter: it maps the vision encoder's output dimension to the LLM's embedding dimension and aligns them semantically. A linear layer, an MLP, or a Q-Former-style compression module is used.
  3. LLM Decoder: it receives text tokens and projected visual tokens as one sequence and predicts the next token.

The overall data flow looks like this.

[image]
   |
   v
[patch split] --> flatten patches and embed
   |
   v
[ViT vision encoder] --> sequence of visual features
   |
   v
[projector / adapter] --> project to LLM dim (+ compression)
   |
   v
[visual tokens] + [text tokens]  --> interleaved into one sequence
   |
   v
[LLM decoder] --> autoregressively generate text

This stack is sometimes trained end to end, and sometimes only the projector and LLM are trained while the vision encoder stays frozen. Training strategy is covered in a separate post; here we focus on how data flows at inference.

Deep Dive 1: From Patches to Tokens

Patch Splitting and Patch Embedding

A ViT begins by dividing the image into fixed-size patches. Split a 224 x 224 image into 14 x 14 pixel patches and you get 16 across and 16 down, 256 patches in total. Each patch is flattened and passed through a linear projection to become a single embedding vector.

Tracking it by tensor shape:

input image:        B x 3 x H x W          (batch, channels, height, width)
after patch split:  B x N_patch x (P*P*3)  (P is patch side length in pixels)
after patch embed:  B x N_patch x D        (D is vision encoder hidden dim)

Here N_patch is (H / P) x (W / P). With a 224 x 224 image and patch size 14, N_patch is 256. At this point the image is already a sequence, and from here it passes through transformer layers exactly like a text token sequence.

Position Information and Transformer Encoding

Flattening patches destroys 2D spatial information, so the ViT adds position embeddings. The sequence then passes through several layers of self-attention and FFN, where each patch absorbs information from its neighbors. The output has the same length as the input, and each position is now a visually enriched feature vector.

The attention inside a ViT is the same scaled dot-product form as in a text transformer.

attention(Q, K, V) = softmax(QK^T / sqrt(d_k)) V

Q, K, V are the query, key, and value obtained by linear projection from patch features, and d_k is the per-head dimension. Multi-head attention learns relationships in several subspaces at once.

Token Count Is Cost

One important intuition at this stage. The number of visual tokens scales with image resolution. Double the resolution and the patch count roughly quadruples, and the sequence the LLM must process grows accordingly. Since LLM attention costs scale quadratically with sequence length, reducing visual token count maps directly to cost. That is exactly why a projector sometimes compresses tokens rather than merely matching dimensions.

Deep Dive 2: The Projector — A Bridge Between Vision and Language

The vision encoder's output dimension may differ from the LLM's embedding dimension, and their semantic spaces are not aligned. The projector closes this gap. There are two main families.

Linear / MLP Projector

The simplest approach. Each visual feature the vision encoder emits is projected into the LLM dimension by a linear layer or a two-layer MLP. The token count stays the same.

vision encoder out:   B x N_v x D_vis
after linear/MLP:      B x N_v x D_llm   (token count N_v preserved, only dim D_vis -> D_llm)

The LLaVA family popularized this simple MLP approach. It is easy to implement and performs well, so it is widely used.

Q-Former-Style Learnable Query Compression

The other family places learnable query vectors and uses cross-attention to pull information out of the visual features into a fixed number of tokens. BLIP-2's Q-Former is the canonical example.

learnable queries:    B x N_q x D        (N_q fixed, e.g. 32, independent of patch count)
vision features (K/V): B x N_v x D_vis
cross-attention:       queries absorb information from vision features
output:                B x N_q x D_llm    (compressed N_v -> N_q)

The compressing family suits tasks where the big picture matters, such as natural-image captioning, while the non-compressing (MLP) family tends to be better for document work that requires reading small text or tables.

Deep Dive 3: Injecting Visual Tokens Into the LLM

Visual tokens that have passed through the projector now live in the LLM's embedding space. Weaving them into one sequence with text is called interleaving.

The Basic Form of Interleaving

In the chat input, a special placeholder token marks where an image goes, and just before feeding the LLM that slot is replaced by the visual tokens.

original sequence:   [text...] [IMG_PLACEHOLDER] [text...]
after substitution:  [text...] [v1][v2]...[vN] [text...]

The resulting unified sequence is, from the LLM's view, just one long embedding sequence. Self-attention sees text tokens and visual tokens together without distinction, learning to let the question text point at a specific region of the image.

Multiple images and interleaved image-text documents follow the same principle: insert a visual token block in the order images appear.

multi-image:  [text] [img1 tokens] [text] [img2 tokens] [text]

Causal Mask and Visual Tokens

The LLM decoder uses a causal mask so each position sees only tokens before it. Visual tokens follow this rule too. Text that comes after an image can reference the image tokens, but the image tokens themselves usually see only their preceding context. Some models allow bidirectional attention among image tokens so that within one image all patches can see each other. This is a per-implementation design choice.

Deep Dive 4: Arbitrary-Resolution Handling

Early VLMs force-resized every input to a fixed size (e.g., 336 x 336). That smears wide documents or small text and loses information. Recent models try to handle arbitrary resolution as is.

Qwen2-VL's Naive Dynamic Resolution

Qwen2-VL does not force a fixed size; instead it dynamically generates a number of visual tokens proportional to the original resolution. Larger images become more tokens, smaller images fewer. As a result aspect ratio and fine detail are preserved, which helps document and chart understanding.

small image (e.g. 448x448):   few visual tokens
large image (e.g. 1568x1568): many visual tokens
wide document:                original ratio kept, more patches along the width

In practice you cap the minimum and maximum token counts so an overly large image does not explode the context.

M-RoPE: Multimodal Rotary Position Embedding

Position encoding is also a problem. Text has a 1D order while an image is a 2D grid. Qwen2-VL introduces M-RoPE (Multimodal Rotary Position Embedding), decomposing position into several components: time, height, width. Text tokens share the same value across these components, while image tokens have height and width components that reflect 2D coordinates.

text token:   (t, t, t)        three components share one position index
image token:  (t, h, w)        height/width components reflect grid coordinates

This lets a single model represent 1D text and 2D image positions consistently, and extends naturally to cases like video where a time axis is added.

Comparison Table: Projector Design Choices

ItemLinear / MLP ProjectorQ-Former-style Compression
Token countpreserves patch countcompresses to a fixed small number
Visual fidelityhigh, near losslesssome loss during compression
LLM sequence lengthlong, costlyshort, cost saving
Document/OCR fitfavorablerelatively weaker
Natural-image captioningsuitablesuitable
Implementation complexitylowhigh
Representative caseLLaVA familyBLIP-2 Q-Former
ItemFixed ResolutionArbitrary (Dynamic) Resolution
Input handlingresize to fixed sizepreserves original ratio/resolution
Small text/detailrisk of losswell preserved
Token countconstantscales with image size, variable
Document/chart understandingweakstrong
Context costpredictableneeds cap management

Deep Dive 5: Tracing the Token Budget With Numbers

Abstract explanations do not build a sense of cost, so let us trace concrete numbers. Assume patch size 14 and vision encoder hidden dim 1024.

case A: ordinary photo 896 x 896
  patches across = 896 / 14 = 64
  patches down   = 896 / 14 = 64
  visual tokens  = 64 x 64 = 4096

case B: wide document 1568 x 784
  patches across = 1568 / 14 = 112
  patches down   = 784 / 14 = 56
  visual tokens  = 112 x 56 = 6272

case C: thumbnail 224 x 224
  patches across/down = 16 x 16
  visual tokens       = 256

One thing becomes clear. With the same model, visual tokens swing from 256 to over 6000 depending on input image size. Put several images in a multi-turn conversation and visual tokens may take most of the context. That is why many models place downsampling such as 2 x 2 patch merging after the vision encoder, cutting the tokens passed to the LLM to one quarter.

patch merge (2x2) example
  vision encoder output:   64 x 64 = 4096 tokens
  after 2x2 merge:         32 x 32 = 1024 tokens  (merge four into one)

This downsampling sacrifices a little detail but greatly reduces the LLM sequence. When small text matters as in documents, lower the merge strength; when the big picture matters as in ordinary photos, cut more aggressively, tuning to the task.

Deep Dive 6: Choosing the ViT Encoder and Its Pretraining

The vision encoder is not just any ViT; usually you bring an encoder pretrained with large-scale image-text contrastive learning. An encoder trained paired with text already emits visual representations somewhat aligned with language, so the burden on the following projector drops.

The encoder's quality somewhat determines the ceiling of the whole VLM's performance. If the encoder cannot distinguish small text, no matter how smart the following LLM is, it cannot recover that information. So if you target documents and OCR, pay special attention to the resolution and expressiveness at the encoder stage.

Deep Dive 7: Extending to Multiple Images and Video

The principle of processing a single image extends directly to multiple images and video.

video processing flow
  [frame1][frame2]...[frameT]  --> tokenize each
   |
   v
  assign time component t=1,2,...,T  --> preserve frame order
   |
   v
  [question text] + [frame tokens]  --> LLM decoder

Video tokens explode in proportion to frame count, so the key is managing the token budget by jointly adjusting the frame sampling interval and per-frame resolution. The longer the video the sparser you sample frames; for short, detail-critical video you sample densely, a trade-off you tune.

Practical View: What to Watch at Inference

The first thing you hit when serving a vision LLM is visual token cost. A single high-resolution document can take thousands of tokens, so context fills faster than with text alone.

Visual tokens occupy the KV cache along with LLM decoding. So in multi-turn conversations that repeatedly reference the same image, the cache accumulates and memory pressure grows. Prefix cache reuse or image-embedding caching can ease this.

Deep Dive 11: Additional Considerations From a Serving View

When actually serving a vision LLM, a few details beyond text-LLM serving are added.

vision LLM serving pipeline
  [request: image + text]
   |
   v
  [image preprocessing]  (can be a separate worker)
   |
   v
  [vision encoder + projector]  generate visual tokens in prefill
   |
   v
  [LLM prefill + decode]  bundle with continuous batching
   |
   v
  [response streaming]

The key is that adding the image path makes prefill cost and preprocessing cost larger than text-only. Image size caps, preprocessing separation, and caching are the main knobs of serving efficiency.

Pitfalls and Troubleshooting

Deep Dive 7.5: The Details of Image Preprocessing

Half of visual token quality is decided in the preprocessing before it enters the model. It is often overlooked but greatly affects the result.

preprocessing checklist
  [ ] same resize rule as training
  [ ] same normalization statistics as training (mean/std)
  [ ] matching channel order (RGB)
  [ ] size divisible by patch size
  [ ] matching aspect-ratio handling

If any one of this checklist is off, the model produces worse results than at training even with the same weights. The maxim to suspect preprocessing before the model when debugging is especially valid for vision LLMs.

Deep Dive 8: Fusion Method — Interleaving vs Cross-Attention

So far we centered on interleaving (or decoder fusion), where visual tokens are inserted into the same sequence as text. But that is not the only way to mix vision and language. There are two main families.

method A: interleaving (decoder fusion)
  insert visual tokens into the same input sequence as text tokens
  the LLM's self-attention processes both together
  representative: LLaVA-like, Qwen2-VL-like

method B: cross-attention fusion
  insert cross-attention blocks between LLM layers
  text via self-attention, image info injected via cross-attention
  representative: some gated cross-attention structures

Many recent open models tend to choose interleaving for its simplicity and power. But for high-resolution and video where context cost is large, the advantage of cross-attention-style injection is sometimes revisited. There is no single right answer; it depends on the task's token budget and implementation constraints.

Deep Dive 9: Clearing Up Commonly Confused Concepts

Clarifying these concepts lets you quickly grasp which design choices a new VLM's technical report made.

Deep Dive 9.5: A Decision Guide for Model Selection

Let us connect the design axes seen so far to actual model selection. Priorities change with task nature.

recommended direction by task
  receipt/document OCR  -> high resolution + non-compressing (MLP) projector + arbitrary resolution
  general image captioning -> medium resolution + compression allowed, cost efficiency first
  chart/table analysis  -> high resolution + model trained for structured output
  multi-image reasoning -> emphasize token efficiency (compression or patch merge)
  video understanding   -> frame sampling + time position encoding support

This guide is only a starting point; in practice, directly evaluating a few candidate models on your own data is most certain. Benchmark scores show average tendencies only; how it behaves on my documents and my images must be measured directly.

Ask yourself three core questions. First, do small text or fine details matter? If so, prioritize resolution and the non-compressing path. Second, how many images go into one request? If many, token efficiency governs cost. Third, is the output free text or a structured format? If structured is needed, you must pick a model trained for that format.

Deep Dive 10: Redrawing the Whole Picture

Finally, let us bundle the parts so far back into one big picture.

[original image]
   |  preprocessing: resize/normalize (same as training)
   v
[patch split + patch embed]      B x N_patch x D_vis
   |  add position embedding
   v
[ViT vision encoder: self-attention x L layers]   B x N_patch x D_vis
   |  (optional) downsample with 2x2 patch merge
   v
[projector: MLP or Q-Former]   B x N_v x D_llm
   |  insert at placeholder positions
   v
[unified sequence: text + visual tokens]   interleaving
   |  assign positions via M-RoPE etc., causal mask
   v
[LLM decoder: self-attention x M layers]
   |  next-token prediction
   v
[output: text / coordinates / structured]

This single flow is the whole of a vision LLM. Which design you choose at each stage determines the model's character. Encoder resolution, whether the projector compresses, position-encoding method, fusion strategy: turning these four knobs produces countless variants.

Closing

The heart of a vision LLM is, in the end, the transformation that turns an image into tokens. The ViT makes the image a patch sequence, the projector aligns it into the LLM's language space, and interleaving weaves text and visual tokens into one flow. Add devices like arbitrary resolution and M-RoPE, and the model can flexibly handle everything from dense-text documents to wide charts.

Design choices are always trade-offs. Compress tokens and it gets cheaper but loses detail; raise resolution and it reads better but costs more. Knowing what your task values most makes clear which model structure to pick. The next post covers how to train this structure, and how to teach its inputs and outputs.

References

Read the original: https://www.youngju.dev/blog/llm/2026-06-26-vision-language-model-architecture.en

Comments

No comments yet.

Sign in to leave a comment