LabHub

Blog

Wan Text-to-Video/Image-to-Video and Z Image Turbo Complete Analysis: Architecture and Applications of Next-Gen Video/Image Generation Models

2026-03-01 · 36 min read · #wan#text-to-video#image-to-video#z-image-turbo#video-generation#image-generation#diffusion#deep-learning#generative-ai#alibaba

한국어English日本語

1. Introduction: A New Horizon for Video and Image Generation AI

2025 was the year video generation AI grew explosively. Amid a flood of commercial services such as OpenAI Sora, Google Veo, Runway Gen-3, and Kling, the Wan series that Alibaba released as open source proved that "open source can compete with commercial models too." At the same time, in image generation, Z-Image Turbo matched the quality of 20B-80B class models with 6B parameters in just 8 steps, setting a new bar for efficiency.

[Video/Image Generation AI Timeline 2024-2026]

2024 Q1-Q2          2024 Q3-Q4          2025 Q1-Q2          2025 Q3-Q4          2026 Q1
    |                   |                   |                   |                   |
    v                   v                   v                   v                   v
 Sora Preview       HunyuanVideo        Wan 2.1             Wan 2.2 (MoE)       Wan 2.6
 Runway Gen-3       CogVideoX           Z-Image             Wan 2.5-Preview     Veo 3.1
 Pika 1.0           Kling 1.0           Z-Image Turbo       Kling 2.6           Seedance 1.5
                                                             Sora 2

This article analyzes the full architecture and training methodology of the Wan series in depth, dissects the ultra-fast image generation technology of Z-Image Turbo, and then covers comparisons with competing models and a practical usage guide.

2. Introducing the Wan Models

2.1 Background: Alibaba and the Wan Team

Wan is a series of large-scale video generation models developed by Alibaba Cloud's Wan-AI team. The name "Wan" means "ten thousand" (萬) in Chinese and carries the vision of opening up ten thousand possibilities. It was developed in close cooperation with Alibaba's Tongyi research lab, and it is also part of Alibaba's strategic investment in the open source ecosystem.

Key milestones:

VersionRelease dateMain features
Wan 2.1February 2025First open source release, T2V/I2V support
Wan 2.2July 2025MoE architecture introduced, 27B parameters
Wan 2.5-PreviewSeptember 2025Native Multimodality (text+image+video+audio)
Wan 2.6December 2025Reference-to-Video, multi-shot narrative

2.2 Text-to-Video and Image-to-Video Capabilities

Wan supports two core modalities:

Text-to-Video (T2V): Generates high-quality video from a text prompt alone. Through a multilingual T5 encoder it can handle prompts in several languages, including English and Chinese, and Classifier-Free Guidance controls how faithfully it follows the prompt.

Image-to-Video (I2V): Takes a single image as input and generates natural video with that image as the first frame. The key is adding temporal motion while preserving the image's visual characteristics (composition, color, objects).

[Wan model pipeline overview]

Text-to-Video:
  text prompt ──> T5 Encoder ──> Cross-Attention ──> DiT Backbone ──> 3D VAE Decoder ──> video

Image-to-Video:
  input image ──> VAE Encoder ──┐
                                ├──> DiT Backbone ──> 3D VAE Decoder ──> video
  text prompt ──> T5 Encoder ──┘

2.3 The State of the Open Source Release

Wan's biggest differentiator is that it is fully open source. It is published under the Apache-2.0 license, so commercial use is unrestricted.

Published resources:

3. Analyzing the Wan Architecture

Wan's architecture builds on the Diffusion Transformer (DiT) paradigm and includes several innovative components optimized for video generation.

3.1 3D Causal VAE (Wan-VAE)

Wan-VAE is the core component that compresses video data efficiently, compressing the spatial and temporal dimensions at the same time.

[Wan-VAE compression structure]

Input video: [T, H, W, 3]     (e.g. 81 frames, 720, 1280, RGB)
                |
                v
        3D Causal VAE Encoder
                |
                v
Latent Space: [T/4, H/8, W/8, 16]  (compression ratio: 4x8x8, latent dim=16)
                |
                v
        3D Causal VAE Decoder
                |
                v
Reconstructed video: [T, H, W, 3]

Core design principles:

  1. Causal Convolution: Causal convolution is used along the time axis so that information from future frames does not influence the encoding of past frames. This makes streaming encoding of infinitely long video possible.

  2. 4x8x8 compression ratio: 4x compression on the time axis and 8x8 on the spatial axes. The latent dimension is 16, which sets it apart from designs like Open-Sora Plan (latent dim=4) or SVD (compression ratio 1x8x8, latent dim=4).

  3. 3-stage training strategy:

    • Stage 1: Train a 2D Image VAE of identical structure on image data to secure spatial compression ability
    • Stage 2: Inflate the trained 2D VAE into a 3D Causal VAE to provide a temporal compression prior
    • Stage 3: Train initially on low-resolution (128x128), short-frame (5 frame) video, then progressively expand resolution and frame count

  4. Temporal Chunk Processing: The number of frames in each processing chunk is capped at 4, following the time-axis compression ratio, to prevent memory overflow.

Performance characteristics:

3.2 Diffusion Transformer (DiT) Backbone

Wan's core generative model adopts the Diffusion Transformer (DiT) architecture. Unlike traditional U-Net-based diffusion models, it exploits the scalability of the Transformer.

[Wan DiT Backbone structure]

                    Time Embedding
                         |
                    MLP (Linear + SiLU)
                         |
                    6 Modulation Parameters
                         |
            ┌────────────┼────────────┐
            v            v            v
    ┌──────────────────────────────────────┐
    │         Transformer Block            │
    │                                      │
    │  ┌─────────────────────────────────┐ │
    │  │  Spatio-Temporal Self-Attention  │ │
    │  │  (Multi-Head)                   │ │
    │  └─────────────────────────────────┘ │
    │              |                       │
    │  ┌─────────────────────────────────┐ │
    │  │  Cross-Attention                │ │
    │  │  (Text Conditioning)            │ │
    │  └─────────────────────────────────┘ │
    │              |                       │
    │  ┌─────────────────────────────────┐ │
    │  │  Feed-Forward Network           │ │
    │  └─────────────────────────────────┘ │
    └──────────────────────────────────────┘
                    |
              (x N blocks)
                    |
                Output Prediction

Main characteristics:

  1. Spatio-Temporal Self-Attention: Multi-head self-attention that handles space and time together is applied to the patchified video latent representation. This lets the model learn temporal consistency across frames and spatial consistency within a frame at the same time.

  2. Modulation mechanism: The timestep embedding is processed by an MLP (Linear + SiLU) to predict 6 modulation parameters. This MLP is shared across every transformer block, and each block learns its own set of biases.

  3. Cross-Attention for Text Conditioning: The text embedding from the T5 encoder is injected into each transformer block via cross-attention. This way the semantic content of the text prompt influences the whole generation process.

3.3 Text Encoder (T5/CLIP Integration)

Wan uses a multilingual T5 encoder as its text encoder. This offers several advantages over models that use CLIP alone:

AttributeT5 (Wan)CLIP (earlier models)
Text comprehensionDeep semantic understandingFocused on image-text alignment
Multilingual supportExcellent (trained multilingually)English-centric
Long prompt handlingExcellentLimited (77 tokens)
Complex relational reasoningPossibleLimited
Parameter freezingFrozenFrozen

The T5 encoder's parameters are frozen during training, and it supplies prompt embeddings to the DiT backbone through cross-attention. Adopting a multilingual T5 means prompts in many languages, not just English but Chinese, Korean, and others, are handled naturally.

3.4 Flow Matching Training

Wan 2.1 adopts the Flow Matching framework, greatly improving training efficiency over traditional DDPM.

[Flow Matching vs DDPM]

DDPM (Denoising Diffusion Probabilistic Models):
  x_0 ──(many discrete steps)──> x_T (Gaussian noise)
  training: predict the noise at each step
  problem: needs hundreds to thousands of steps, complex noise schedule

Flow Matching:
  x_0 ──(straight path)──> x_1 (Gaussian noise)
  training: predict the velocity field v(x_t, t)
  advantage: a straight path yields high quality in fewer steps

The core principle of Flow Matching:

Flow Matching learns a straight path between the data distribution and the noise distribution. Where DDPM removes noise gradually through a complex noise schedule, Flow Matching learns a more efficient path on an ODE (Ordinary Differential Equation) basis.

The core training objective:

L_FM = E_{t, x_0, x_1} [ || v_theta(x_t, t) - (x_1 - x_0) ||^2 ]

where:
  x_t = (1-t) * x_0 + t * x_1   (straight-line interpolation)
  v_theta: the learnable velocity field
  t ~ U(0, 1): the time parameter

Thanks to this approach, Wan can generate high-quality video with fewer inference steps than traditional diffusion models.

3.5 Motion Module / Temporal Attention

One of the most important factors in video generation is temporal consistency. Wan relies on spatio-temporal self-attention as its main tool for this.

[How Spatio-Temporal Attention works]

Frame 1    Frame 2    Frame 3    Frame 4
  |           |           |           |
  v           v           v           v
[patchify] --> [patchify] --> [patchify] --> [patchify]
  |           |           |           |
  └───────────┴───────────┴───────────┘
                    |
         Spatio-Temporal Self-Attention
         (attention across every patch of every frame)
                    |
  ┌───────────┬───────────┬───────────┐
  |           |           |           |
Frame 1    Frame 2    Frame 3    Frame 4

Because every patch of every frame attends to every other, an object's movement in one frame connects naturally to the next. This produces far more natural motion than generating each frame independently and then trying to align temporal consistency afterwards.

3.6 Resolution and Duration Capabilities

Wan's resolution and duration support:

ModelResolutionFramesFPSVideo length
Wan 2.1 T2V-1.3B832x480 (480P)~8116~5s
Wan 2.1 T2V-14B1280x720 (720P)~8116~5s
Wan 2.1 I2V-14B1280x720 (720P)~8116~5s
Wan 2.2 TI2V-5B1280x720 (720P)~9724~4s
Wan 2.2 A14B (MoE)1280x720 (720P)~9724~4s

In Wan 2.2 the FPS went from 16 to 24, which smooths out the slight choppiness of the earlier version.

4. Wan Model Variants

4.1 Wan 2.1: 1.3B (Lightweight Model)

Wan 2.1's 1.3B model is a lightweight model that can run on a consumer-grade GPU.

Main characteristics:

4.2 Wan 2.1: 14B (Large Model)

The 14B model is Wan 2.1's full-scale model, and it supports 720P reliably.

Main characteristics:

4.3 Wan 2.2: MoE Architecture (A14B)

Wan 2.2 is a landmark version that brings the Mixture-of-Experts (MoE) architecture, already proven in large language models (LLMs), into a video generation diffusion model.

[Wan 2.2 MoE architecture]

Denoising Process Timeline:
t=T (pure noise) ─────────────────────────────> t=0 (clean video)

            High-Noise Phase          Low-Noise Phase
           ┌──────────────┐         ┌──────────────┐
           │  Expert 1    │         │  Expert 2    │
           │  (14B)       │         │  (14B)       │
           │              │         │              │
           │  overall     │  SNR    │  detail      │
           │  layout      │  switch │  refinement  │
           │  & structure │  point  │  & texture   │
           └──────────────┘         └──────────────┘

Total parameters: ~27B
Active parameters at inference: ~14B (only one expert active per step)

Core design:

  1. Two-Expert Design: It uses 2 expert models, each specialized for a part of the diffusion model's denoising process.

    • High-Noise Expert: activates in the early denoising stages and decides the overall layout and structure
    • Low-Noise Expert: activates in the later denoising stages and refines the video's details

  2. SNR-based switching: The switch point between the two experts is decided by the Signal-to-Noise Ratio (SNR). SNR decreases monotonically as the denoising step increases, and that value is used to pick the appropriate expert.

  3. Efficient inference: There are 27B parameters in total, but only 14B activate at each inference step, so GPU memory and compute cost stay at the level of a single 14B model.

4.4 Wan 2.2: TI2V-5B (Dense Model)

Alongside the MoE model, a 5B Dense model was released as well.

Main characteristics:

Wan2.2-VAE reaches a much better compression ratio (16x16x4) than the earlier Wan-VAE, so it can represent video of the same quality with fewer latent tokens.

4.5 Differences Between Text-to-Video and Image-to-Video

AttributeText-to-Video (T2V)Image-to-Video (I2V)
InputText promptImage + text prompt
First frameGenerated freely by the modelMust match the input image
ConditioningT5 text embeddingT5 embedding + VAE image latent
Creative freedomHighConstrained by the image
Main usesPure creation, storyboardingPhoto animation, product demos
1.3B supportOX
14B supportOO

5. Training Data and Methodology

5.1 Training Datasets

Wan 2.1 was trained on large-scale image-text and video-text pair datasets. Wan 2.2 achieved a sweeping data expansion over Wan 2.1: a 65.6% increase in image data and an 83.2% increase in video data.

The 4-stage data quality management pipeline:

[Wan data refinement pipeline]

Raw data (web crawl)
       |
       v
┌──────────────────────┐
│ Stage 1: Visual Fidelity    │
│ - resolution/quality filter │
│ - remove watermarks/overlays│
│ - detect compression artifacts│
└──────────────────────┘
       |
       v
┌──────────────────────┐
│ Stage 2: Motion Smoothness  │
│ - frame consistency check   │
│ - filter unstable camera work│
│ - detect abrupt scene cuts  │
└──────────────────────┘
       |
       v
┌──────────────────────┐
│ Stage 3: Diversity          │
│ - remove semantic duplicates│
│ - balance across categories │
│ - secure varied motion/scenes│
└──────────────────────┘
       |
       v
┌──────────────────────┐
│ Stage 4: Frame Integrity    │
│ - detect dropped frames     │
│ - final temporal consistency check│
│ - verify caption-video match│
└──────────────────────┘
       |
       v
Final training dataset

5.2 Progressive Training Strategy

Wan adopts a Progressive Training strategy to achieve efficient and stable training.

3-stage Progressive Training:

  1. Stage 1 - Image Pre-training:

    • Learn spatial generation ability from large-scale image-text data
    • Secure text-image alignment
    • Resolution: start at 256x256 and increase progressively

  2. Stage 2 - Low-Resolution Video Training:

    • Learn temporal motion from low-resolution (256x256 to 480P) video
    • Learn basic motion patterns from short clips (16-33 frames)
    • Add temporal modeling while preserving image generation ability

  3. Stage 3 - High-Resolution Fine-tuning:

    • Final fine-tuning on high-resolution (720P) video
    • Extend to long sequences (81-97 frames)
    • Maximize quality using mostly high-quality curated data
[Progressive Training flow]

Stage 1          Stage 2               Stage 3
Image 256x256 -> Video 480P 16fps -> Video 720P 24fps
text-image       temporal motion       high-quality
alignment        short clips           fine-tuning, long sequences

5.3 Quality Filtering Pipeline

The quality of the training data is the decisive factor in a generative model's performance. Wan developed automated quality evaluation metrics and applied them to data filtering.

Automated evaluation metrics:

5.4 Classifier-Free Guidance

Wan uses Classifier-Free Guidance (CFG) to control how faithfully the output follows the text prompt.

CFG formula:
  output = uncond_output + guidance_scale * (cond_output - uncond_output)

where:
  cond_output: text-conditioned output
  uncond_output: unconditional output (empty prompt)
  guidance_scale: the CFG scale (higher means more faithful to the prompt)

During training the text condition is dropped with some probability, so the model learns both unconditional and conditional generation. At inference the guidance_scale parameter tunes the balance between prompt fidelity and output diversity.

6. A Practical Guide to Using Wan

6.1 Installation

Installing with pip

# Requires Python 3.10+ and PyTorch 2.4.0+
pip install torch torchvision torchaudio

# Clone the Wan 2.1 repository
git clone https://github.com/Wan-Video/Wan2.1.git
cd Wan2.1

# Install dependencies
pip install -r requirements.txt

# Download the model (Hugging Face)
pip install "huggingface_hub[cli]"
huggingface-cli download Wan-AI/Wan2.1-T2V-14B --local-dir ./Wan2.1-T2V-14B

Installing Wan 2.2

# Clone the Wan 2.2 repository
git clone https://github.com/Wan-Video/Wan2.2.git
cd Wan2.2

# Install dependencies
pip install -r requirements.txt

# Download the Wan 2.2 MoE model
huggingface-cli download Wan-AI/Wan2.2-T2V-A14B --local-dir ./Wan2.2-T2V-A14B

6.2 Basic Inference Code

Text-to-Video (1.3B - lightweight)

python generate.py \
    --task t2v-1.3B \
    --size 832*480 \
    --ckpt_dir ./Wan2.1-T2V-1.3B \
    --prompt "A futuristic city with flying cars and neon lights at dusk."

Text-to-Video (14B - high quality)

python generate.py \
    --task t2v-14B \
    --size 1280*720 \
    --ckpt_dir ./Wan2.1-T2V-14B \
    --prompt "Two anthropomorphic cats in comfy boxing gear and bright gloves \
    fight intensely on a spotlighted stage."

Image-to-Video (14B)

python generate.py \
    --task i2v-14B \
    --size 1280*720 \
    --ckpt_dir ./Wan2.1-I2V-14B-720P \
    --image input_image.jpg \
    --prompt "The camera slowly zooms in as the subject begins to smile."

Memory Optimization Options

# Model offloading (saves VRAM)
python generate.py \
    --task t2v-14B \
    --size 1280*720 \
    --ckpt_dir ./Wan2.1-T2V-14B \
    --offload_model True \
    --t5_cpu \
    --prompt "A serene lake surrounded by mountains at sunrise."

Multi-GPU Acceleration

# 8-GPU parallel inference
torchrun --nproc_per_node=8 generate.py \
    --task t2v-14B \
    --size 1280*720 \
    --ckpt_dir ./Wan2.1-T2V-14B \
    --dit_fsdp \
    --t5_fsdp \
    --ulysses_size 8 \
    --prompt "A timelapse of a flower blooming in a garden."

6.3 ComfyUI Integration

ComfyUI is the most popular frontend interface for Wan.

Setting up Wan 2.2 in ComfyUI:

  1. Update ComfyUI to the latest version
  2. Load the "Wan2.2 5B video generation" template from Workflow -> Browse Templates -> Video
  3. Or drag and drop the official JSON workflow file onto the canvas

GGUF quantized models (low-spec GPUs):

Using a quantized model in GGUF format, Wan 2.2 will run on 12GB of VRAM or less:

Additional acceleration techniques:

6.4 A Guide to Writing Prompts

Prompt strategies for getting the best results out of Wan:

Basic structure:

[subject/action] + [style/mood] + [camera work] + [lighting/environment]

An example of an effective prompt:

"A young woman walks through a bustling Tokyo street at night.
 Neon signs reflect on wet pavement. Cinematic lighting.
 The camera follows her from behind in a smooth tracking shot.
 High detail, photorealistic, 4K quality."

Prompt enhancement:

Wan 2.1 offers automatic prompt enhancement:

Tips:

6.5 GPU Requirements and Memory Optimization

ModelMin VRAMRec. VRAMGeneration time (5s video)
Wan 2.1 T2V-1.3B (480P)8GB12GB~4 min (RTX 4090)
Wan 2.1 T2V-14B (720P)24GB32GB+~8 min (RTX 4090)
Wan 2.2 TI2V-5B (720P)12GB16GB~9 min (consumer GPU)
Wan 2.2 A14B MoE (720P)24GB32GB+~10 min (RTX 4090)
Wan 2.2 14B GGUF (480P)6GB12GB~15 min (RTX 4050)

Memory optimization strategies:

  1. Model Offloading: move unused model components to CPU memory with --offload_model True
  2. T5 CPU Inference: run the T5 encoder on CPU with --t5_cpu to save GPU memory
  3. GGUF quantization: a large cut in memory usage compared with FP16/BF16
  4. FSDP (Fully Sharded Data Parallel): shard model parameters across a Multi-GPU setup
  5. Flash Attention: improves the memory efficiency of the attention computation

7. Comparison with Competing Models

7.1 Wan vs Sora (OpenAI)

Sora 2 reaches the current state of the art in physics simulation and object persistence. It can generate photorealistic video up to 1 minute long, and it renders physical phenomena such as falling objects, collisions, and reflections naturally.

ItemWan 2.2/2.6Sora 2
Open sourceApache-2.0Closed
Max resolution720P (1280x720)1080P
Max length~5s (local), longer via the API~60s
Physics simulationExcellentBest in class
Local executionYesNo
PriceFree (local)Paid subscription
CustomizationYes (fine-tuning)No

7.2 Wan vs Kling (Kuaishou)

Kling, developed by ByteDance/Kuaishou, is especially strong on character motion and lip sync.

ItemWan 2.2/2.6Kling 2.6
Open sourceApache-2.0Closed
Motion qualityExcellentBest in class (people especially)
Max length~5s (local)~120s
Lip syncLimitedExcellent
Camera motionExcellentExcellent
AccessibilityLocal/APIAPI only

7.3 Wan vs Runway Gen-3

Runway Gen-3 Turbo is a commercial service with strengths in cinematic quality and camera work.

ItemWan 2.2/2.6Runway Gen-3
Open sourceApache-2.0Closed
Cinematic qualityExcellentBest in class
Motion SynthesisExcellentExcellent
Resolution720P1080P
PriceFree (local)Paid (credit-based)

7.4 Wan vs Pika

Pika is a service focused on ease of use and fast generation.

ItemWan 2.2/2.6Pika
Open sourceApache-2.0Closed
Ease of useMedium (technical knowledge needed)Best in class (web UI)
Generation speedMinutes (local)Seconds (cloud)
CustomizationYesLimited
Style varietyExcellentExcellent

7.5 Wan vs CogVideoX (Tsinghua/ZhipuAI)

CogVideoX is an open source video generation model developed by Tsinghua University and ZhipuAI.

ItemWan 2.2CogVideoX-5B
Open sourceApache-2.0Apache-2.0
Parameters27B (MoE)5B
Resolution720P (1280x720)480P (720x480)
Max length~5s~6s
FPS248
I2V qualityExcellentBest in class (I2V-focused)
VRAM24GB+8-12GB

7.6 Wan vs HunyuanVideo (Tencent)

HunyuanVideo is a 13B parameter video generation model developed by Tencent.

ItemWan 2.2HunyuanVideo
Open sourceApache-2.0Open source
Parameters27B (MoE) / 5B13B
Resolution720P720P (1280x720)
Max length~5s~15s
FPS2424
Cinematic qualityExcellentBest in class (cinematic-focused)
GPU requirement24GB+A100/H100 recommended

7.7 Overall Comparison Table

ModelResolutionMax lengthQualityOpen sourceAccessStrengths
Wan 2.6720P~5s+HighO (Apache-2.0)Local/APIMulti-shot narrative, R2V
Sora 21080P~60sBest in classXAPIPhysics simulation
Kling 2.61080P~120sBest in classXAPIMotion/lip sync
Runway Gen-31080P~10sBest in classXAPICinematic quality
Pika1080P~4sHighXWebEase of use
CogVideoX480P~6sUpper-middleO (Apache-2.0)LocalI2V on low-spec hardware
HunyuanVideo720P~15sBest in classOLocalCinematic quality
Veo 3.11080P~25sBest in classXAPIAudio integration
Seedance 1.51080P~10sHighXAPIMultilingual/fast iteration

The key takeaway: Wan stands alone on the combination of "open source + commercial freedom + runs locally." It trails commercial models on resolution and length, but it holds an overwhelming edge on customizability and cost efficiency.

8. Introducing Z-Image Turbo

8.1 An Ultra-Fast Image Generation Model

Z-Image Turbo is an ultra-fast text-to-image generation model developed by Alibaba's Tongyi-MAI (Machine AI) lab. Released on November 26, 2025, it matches the quality of 20B-80B class competitors with 6B parameters while running inference several times faster.

Key results:

[Z-Image Turbo positioning]

                   quality
                     ^
                     |   * Flux Pro (12B)
                     |           * DALL-E 3
                     |     * Z-Image Turbo (6B)  <-- here!
                     |   * SDXL (2.6B)
                     |
                     | * SD 1.5 (0.9B)
                     |
                     +────────────────────> speed
                     slow                    fast

Z-Image Turbo: high quality from few parameters + fast

8.2 What the Turbo/Distillation Technique Means

The "Turbo" name denotes a model that, through knowledge distillation, preserves the original model's quality while cutting the number of inference steps drastically.

[The Teacher-Student Distillation idea]

Teacher Model (Z-Image): 100 step denoising
       |
       | knowledge distillation
       v
Student Model (Z-Image Turbo): 8 step denoising

Result: the quality of 100 steps achieved in 8 steps
      -> about a 12.5x speedup

Where traditional diffusion models need 50-100 denoising steps, Z-Image Turbo reaches equivalent quality in just 8 NFE (Number of Function Evaluations).

9. Z-Image Architecture and Technical Analysis

9.1 S3-DiT (Scalable Single-Stream Multi-Modal Diffusion Transformer)

Z-Image's core architecture is S3-DiT, an efficient design that challenges the "scaling up means costing more" paradigm.

[S3-DiT architecture in detail]

Input streams:
  Qwen3-4B text tokens   ──┐
  FLUX VAE image tokens  ──┤──> Single-Stream Fusion ──> Unified Sequence
  SigLIP 2 semantic tokens ──┘

Unified Sequence
       |
       v
┌──────────────────────────────────────────┐
│          S3-DiT Transformer Layer         │
│                                          │
│  ┌────────────────────────────────────┐  │
│  │  Modality-Specific Pre-Processing  │  │
│  │  (lightweight per-modality proc.)  │  │
│  └────────────────────────────────────┘  │
│              |                           │
│  ┌────────────────────────────────────┐  │
│  │  Single-Stream Self-Attention      │  │
│  │  (QK-Norm + Sandwich-Norm)        │  │
│  │  - 32 attention heads              │  │
│  │  - hidden dim: 3840               │  │
│  └────────────────────────────────────┘  │
│              |                           │
│  ┌────────────────────────────────────┐  │
│  │  Conditional Injection             │  │
│  │  (Scale + Gate parameters)        │  │
│  │  - Shared low-rank down-projection│  │
│  │  - Layer-specific up-projection   │  │
│  └────────────────────────────────────┘  │
│              |                           │
│  ┌────────────────────────────────────┐  │
│  │  Feed-Forward Network              │  │
│  │  (FFN dim: 10,240)               │  │
│  └────────────────────────────────────┘  │
│                                          │
└──────────────────────────────────────────┘
              | (x 30 layers)
              v
         output image latent

S3-DiT core specifications:

ComponentSpecification
Transformer Layers30
Hidden Dimension3,840
Attention Heads32
FFN Dimension10,240
Total parameters6.15B
Text EncoderQwen3-4B (Frozen)
Image VAEFLUX VAE (Frozen)
Semantic EncoderSigLIP 2 (Frozen, for editing tasks)

9.2 Single-Stream vs Dual-Stream Design

Z-Image's S3-DiT adopts a Single-Stream architecture, maximizing cross-modal interaction compared with existing Dual-Stream models (for example FLUX's early blocks).

[Dual-Stream vs Single-Stream]

Dual-Stream (e.g. FLUX's early blocks):
  text tokens  ──> [Text Stream Blocks] ──┐
                                          ├──> merge later on
  image tokens ──> [Image Stream Blocks] ──┘
  -> cross-modality interaction concentrated in the later stages

Single-Stream (Z-Image S3-DiT):
  text + image + semantic ──> [Unified Blocks] ──> output
  -> dense cross-modal interaction at every layer
  -> parameter sharing maximizes efficiency

Advantages of Single-Stream:

  1. Cross-modal information exchange at every layer -- text and image representations influence each other from the start
  2. Parameter efficiency -- no separate per-modality deep blocks needed
  3. Training efficiency -- more expressive power for the same parameter count

9.3 QK-Norm and Sandwich-Norm

To keep large-scale Transformer training stable, Z-Image introduces two normalization techniques:

QK-Norm: Normalizes the Query and Key vectors in the attention layer to prevent attention scores from blowing up. This matters most in the early stages of training, and it eases gradient vanishing/exploding problems.

Sandwich-Norm: Applies normalization at the boundaries of a Transformer block (input and output) to control signal amplitude. It applies both Pre-Norm and Post-Norm, guaranteeing stable training in deep networks.

[Sandwich-Norm structure]

input -> [Pre-Norm] -> Attention/FFN -> [Post-Norm] -> output
         ^                                  ^
         |                                  |
         Sandwich-Norm: normalize on both sides

9.4 Conditional Information Injection

Conditional information (timestep, text, and so on) is injected through a scale-gate mechanism:

  1. Shared Low-Rank Down-Projection: compresses the conditional information with a low-dimensional projection shared across all layers
  2. Layer-Specific Up-Projection: a per-layer up-projection tunes the Attention and FFN paths independently
  3. Learnable Scale-Gate: learnable scale/gate parameters adjust the normalized inputs and outputs

This design is parameter-efficient while still letting each layer respond independently to the conditional information.

9.5 Decoupled-DMD (Distribution-Matching Distillation)

Z-Image Turbo's core distillation algorithm is Decoupled-DMD. It decomposes the existing DMD into two independent mechanisms and optimizes them separately.

[Decoupled-DMD structure]

Teacher Model (100 steps)
       |
       v
┌──────────────────────────────────┐
│     Decoupled-DMD Loss           │
│                                  │
│  ┌────────────────────────────┐  │
│  │ Term 1: CFG Augmentation   │  │
│  │ - Classifier-Free Guidance │  │
│  │ - tailored renoising sched.│  │
│  └────────────────────────────┘  │
│              +                   │
│  ┌────────────────────────────┐  │
│  │ Term 2: Distribution       │  │
│  │         Matching           │  │
│  │ - distribution-match optim.│  │
│  │ - tailored renoising sched.│  │
│  └────────────────────────────┘  │
│                                  │
│  each term's renoising schedule  │
│  is optimized independently      │
└──────────────────────────────────┘
       |
       v
Student Model (8 steps)

The key insight: CFG augmentation and distribution matching have different optimal renoising schedules. Decoupling the two mechanisms and optimizing them independently greatly improves few-step generation performance.

9.6 DMDR (Distribution Matching Distillation with Reinforcement)

After distillation, quality is improved further through reinforcement learning (RL) based post-training.

2-stage post-training:

  1. Offline DPO (Direct Preference Optimization):

    • Uses preference pairs generated by a Vision-LLM
    • Improves quality on aspects such as text rendering and object counting
    • Efficient training on offline data

  2. Online GRPO (Group Relative Policy Optimization):

    • Applies a multi-axis reward design
    • Policy optimization based on real-time feedback
    • Rewards designed across several quality axes (aesthetic quality, text alignment, structural accuracy)

9.7 Few-Step Generation Performance

Z-Image Turbo's few-step generation performance is as follows:

StepsQuality (relative)Speed (on H800)Notes
4 stepsUpper-middle~0.4sFor ultra-fast previews
8 steps (recommended)Best in class~0.8sOptimal quality/speed balance
16 stepsBest in class+~1.5sMarginal quality improvement
100 steps (Teacher)Reference~8sThe original Z-Image

8 steps is the recommended setting, and at that setting the model reaches quality close to the 100 step teacher model.

9.8 Data Infrastructure

Z-Image's other innovation lies in its data infrastructure. Rather than "simply gathering more data," it takes a systematic approach to maximizing information density.

[Z-Image Data Infrastructure]

┌─────────────────────────────────────────────┐
│              4 core modules                 │
│                                             │
│  ┌──────────────────┐  ┌──────────────────┐ │
│  │ Data Profiling    │  │ World Knowledge  │ │
│  │ Engine           │  │ Topological Graph│ │
│  │                  │  │                  │ │
│  │ quantitative     │  │ builds semantic  │ │
│  │ data-quality base│  │ relation structure│ │
│  └──────────────────┘  └──────────────────┘ │
│                                             │
│  ┌──────────────────┐  ┌──────────────────┐ │
│  │ Curriculum        │  │ Quality          │ │
│  │ Learning Manager  │  │ Caption Engine   │ │
│  │                  │  │                  │ │
│  │ optimizes data   │  │ auto-generates   │ │
│  │ mix per stage    │  │ ultra-detailed captions│
│  └──────────────────┘  └──────────────────┘ │
│                                             │
└─────────────────────────────────────────────┘

Core principles:

10. A Practical Guide to Z-Image Turbo

10.1 Using Python Diffusers

# Requires the latest version of diffusers
# pip install git+https://github.com/huggingface/diffusers
import torch
from diffusers import ZImagePipeline

# Load the pipeline
pipe = ZImagePipeline.from_pretrained(
    "Tongyi-MAI/Z-Image-Turbo",
    torch_dtype=torch.bfloat16,
    low_cpu_mem_usage=False
)
pipe.to("cuda")

# Generate an image
prompt = "A photorealistic portrait of a young woman with flowing red hair, \
golden hour lighting, shallow depth of field, 8K quality"

image = pipe(
    prompt=prompt,
    height=1024,
    width=1024,
    num_inference_steps=8,       # Turbo: 8 steps recommended
    guidance_scale=0.0,          # Turbo models do not need CFG
    generator=torch.Generator("cuda").manual_seed(42)
).images[0]

image.save("output.png")

Optimization options:

# Enable Flash Attention (speeds things up)
pipe.transformer.set_attention_backend("flash")       # Flash-Attention-2
# or
pipe.transformer.set_attention_backend("_flash_3")    # Flash-Attention-3

# Extra speedup with torch.compile
pipe.transformer = torch.compile(pipe.transformer, mode="reduce-overhead")

10.2 ComfyUI Integration

Where the model files go:

ComfyUI/
  models/
    vae/
      ae.safetensors                    # Z-Image VAE
    text_encoders/
      qwen_3_4b.safetensors            # Qwen3-4B text encoder
    diffusion_models/
      z_image_turbo_bf16.safetensors   # Z-Image Turbo model

Setup steps:

  1. Update ComfyUI to the latest version
  2. Download the 3 model files above and place them in the designated folders
  3. Start ComfyUI, then drag and drop the official workflow JSON onto the canvas
  4. Set num_inference_steps to 8

FP8 quantized version: In VRAM-constrained environments you can use the FP8 quantized model, which runs smoothly on 6-12GB of VRAM.

10.3 Optimal Settings Guide

SettingRecommended valueDescription
num_inference_steps8Optimal quality/speed balance
guidance_scale0.0Turbo models do not need CFG
height / width1024x1024The default square
torch_dtypetorch.bfloat16BF16 is optimal
Flash AttentionEnable it20-30% faster

Performance by resolution:

ResolutionGeneration time (H800)VRAM usageNotes
512x512~0.4s~8GBPreview/testing
768x768~0.6s~12GBMedium quality
1024x1024~0.8s~16GBRecommended setting
1024x1536~1.2s~20GBPortrait format

10.4 LoRA Fine-tuning

Z-Image Turbo supports fine-tuning through LoRA (Low-Rank Adaptation). You can teach it a particular style or concept, and the process is similar to an existing Stable Diffusion LoRA training pipeline.

# LoRA training example (conceptual)
# For the actual implementation, see diffusers train_text_to_image_lora.py

from diffusers import ZImagePipeline
from peft import LoraConfig

lora_config = LoraConfig(
    r=16,
    lora_alpha=16,
    target_modules=["to_q", "to_k", "to_v", "to_out.0"],
    lora_dropout=0.0,
)

# Apply the LoRA after training
pipe.load_lora_weights("path/to/lora/weights")

11. The Outlook for the Video and Image Generation AI Ecosystem

1. The rise of open source: Open source models such as Wan, CogVideoX, and HunyuanVideo have reached a level where they can compete with commercial models. Wan 2.6 in particular is free to deploy and customize in a company under the Apache-2.0 license, giving it an edge over commercial models on long-term cost management.

2. MoE arrives in video generation: Mixture-of-Experts, proven in LLMs, has spread into video generation. Wan 2.2 leads this trend, achieving efficient scaling that raises the total parameter count while holding inference cost steady.

3. Ultra-fast image generation: The few-step generation techniques exemplified by Z-Image Turbo have entered a mature phase. Advanced distillation techniques such as Decoupled-DMD combined with RL-based post-training deliver 100 step quality in 8 steps.

4. Multimodal integration: Native multimodality that generates video, audio, and text together is becoming reality. Wan 2.5-Preview demonstrated synchronized generation of text, image, video, and audio, and Google Veo 3.1 leads on audio integration.

5. Better physics-based simulation: Simulation accuracy for physical phenomena such as falling, collisions, fluid dynamics, and cloth behavior has improved considerably. Sora 2 sits at the frontier here, rendering even the physically accurate trajectory of glass shards and the refraction of light.

11.2 The Challenges of Video Generation

Temporal Consistency:

This is the most fundamental and persistent challenge. The longer the frame sequence runs, the worse the following problems get:

Today a single generation can produce 30-60 seconds of photorealistic video, but complex multi-character interaction still has limits.

Long-Form Generation:

The maximum generation length of most current models sits around 5-60 seconds. Generating long-form content of several minutes or more with consistency remains an unsolved problem. Wan 2.6's multi-shot narrative feature can be seen as an early attempt in this direction.

Physics Simulation:

Contact, deformables, and multi-agent dynamics remain hard. Natural motion for a single character has been achieved, but two people in conversation or complex physical interactions are difficult to keep consistent.

Evaluation:

There is a shortage of standard metrics for objectively assessing video generation quality. Existing metrics such as FVD and FID diverge from human perceptual judgment, and scenario-based temporal testing is regarded as more meaningful.

11.3 Where Things Are Headed

1. World Foundation Models:

Video generation models are evolving beyond mere content generation toward world simulation. In robotics, autonomous driving, and embodied AI, action-conditioned video generation is emerging as a core technology.

2. Hybrid Architecture:

Hybrid systems that step away from pure diffusion and combine several architectures are appearing. Attempts underway include combining Diffusion Transformer + Auto-Regressive + Flow Matching, or layering GAN-based ultra-fast refinement on top of diffusion.

3. The efficiency revolution:

As Z-Image Turbo showed, the "efficiency first" paradigm of shrinking the model while holding quality will spread. That opens up new applications such as on-device generation and real-time video generation.

4. Personalization and controllability:

As with Wan 2.6's Reference-to-Video, users will generate video with their own face and voice, and fine-grained control mechanisms such as ControlNet/IP-Adapter will be applied broadly to video generation as well.

12. Key Paper References

  1. Wan: Open and Advanced Large-Scale Video Generative Models

    • Authors: Wan-AI Team (Alibaba)
    • ArXiv: 2503.20314
    • Core: Wan 2.1's full architecture, Wan-VAE, Flow Matching training, the data pipeline

  2. Wan-Move: Motion-controllable Video Generation via Latent Trajectory Guidance

    • Authors: Ali-vilab (NeurIPS 2025)
    • GitHub: ali-vilab/Wan-Move
    • Core: motion-controlled video generation built on Wan
  1. Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer
    • Authors: Tongyi-MAI (Alibaba)
    • ArXiv: 2511.22699
    • Core: the S3-DiT architecture, data infrastructure, efficient training strategy

  1. Scalable Diffusion Models with Transformers (DiT)

    • Authors: Peebles & Xie
    • ArXiv: 2212.09748
    • Core: the prototype of the Diffusion Transformer architecture

  2. Flow Matching for Generative Modeling

    • Authors: Lipman et al.
    • ArXiv: 2210.02747
    • Core: the theoretical foundation of the Flow Matching framework

  3. Classifier-Free Diffusion Guidance

    • Authors: Ho & Salimans
    • ArXiv: 2207.12598
    • Core: the Classifier-Free Guidance technique

  4. Distribution Matching Distillation (DMD)

    • Authors: Yin et al.
    • ArXiv: 2311.18828
    • Core: the basis of few-step distillation techniques

  5. VideoGPT: Video Generation using VQ-VAE and Transformers

    • Authors: Yan et al.
    • ArXiv: 2104.10157
    • Core: a VQ-VAE + Transformer approach to video generation

  6. CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer

    • Authors: ZhipuAI/Tsinghua
    • ArXiv: 2408.06072
    • Core: video generation based on an Expert Transformer

  7. HunyuanVideo: A Systematic Framework For Large Video Generative Model

    • Authors: Tencent
    • ArXiv: 2412.03603
    • Core: a 13B large-scale video generation framework

13. References

Official Resources

Tutorials and Guides

Model Comparisons and Benchmarks

Hugging Face Model Pages

14. Conclusion

Wan and Z-Image Turbo represent the two directions Alibaba is pursuing in generative AI.

Wan goes after "the democratization of open source video generation." Fully open source under the Apache-2.0 license, it spans a range of scales from the 1.3B lightweight model to the 27B MoE large model. It brings together the latest techniques, including a 3D Causal VAE, a DiT backbone, Flow Matching, and an MoE architecture, and it secured the accessibility to run even on consumer-grade GPUs. It does not reach the resolution or length of commercial models such as Sora and Kling, but it holds a singular position on customization freedom and cost efficiency.

Z-Image Turbo is the vanguard of the "efficiency revolution." It matches the quality of 20B-80B class models with 6B parameters in just 8 steps, realized through a 3-part innovation: the single-stream design of S3-DiT, Decoupled-DMD distillation, and DMDR reinforcement learning post-training. Its training cost of 314K GPU hours is more than 10x more efficient than earlier large models, and its sub-second inference speed opens the door to real-time applications.

Both models carry the message that "bigger and more expensive is not always better." They demonstrated empirically that efficient architectural design, data quality management, and advanced training strategies can matter more than brute-force scaling. That will serve as an important foundation as video and image generation AI moves toward on-device, real-time, and personalized applications.

Comments

No comments yet.

Sign in to leave a comment