Tag: #flash-attention

한국어English日本語中文

← All posts

• LLM Landmark Papers Roundup 2026 - Transformer / Scaling Laws / Flash Attention / Mamba / DeepSeek-R1 / Titans Deep Dive

  From the 2017 Attention Is All You Need paper to 2026 Titans and DeepSeek-R1, a thematic roundup of the 50+ landmark papers that built the LLM era. Transformer, BERT, GPT 1-3, Scaling Laws, Chinchilla, InstructGPT, PaLM,

  2026-05-16 · 21 min read #llm-papers#transformer#scaling-laws#flash-attention#mamba

• Complete LLM Serving Optimization Guide: KV Cache, PagedAttention, and Quantization

  A complete engineering deep dive into LLM serving optimization. From the KV cache memory crisis to the virtual-memory revolution of PagedAttention, continuous batching, speculative decoding, quantization techniques, and

  2026-03-18 · 30 min read #model-serving#kv-cache#paged-attention#vllm#quantization

• How Matrices Fly on GPU: Complete Deep Dive from GEMM to FlashAttention

  Matrix multiplication accounts for over 80% of deep learning compute. This post dissects every layer of optimization from naive O(n³) code to cache blocking, cuBLAS, Tensor Cores, and the IO-aware revolution of FlashAtte

  2026-03-18 · 15 min read #gemm#matrix-multiplication#flash-attention#gpu#parallel-computing

• LLM Pretraining & Scaling Laws: From Chinchilla to Flash Attention and MoE

  A complete guide to LLM pretraining: Chinchilla scaling laws, Common Crawl data pipelines, Flash Attention 2, GQA, MoE architectures, and the latest pretraining recipes from DeepSeek-V3, Llama 3.1, and phi-4.

  2026-03-17 · 14 min read #llmpretraining#scalinglaws#chinchilla#flash-attention#mixtralmoe

• Advanced CUDA GPU Programming: Warp Optimization, Tensor Cores, and Triton Kernels

  A comprehensive deep dive into CUDA memory hierarchy, Warp optimization, Tensor Core WMMA API, Flash Attention implementation, and Triton custom kernel authoring for accelerating AI model training.

  2026-03-17 · 19 min read #cuda#gpuprogramming#tensorcore#triton#flash-attention

• FlashAttention Paper Analysis: Revolutionizing Transformer Training and Inference with IO-Aware Exact Attention

  An in-depth analysis of the FlashAttention series (v1 through v3). Covers the tiling strategy of the IO-aware algorithm, GPU SRAM/HBM memory hierarchy utilization, backward pass recomputation, parallelism improvements in

  2026-03-09 · 22 min read #ai-papers#flash-attention#transformer#gpu-optimization#attention-mechanism

• FlashAttention: Optimizing Attention Through GPU Memory Hierarchy

  A review of the FlashAttention paper with an in-depth analysis of IO-aware attention optimization leveraging the GPU HBM/SRAM memory hierarchy.

  2026-03-01 · 18 min read #ai-papers#flash-attention#gpu#optimization#transformer