Qwen3.8-Flash-Next: Superblock Cutaway
Read the architecture at four scales: the 48-layer rhythm, one Superblock, sparse activation across 512 experts, and the memory highway that reads and writes across layers.
LM BODY 125 B
ACTIVE 6 B
LAYERS 48
CONTEXT 262.144 K
Four routes, four distinct roles
Teal: state. Coral: retrieval. Purple: experts. Dashed grey: control.
SUPERBLOCK ×12
Three Gated DeltaNet layers plus one Qwen Sparse Attention layer, repeated 12 times.
L25–L28
Block 7 is the selected cutaway. Identical structure does not imply shared parameters.
State updates, expert routing and sparse retrieval
Follow the purple and coral connections through a forward pass, then the teal loop into the next block. Zoom in to inspect modules; choose Fit to restore the complete view.
Gated Residual Highway
Input, token memory, cross-layer state and three Delta updates share one cutaway.
x_in: TOKEN
N-GRAM: 20M entries
GR READ ×4: state
L1 · Δh1: Gated DeltaNet
L2 · Δh2: Gated DeltaNet
L3 · Δh3: Gated DeltaNet
M[1] □ □ □ □ ··· □ □ M[N]: fixed-size recurrent state
MoE Router → Expert Field
Gating scores route each token to 10 routed experts, with one shared expert always active.
E041: .96
E087: .91
E233: .83
... +7: TOP-10
10 selected + 1 shared: 501 routed experts remain cold
QSA → Σ → GR Write
Compress the index first, compute globally over only the Top-k relevant tokens, then aggregate, add the residual and write state back.
R1 · MICRO-BLOCK: 4-token compression
R2 · SCORE: block-level index
R3 · TOP-K: 2048 tokens
R4 · ATTEND: sparse global view
Σ: RESIDUAL
GR WRITE ×4: STATE
x_out: OUTPUT
STATE: Read / write / local memory
ROUTE: Expert selection / capacity
RETRIEVAL: Compression / Top-k / QSA
RESIDUAL: Backbone / control
The 3 + 1 rhythm, layer by layer
Twelve blocks across, four slots down. The fourth slot in every block is QSA.
Qwen 48-layer rhythm
Each of the 12 Superblocks contains three GDN slots and one QSA slot.
The compute path of one token
The model body holds 125B parameters; each token activates about 6B.
Activated parameters per token
6B activated parameters compared with a 125B model body.
ACTIVATION RATIO: ≈ 4.8% of LM body
Four mechanisms, four bottlenecks
GDN compresses history into fixed-size state. QSA restores a sparse global view at every fourth layer. MoE separates parameter capacity from per-token computation. GR gates reads and writes across four residual branches.
From input to write-back in 10 checkpoints
The logical sequence for one token. Color identifies the mechanism responsible for each stage.
00: Input
01: N-gram
02: GDN #1
03: GDN #2
04: GDN #3
05: Route
06: Experts
07: Top-k
08: QSA
09: GR Write
Sources, definitions and teaching abstractions
Sources frozen on 2026-08-31. Layer count, 3:1 attention rhythm, MoE configuration, context length, Gated Residual branches and N-gram configuration come from the official Qwen repository, technical report, blog and model config. Repeated per-layer MoE modules are shown as one shared capacity cutaway for teaching; this is not an operator-level execution graph.