N
Llama-style decoder
Nanbeige4.3-33B
A uniform stack: every configured layer uses full attention and a dense feed-forward network.
52full-attention layers
Architecture comparison 01 / 03
Two dense decoders.
Different paths through a token.
Nanbeige repeats full attention in 52 layers. Qwen alternates recurrent linear attention with full attention across 64 text-decoder layers. Explore the published configuration, one layer at a time.
Configuration viewLayer types and dimensions come from pinned model configs. Nanbeige weights, activations and routing were not measured here.
01 / At a glance
The decoder shapes
Q
Hybrid text decoder
Qwen3.8-27B
A repeating pattern: three recurrent linear-attention layers, then one full-attention layer.
48 + 16linear + full layers
| Configuration | Nanbeige | Qwen text decoder |
|---|
02 / Cache lens
How full-attention K/V would scale
Choose a context length to calculate raw f16 key and value tensor storage for one sequence if an inference runtime enables caching.
2K tokens256K tokens
Copy the page URL to share this context.
Nanbeige4.3-33B
Qwen3.8-27B text decoder
Formula: tokens × full-attention layers × 2 (K and V) × KV heads × head dimension × 2 bytes.
Calculation scope: one sequence, f16 K and V for full-attention layers. Nanbeige's released model and generation config set use_cache: false, so its number applies when a runtime enables caching. Qwen's recurrent layers maintain separate state; weights and runtime overhead are separate.
03 / Layer map
Follow the stack
Each square is one configured decoder layer. Select a layer to inspect its attention and feed-forward dimensions.
Full attentionRecurrent linear attention
Nanbeige
52 layersQwen text decoder
64 layersSelected layer
Loading configuration
Pinned sources
Inspect the configs
Qwen values refer to the text_config decoder inside its released model configuration. Context values are configured maximum positions, not measured usable context.