Sparse MoE Architecture Leap: DeepSeek V4 & Mistral Small 4

Open-weight releases like DeepSeek V4-Pro and Mistral Small 4 demonstrate an extreme shift toward parameter sparsity to cut active inference overhead. DeepSeek V4-Pro activates only 49B parameters per token from its 1.6T parameter pool, while Mistral Small 4 activates just 6.5B of its 119B parameters. This enables smooth deployment on standard single-GPU servers and unified memory workstations.