02 · Model Architectures
MoE
Mixture of Experts
A design that splits part of a model into multiple expert subnetworks and routes each token to a subset of them. It can increase total capacity without activating every parameter for every token, though routing and communication add complexity.
Concrete example
A model might hold two trillion parameters total but light up only a fraction of them for any one query.
Why it matters
A key efficiency trick in several frontier models — big “brain,” cheaper to run per query.