Researchers have introduced HOPE (Higher-Order Pruning of Experts), a method for shrinking Mixture-of-Experts (MoE) language models by deleting experts. The source is a HuggingFace blog post that describes the paper.
According to the post, MoE models can reach hundreds of billions of parameters, but every expert must stay in memory at inference time even though only a few are used for each token. The post gives Qwen3.5-122B as an example. It has 256 experts in each of its 48 MoE layers, and the expert MLPs take up 95% of its roughly 250 GB memory footprint. For any given token, only 8 experts per layer are active.
Existing pruning methods score each expert separately and remove the lowest scorers. The post calls these first-order methods. The authors argue that experts form cooperative groups that are consistently activated together, so scoring them independently can lead to weaker pruning decisions, especially at high pruning rates.
HOPE accounts for pairwise interactions between experts. It builds an interaction matrix from a calibration dataset and selects the set of experts to remove by solving a quadratic program. The post says the calibration step is the same as for other pruning methods, and that solving the program adds about 1 to 2