# Your Experts Have Partners. There's HOPE for Keeping Them Together

**Published:** November 9, 2026
**Author:** Hugging Face - Blog
**Source:** Hugging Face Blog
**Language:** en-US
**HTML:** https://aiqazaqstan.com/en/articles/your-experts-have-partners-there-s-hope-for-keeping-them-together

A Blog post by Amazon on Hugging Face

Mixture-of-Experts (MoE) models have enabled state-of-the-art LLMs to be scaled to hundreds of billions of parameters, but they come with a fundamental and costly caveat: every expert MLP must occupy memory at inference time, even though the vast majority are idle for any given token. For example, Qwen3.5-122B has 256 experts in each of its 48 MoE layers, and 95% of its ~250 GB parameter memory footprint is taken up entirely by these expert MLPs alone. However, for any given token, only 8 experts in each layer are actually used. Expert pruning is one of the most direct fixes to this memory bottleneck. That is, permanently delete the least useful experts. Pruning avoids the latency issues which make dynamic loading/off-loading of experts impractical; it does not require any training; and it composes with quantization. Given these attractive benefits and the simplicity of the idea, it's easy to see why expert pruning has become a go-to approach for shrinking MoEs. The difficulty lies in deciding which experts are safest to remove. Existing pruning methods score each expert independently, assigning every expert a scalar importance score (based on a calibration dataset), and cutting out the least important experts (in each layer independently). These are first-order methods, because they assume each expert's contribution is independent of all others. In reality, experts form cooperative groups. We observe consistent clusters of co-activated experts in our own experiments, and independent work has shown that expert cooperation is a stable property of MoEs and is important for model performance. Ignoring these interactions leads to suboptimal pruning, especially at aggressive rates. Our newest paper derives HOPE (Higher-Order Pruning of Experts), a second-order method that accounts for pairwise expert interactions. At 50% pruning, HOPE achieves the best average rank of any method, with gains of up to +6.1% on agentic coding tasks over the next-best pruning method. In this post, we explain how HOPE works, show where it outperforms existing methods, and walk through how to use it for your own models. For readers unfamiliar with MoE architectures, see Mixture of Experts (MoEs) in Transformers. In a standard MoE, each layer has EE experts, and each input token xx is routed independently to KK such experts ( K=8 K = 8 is a typical choice). Routing is performed by a single layer which produces softmax-normalized gate weights gk(x)g_{k}(x) for each expert kk, wherein the KK experts with the highest gate weights are selected to form the active set of experts, T(x)⊂{1,..,E}T(x) \subset\{1,..,E\}. The output of the layer is then ∑k∈T(x)gk(x)fk(x)\sum\limits_{k\in T(x)}g_{k}(x)f_{k}(x), where fk(x)f_{k}(x) is the output of expert kk for token xx. Left: typical MoE layer, where each token is routed to the top-K experts. Right: first-order methods rank and delete experts based on a scalar importance score, but the best pruning decisions will also consider the strength of interactions between experts. Importantly, experts in the selected set T(x)T(x) are not independent. Experts form stable cooperative groups that get consistently co-activated together. This has been observed before in the literature, as well: expert co-selection patterns emerge early in training and stabilize throughout. This means that when we prune experts from an MoE layer, pruning decisions need to consider the higher-order effects between these experts. Removing expert A might seem fine on its own, but if expert B—which frequently cooperates with and relies on A—is highly important, then removing A would dismantle a cooperative group just through a single deletion. First-order methods miss this entirely, because they score A and B separately. This effect compounds at high pruning rates. When we remove up to 50% of experts, we risk breaking many cooperative links. This is where second-order pruning is most advantageous. (See Appendix A in our corresponding paper for the full derivation of the HOPE objective) When we delete a set of PP experts from some MoE layer, the selected set of experts T(x)T(x) (for any token xx) can change. The pruned experts that would have been selected are replaced by the experts with the next-highest gate weights. The resulting change in layer output is the pruning error. Our goal is to find the prune-set PP (of some predefined pruning budget ∣P∣\vert P\vert) that minimizes the resulting pruning error (over tokens xx from our calibration set). Unfortunately, minimizing the pruning error over PP directly is intractable. The pruning error depends on the identities of the replacement experts, which in turn depend on the prune-set PP itself. HOPE sidesteps this by defining a tractable surrogate ZZ, which strictly upper-bounds the dominant component of the pruning error: Z=[∑j∈P∩T(x)gj(x)∥fj(x)∥2]2≥true errorZ = \bigl[\sum\limits_{j \in P\cap T(x)} g_{j}(x)\Vert f_j(x)\Vert_{2}\bigr]^{2} \geq \text{true error} Because ZZ is a squared sum, expanding it naturally produces cross-terms gi(x)gj(x)∥fi(x)∥2∥fj(x)∥2g_{i}(x)g_{j}(x)\Vert f_{i}(x)\Vert_{2}\Vert f_{j}(x)\Vert_{2} between pairs of experts. These are the interaction terms that first-order methods discard. To find the optimal prune-set, HOPE minimizes E[Z]\mathbb{E}[Z] over the calibration set, which can be reformulated as the following quadratic program: P∗=arg min⁡p p⊤Fps.t.p∈{0,1}E,∑k=1Epk=∣P∣P^{*} = \argmin_{p}\; p^{\top}Fp \quad \text{s.t.} \quad p \in \{0,1\}^{E}, \sum\limits_{k=1}^{E}p_{k} = \vert P\vert where F∈RE×EF\in\mathbb{R}^{E\times E} is defined as: Fi,j=1∣Xi,j∣∑x∈Xi,jgi(x)gj(x)∥fi(x)∥2∥fj(x)∥2whereXi,j={x:i,j∈T(x)}F_{i,j} = \frac{1}{\vert \mathcal{X}_{i,j}\vert} \sum\limits_{x\in\mathcal{X}_{i,j}} g_{i}(x)g_{j}(x)\Vert f_{i}(x)\Vert_{2}\Vert f_{j}(x)\Vert_{2}\quad\text{where}\quad \mathcal{X}_{i,j} = \{x: i,j \in T(x)\} This F-matrix is collected on the calibration dataset, and the optimal prune-set is obtained by solving the resulting quadratic program (QP) with a continuous relaxation (then rounding to binary solutions). Each entry Fi,jF_{i,j} captures the average gate-weighted output product of experts ii and jj, across tokens where both are co-activated. Whereas the diagonal Fk,kF_{k,k} measures individual expert importance, the off-diagonal entries ( Fi,jF_{i,j} for i≠ji\neq j) measure pairwise cooperative contribution. What does FF look like in practice? It shows the structure of expert cooperation, which depends on layer: Visualization of the interaction matrix F for GLM-4.5-Air (128 experts per layer, 106B parameters total) at three representative layers (early, middle, late). Experts have been clustered to reveal block structure. Bright blocks along the diagonal indicate groups of experts with strong pairwise interaction scores. In terms of practical cost, HOPE requires the same calibration pass as any other expert-pruning method (one forward pass over calibration data). The only addition is solving the quadratic program, which is non-convex, but can be solved efficiently via a standard QP solver (1–2 seconds each layer). The total additional overhead is negligible compared to first-order methods, as the time for calibration dominates. Notably, HOPE has a strong connection to Router-weighted Expert Activation Pruning (REAP), a state-of-the-art first-order expert-pruning method. HOPE's F-matrix encodes individual expert importance scores on the diagonal, and this diagonal effectively recovers the scores used by REAP to independently rank experts for pruning (differing only by the variance of expert contributions). The diagonal of FF recovers REAP's expert rankings almost perfectly (Spearman ρ=0.988\rho = 0.988). Thus, any improvement HOPE achieves over REAP is attributable to the second-order interaction terms in the off-diagonal entries of the FF-matrix. We evaluated HOPE as an expert pruning method using 3 models (Qwen3.5-122B, Qwen3.5-35B, and GLM-4.5-Air), with 2 calibration datasets (Evol-CodeAlpaca and SWE-Bench Verified trajectories). We used the Tulu3 suite and SWE-Bench Pro (agentic coding) to test the resulting pruned models, comparing HOPE to 4 first-order baselines (REAP, MAN, EAN, and Frequency) across pruning rates spanning from 10% to 50%. HOPE achieves the best overall rank across conditions and dominates at aggressive pruning rates. Left: average rank (lower = better) across all models, calibration sets, and benchmarks. Right: SWE-Bench Pro performance on Qwen3.5-35B. Across all conditions, HOPE achieved the best average rank (2.07 out of 5 methods), with the highest rate of top-1 finishes (39% of conditions) and top-2 finishes (67%). In a head-to-head comparison, HOPE beats every other baseline in the majority of conditions. As expected, HOPE's advantage grows at higher pruning rates. As more experts are removed, more cooperative links are broken, and the off-diagonal terms of FF become more important. At 50% pruning, HOPE's average rank improves to 1.58 (vs. 2.42 for the next-best method, REAP). This is also the most deployment-relevant setting, where memory savings are large enough to accommodate longer contexts, deeper reasoning, and more tool calls. HOPE's advantage is most pronounced on SWE-Bench Pro, an agentic coding benchmark where models must solve real GitHub issues autonomously. These problems involve compositionally diverse tasks (e.g. coding, tool calls, reasoning, etc.) over many turns and likely exercise diverse expert combinations—exactly the regime where preserving cooperative structure matters most. On Qwen3.5-35B-A3B at 50% pruning, HOPE achieved 42.3 on SWE-Bench Pro, matching the unpruned model's 41.8 (while REAP dropped to 36.2, a 13% regression). Notably, pruned models can occasionally match or exceed the unpruned baseline, likely because the removal of low-relevance experts reduces noise on the target tasks. This effect is not unique to HOPE and has been observed across pruning methods. Furthermore, our experiments showed that code-focused calibration with any pruning method (including HOPE) did not severely degrade non-coding benchmarks, and HOPE performed well across the Tulu3 suite regardless of calibration set. Average strength of cooperation (measured by F-matrix value) with retained experts, split by experts on which HOPE and REAP disagree. HOPE's advantage stems from optimizing over the entire interaction matrix FF. To understand how this affects pruning decisions in practice, we examine the experts where HOPE and REAP disagree. For each disputed expert, we compute its cooperative score (i.e. its average F-value with other retained experts). Experts retained only by HOPE consistently have higher cooperative scores than experts retained only by REAP. In other words, as part of the QP optimization, HOPE sacrifices some individually high-scoring experts in favor of preserving cooperative clusters. The gains in benchmark performance (shown above) validate that this trade-off is advantageous. We created a basic package to run HOPE. It has three steps: 1) calibrate; 2) solve; 3) prune. Our package has been tested on Qwen3, Qwen3-Next, Qwen3.5, and GLM-4.5. The calibration step runs a single forward pass over calibration data, whether it be agentic rollouts or single-turn traces. This uses HuggingFace transformers to hook into the forward pass of each individual MoE layer. For each token, this records which experts are selected, their gate weights, output norms, and pairwise co-activation statistics. The output of this step is the F-matrix, stored in an HDF5 file. This command materializes the F-matrix and solves the binary quadratic program via a continuous relaxation. This takes a few seconds per layer, and the output is a JSON file which contains the prune-set. This is technically the final output of the HOPE algorithm. To complete the pipeline, we run this command to actually create the smaller model checkpoint from the prune-set. The pruned model is a standard HuggingFace checkpoint which is smaller on disk and in memory (no changes to vLLM or SGLang model definitions needed). We also released a more compact Python API: HOPE's advantage is most pronounced at higher pruning rates (40–50%). At lower pruning rates (10–20%), first-order methods like REAP and MAN are sometimes competitive, and HOPE's advantage shrinks. Additionally, as with most expert-pruning algorithms, HOPE prunes each layer independently and does not account for cross-layer interactions. In our paper, we presented a theoretical framework for cross-layer HOPE, but the results remain preliminary. Finally, HOPE captures pairwise interactions, but even higher-order interactions (triplets and beyond) could provide additional signal, although they are combinatorially difficult to estimate. For more information, see our corresponding paper and codebase. Paper: https://arxiv.org/abs/2609.18916 Code: https://github.com/awslabs/HOPE To see how HOPE can be applied to next-generation hybrid architectures, see our corresponding repository.

---

*From the original source: [Hugging Face Blog](https://huggingface.co/blog/amazon/higher-order-pruning-experts). Open original.*

**Tags:** #Papers with Code #arXiv #Hugging Face

---

*AI Qazaqstan — non-commercial informational portal about AI in Kazakhstan. Machine translations and summaries are AI-generated.*