CRISP: Co-designing Pruning and Scheduling for Efficient MoE Inference on Edge Servers

What it claims

CRISP prunes MoE experts per routed expert combination rather than uniformly, then schedules the resulting mixed-width experts so that loading overlaps with compute. The motivating measurement is on an NVIDIA RTX 4090 over PCIe 4.0, running Mixtral-8x7B at 4-bit. There, loading the two routed experts takes 11.2 ms of a 14.4 ms MoE layer (78.0%), while expert computation takes 2.6 ms (18.1%) and the router 0.4 ms. The headline result is up to 2.1x throughput at equal memory with at most 3.6% accuracy loss versus full width, on Mixtral-8x7B and DeepSeek-MoE-16B.

How it works

Profiling uses 2,048 sequences sampled from the C4 validation set. Neurons in each expert are ranked by importance on this set and physically reordered. A width-k execution then reads only the first k neurons, so one stored copy serves every width with no duplicate weights.

Offline profiling also sorts routed combinations into Hot, Warm and Cold tiers by frequency. Within each combination, the expert whose masking hurts perplexity more is labelled dominant and the other auxiliary.

flowchart TD
  R[Router picks expert pair] --> T{Combination tier}
  T -->|Hot| H[Both experts full width]
  T -->|Warm| W[Dominant full, auxiliary narrowed]
  T -->|Cold| C[Dominant only, auxiliary skipped]
  H --> S[Pruning-aware I/O scheduler]
  W --> S
  C --> S
  S --> O[Wider experts first, their compute hides narrower loads]

The scheduler runs wider experts first. Their longer compute windows hide the load of the narrower experts that follow.

Evidence

The motivating measurements are all on Mixtral-8x7B with C4 calibration.

• Skewed combinations. The top 25% of expert combinations carry over 60% of token activations, and the bottom 29% carry under 10%. The skew holds across layers, but which combinations are hot differs by layer.
• Uniform pruning hurts frequent pairs most. At 50% uniform pruning, accuracy drops 4.6% on hot combinations (top 10%), 2.8% on medium ones and 2.0% on cold ones (bottom 50%). That is a 2.3x gap.
• Dropping a whole expert is too coarse. Dropping Expert 3 costs 0.5% to 1.2% accuracy in combinations where it is auxiliary, but 12.5% to 18.7% where it is dominant.
• Experts within a pair are asymmetric. Across the top 20 combinations, masking the dominant expert raises perplexity by 2.64 on average. Masking the auxiliary raises it by 0.71, a 3.7x difference.
• Load and compute scale together. A full-width expert takes 7.5 ms to load and 0.8 ms to compute. At 50% width it takes 3.8 ms and 0.4 ms. Load time exceeds compute time by roughly 9x at both widths.

Limits

The fetched text stops partway through Section 2. It does not include the end-to-end evaluation, so the baselines, tasks and accuracy metric behind "2.1x" and "3.6%" are not shown. The tier cut-offs and the Warm-tier auxiliary width are also not stated.

All profiling uses general web text from C4. Tier assignments may shift on domain-specific traffic, and hot combinations already vary by layer.

The motivating analysis assumes top-2 routing on a discrete GPU with host offload. The paper notes that unified-memory platforms have a different bottleneck: contention for shared bandwidth rather than explicit transfers. Given the ~9x gap between load and compute time, overlap can hide only part of the load time. Most of the gain must come from moving fewer bytes.

The paper is licensed CC BY 4.0. No code release or run instructions appear in the fetched text.