Exex
exex — expert exchange — is a toolkit for memory-efficient training, merging, and pruning of domain-specific experts on Gemma 4 26B MoE.
Stack: Python. License: Apache 2.0. Public repo.
The name is a double meaning: expert exchange (swapping experts between models) and the file extension .ex (as in executables — things that run).
Gemma 4 26B MoE
| Property | Value |
|---|---|
| Total parameters | ~26 B |
| Active parameters per token | ~3.8 B |
| Total experts | 128 + 1 shared |
| Active experts per token | 8 |
| Context window | 256 K tokens |
| Attention | Hybrid sliding-window + global |
| Modalities | Text + Image |
| License | Apache 2.0 |
| VRAM (fp16, 256K ctx) | ~23 GB |
Each transformer block has a standard MLP plus a set of MoE blocks whose outputs are summed. A learned router selects 8 of the 128 specialist experts per token; the shared expert always fires. 26 B model capability at ~3.8 B compute per forward pass.
Features
Memory-efficient expert training:
- LoRA on individual experts — fine-tune a single expert's FFN layers, freeze everything else.
- Gradient checkpointing — trades compute for memory; runs on a single 24 GB GPU.
- 8-bit / 4-bit quantized base — bitsandbytes NF4 / Int8 for the frozen base, full precision adapters.
- Expert-only forward pass — optionally skip non-expert layers to shrink activation memory.
Expert merging:
- Direct weight copy — transplant a trained expert at a chosen index.
- Weighted linear interpolation — blend two expert weight matrices by α.
- Task-vector merging — compute the delta from base, add to any target (works across expert indices).
Expert pruning:
- Utilisation-based — collect router logits over a calibration dataset, drop experts below a threshold.
- Weight-magnitude — remove experts by smallest L2 norm.
- Structured zeroing — zero out pruned expert weights for sparse inference runtimes.