Mixture of Experts (MoE)

Sparse routing mechanisms, gating networks, Top-K dispatch, and auxiliary load balancing.

For a small example, a router assigns four tokens to two experts. Apply top-k scores, respect capacity, combine expert outputs, and observe what happens when every token chooses one expert.

In practice

More parameters are desired without proportional compute per token. Balance expert load, track dropped tokens, tune auxiliary loss, and evaluate specialization versus dense baselines.

Initializing Interactive Playground...

Experiment notes

Record your experiment

Predict how one control changes the Mixture of Experts (MoE) result.

PredictionWhat changed, and why?

Implementation

Repair a failing Mixture of Experts (MoE) starter

The starter deliberately follows a tempting but flawed assumption: Calling MoE a free capacity increase

Run the starter once and inspect the first wrong value, failed check, or incomplete branch.
Repair the smallest relevant part of the code. Do not rewrite the whole solution.
Check the solution, then explain why this correction works: Include router, communication, memory, and imbalance costs.

Check: The browser checks pass and you can name the failure mechanism before opening the explanation.

Failing starter lab

Select Top-2 experts

Given router logits, return the indices of the two highest-scoring experts.

Run the incomplete starter before opening hints. Use the first wrong output or failed check to form a repair hypothesis.

Console

Run the code to inspect its output. Use Check solution only when you want to run the exercise checks.

Code runs only in a time-limited browser worker. It is not sent to the IntuitiveAI server. Do not paste secrets.

Expected result

[1, 3]
ALL TESTS PASSED

Chapter 1 of 6 · Core

Sparse Gating & Expert Dispatch Architecture

Mixture of Experts (MoE) replaces standard dense Feed-Forward Network (FFN) layers in Transformers (e.g., Mixtral 8x7B, DeepSeek-V3) with parallel expert networks and a trainable router (gating network) .

Gating Network Equations:

Expert Output Combination:

Sparse Computation Advantage: Even with experts (total 47B parameters), routing only experts per token executes only 13B active FLOPs per inference step!

Key equations

PyTorch Sparse MoE Router & Top-K Dispatch Blockpython

import torch
import torch.nn as nn
import torch.nn.functional as F

class SparseMoERouter(nn.Module):
    def __init__(self, d_model: int, num_experts: int = 8, top_k: int = 2):
        super().__init__()
        self.num_experts = num_experts
        self.top_k = top_k
        self.gate = nn.Linear(d_model, num_experts, bias=False)
        
        # Parallel expert networks
        self.experts = nn.ModuleList([
            nn.Sequential(
                nn.Linear(d_model, d_model * 4),
                nn.SiLU(),
                nn.Linear(d_model * 4, d_model)
            ) for _ in range(num_experts)
        ])

    def forward(self, x: torch.Tensor):
        # x: [Batch, Seq_len, d_model]
        batch_size, seq_len, d_model = x.shape
        x_flat = x.view(-1, d_model)
        
        # Router logits & Top-K gating
        logits = self.gate(x_flat)
        weights, indices = torch.topk(F.softmax(logits, dim=-1), self.top_k, dim=-1)
        
        out = torch.zeros_like(x_flat)
        for i, expert in enumerate(self.experts):
            # Find tokens dispatched to expert i
            batch_idx, kth_idx = torch.where(indices == i)
            if len(batch_idx) > 0:
                expert_in = x_flat[batch_idx]
                expert_out = expert(expert_in)
                gating_weight = weights[batch_idx, kth_idx].unsqueeze(-1)
                out[batch_idx] += expert_out * gating_weight
                
        return out.view(batch_size, seq_len, d_model)

NextAuxiliary Load-Balancing Loss & Expert Capacity Bounds

Video explanation

Mixture of Experts in AI

IBM Technology

See how routers activate subsets of expert networks and why capacity and load balancing matter.

Watch on YouTube

References

Primary sources and free technical reading for Mixture of Experts (MoE).

Switch Transformers

Studies sparse expert routing and trillion-parameter language-model scaling.

PaperFree

Outrageously Large Neural Networks

A foundational sparsely gated mixture-of-experts architecture.

PaperFree

Back to Topic Library

Feedback & sharing

Mixture of Experts (MoE)