Continuous-time SSMs, selective scan parameters, discretization, and linear-time sequence modeling.
For a small example, a sequence contains signal tokens among irrelevant ones. Let input-dependent gates retain the signal, discard noise, and compare with a fixed linear recurrence.
Hundreds of thousands of readings exceed comfortable attention length. Chunk safely, train a selective SSM baseline, test long-range events, and compare throughput and memory with a transformer.
Initializing Interactive Playground...
Predict how one control changes the State Space Models & Mamba result.
The starter deliberately follows a tempting but flawed assumption: Equating linear complexity with faster execution
Check: Your saved hypothesis predicts a direction, the experiment tests it, and the result changes or supports your explanation.
Explanations after each answer · 80% to pass · State Space Models & Mamba
Chapter 1 of 6 · Core
State Space Models (SSMs) map a continuous 1D input sequence to a continuous output sequence through an -dimensional hidden state representation .
Key equations
import torch
import torch.nn as nn
import math
def make_hippo_matrix(N: int) -> torch.Tensor:
"""Generates the continuous HiPPO-LegS memory matrix A."""
P = torch.sqrt(1 + 2 * torch.arange(N, dtype=torch.float32))
A = P.unsqueeze(1) * P.unsqueeze(0)
A = torch.tril(A, diagonal=-1) + torch.diag(torch.arange(N, dtype=torch.float32) + 1)
return -A
class ContinuousSSM(nn.Module):
def __init__(self, state_dim: int = 64):
super().__init__()
self.state_dim = state_dim
self.A = nn.Parameter(make_hippo_matrix(state_dim))
self.B = nn.Parameter(torch.randn(state_dim, 1) / math.sqrt(state_dim))
self.C = nn.Parameter(torch.randn(1, state_dim) / math.sqrt(state_dim))
self.D = nn.Parameter(torch.ones(1))Two free references selected for State Space Models & Mamba.
Free Paper
Introduces input-dependent selective state-space models and hardware-aware recurrent computation.
Free Paper
Introduces S4 and the structured state-space foundation behind later sequence models.
Primary sources and free technical reading for State Space Models & Mamba.
Introduces input-dependent selective state-space models and hardware-aware recurrent computation.
Introduces S4 and the structured state-space foundation behind later sequence models.