Three architectures for processing sequences, each defined by one core idea. This is the short version of how they differ and why the field moved the way it did.

The one-idea summary

RNN / LSTM   sequential + forgetting   read word by word, slowly forget
Transformer  parallel + global         read everything at once, link any two tokens
Mamba / SSM  linear + stateful         carry a smart running state, cheap on long inputs

1. RNN / LSTM — sequential, with fading memory

An RNN processes one token at a time, carrying a hidden state forward. To compute token 100 it must first pass through tokens 1 to 99.

• Sequential — each step waits for the previous one, so it can't parallelize well for training.
• Forgets — the hidden state is overwritten each step, so early information fades over long sequences (the vanishing-gradient problem).

t1 -> t2 -> t3 -> ... -> t100     (must wait; token 1's info fades by t100)

Dominant in NLP roughly 2014-2017. Died out for large-scale work because of the two weaknesses above.

2. Transformer — parallel, with global connections

The Transformer (2017) dropped recurrence and used self-attention: every token looks at every other token directly and in parallel.

• Parallel — all tokens computed at once, so it trains efficiently on GPUs. This is the practical reason it scaled into LLMs.
• Global — any two tokens are one attention step apart, regardless of distance, so no long-range fading.
• Trade-off — processing a set loses word order, so Transformers add positional encoding.

t1  t2  t3  ...  t100            (all at once)
t1 <----- direct attention -----> t100   (no fade)

The catch: self-attention compares every token to every other token, so cost grows quadratically with sequence length.

length        attention work
1,000         1,000,000
10,000        100,000,000      (10x longer = 100x more work)
100,000       10,000,000,000

3. Mamba / State-Space Models — linear, stateful (the RNN comeback)

Mamba (2023) is the breakout state-space model (SSM). It returns to an RNN-like idea — carry a running hidden state and update it token by token — but fixes the old RNN flaws.

• Linear cost — O(N) in sequence length instead of O(N^2), so it is far cheaper on very long inputs (long documents, audio, genomes).
• Trains in parallel — unlike classic RNNs, SSMs can be computed with a parallel scan, so they are GPU-friendly.
• Selective state — Mamba makes the state update input-dependent (it chooses what to keep or forget), recovering much of the long-range modeling plain RNNs lacked.
• Fast inference — streams one step with a fixed-size state, instead of re-attending to a growing context.

Side by side

                   RNN / LSTM       Transformer         Mamba / SSM
-----------------  ---------------  ------------------  ------------------
Core idea          hidden state     self-attention      selective state
Processing         sequential       parallel            parallel (scan)
Cost vs length     linear           quadratic O(N^2)    linear O(N)
Long-range memory  fades            direct/global       strong, learned
Trains on GPU      poorly           excellently         well
Status             legacy           dominant (LLMs)     promising challenger

The honest status

• Transformers still dominate — nearly all major LLMs today are Transformers, with mature tooling and proven scaling.
• Mamba/SSMs are promising, especially for long-context and streaming; hybrids (attention + SSM layers) aim to combine both.
• Is the RNN dead? No — Mamba is essentially a modern, fixed RNN showing the recurrent idea is still competitive.

One-liner

RNN forgets, Transformer sees everything (but costs O(N^2)),
Mamba brings back a smart linear-time state. Attention made the
modern era; SSMs are the main challenger for very long sequences.