Three nested ideas: NLP is the field, the Transformer is the architecture that revolutionized it, and BERT is a landmark model built from part of that architecture. This guide goes from the problem space down to the math and the model internals.

Part 1 — NLP (Natural Language Processing)

What it is

NLP is the branch of machine learning concerned with getting computers to understand, interpret, and generate human language. The core difficulty is that language is ambiguous, context-dependent, and discrete (symbols, not numbers).

Common task families

• Text classification — sentiment, spam, topic (one label per document)
• Sequence labeling — NER (person/place/org), part-of-speech tagging (one label per token)
• Question answering — extractive (find the span) or generative (write the answer)
• Sequence-to-sequence — translation, summarization (input text to output text)
• Language modeling — predict the next (or missing) token; the basis of pre-training

The evolution that led to Transformers

1. Rule-based (hand-written grammars) — brittle, didn't scale.
2. Statistical / count-based (n-grams, TF-IDF) — no real understanding of meaning.
3. Word embeddings (Word2Vec, GloVe) — words become dense vectors where similar meanings sit close together. But each word had ONE fixed vector, ignoring context ("bank" of a river vs. a money bank).
4. RNNs / LSTMs — processed tokens sequentially with a hidden state. Two fatal weaknesses: sequential (can't parallelize, slow training) and long-range forgetting (early-token information decays).
5. Transformers (2017) — dropped recurrence entirely and used attention, which sees the whole sequence at once and runs in parallel.

Part 2 — The Transformer architecture

Introduced in "Attention Is All You Need" (Vaswani et al., 2017). It is built from two stacks — an encoder and a decoder — but the real engine is attention.

2.1 From text to vectors

• Tokenization — text is split into tokens. Modern models use subword schemes (BPE, WordPiece, SentencePiece) so rare words become pieces of known tokens (e.g. "tokenization" to "token" + "##ization"). This keeps the vocabulary bounded while handling any word.
• Embeddings — each token id maps to a learned vector of size d_model (e.g. 768 for BERT-base).
• Positional encoding — because attention has no inherent sense of order, we add position information to each embedding. The original paper used fixed sine/cosine functions; BERT learns position embeddings instead. Either way, "dog bites man" and "man bites dog" become distinguishable.

2.2 Self-attention — the core mechanism

For every token, self-attention asks: "which other tokens should I pay attention to, and how much?" It uses three learned projections of each token vector:

• Query (Q) — what this token is looking for
• Key (K) — what each token offers
• Value (V) — the actual content each token passes along

The attention score between two tokens is the dot product of one's Query with the other's Key. Scores are scaled, softmaxed into weights, and used to take a weighted sum of Values:

Attention(Q, K, V) = softmax( (Q . K^T) / sqrt(d_k) ) . V

• Q . K^T — similarity of every token to every other token (an n x n matrix)
• / sqrt(d_k) — scaling to keep gradients stable
• softmax — turns scores into a probability distribution (attention weights)
• . V — each token's new representation is a blend of all tokens' Values, weighted by relevance

Result: each token's output vector is contextualized by the entire sequence, in a single parallel operation.

2.3 Multi-head attention

Instead of one attention, the model runs several ("heads") in parallel, each with its own Q/K/V projections. Different heads learn different relationships — one may track syntax, another coreference, another nearby words. Their outputs are concatenated and projected back to d_model.

head_i    = Attention(Q.Wq_i, K.Wk_i, V.Wv_i)
MultiHead = Concat(head_1, ..., head_h) . Wo

2.4 The rest of an encoder block

Each encoder layer is multi-head self-attention, then a position-wise feed-forward network, each wrapped with a residual connection and layer normalization:

x = LayerNorm(x + MultiHeadSelfAttention(x))
x = LayerNorm(x + FeedForward(x))

• Residual connections (x + ...) let gradients flow through deep stacks and preserve the original signal.
• Layer norm stabilizes training.
• Feed-forward adds non-linear processing capacity per token.

Stack N of these (BERT-base uses 12) and you have the encoder.

2.5 Encoder vs decoder

Both are stacks of Transformer layers. The entire difference is what each token is allowed to look at through attention. One-line version: an encoder reads and understands a whole input at once (sees all tokens, both directions); a decoder generates output one token at a time (sees only what came before).

Encoder — bidirectional attention. Every token attends to every other token, left and right. When processing a word it already sees the whole sequence, so it has full context:

Sentence: "I  deposited  cash  at  the  bank"
                                          ^
When encoding "bank" it CAN see "deposited" and "cash"
(left) AND anything to the right. Full context -> knows
"bank" = financial, not riverside.

This is why encoders are great at understanding — classification, NER, retrieval. BERT is encoder-only.

Decoder — masked (causal) attention. Each token may only attend to tokens before it; future positions are masked out. This is required for generation: when predicting the next word, the model must not peek at the answer.

Generating: "I  deposited  cash  at  the  ___"
                                           ^
When predicting this next word it can see ONLY
"I deposited cash at the" — never the future.
Predict -> "bank"

This left-to-right (causal) constraint is why decoders are great at generating text. GPT is decoder-only.

Why the masking matters. If a generator could see the future during training it would cheat — just copy the next word instead of learning to predict it. The mask forces genuine prediction. Encoders never generate, so they have no reason to hide the future; seeing everything gives richer understanding. That single choice cascades into the whole BERT-vs-GPT comparison:

                 Encoder (BERT)          Decoder (GPT)
Attention        bidirectional           masked / causal (left-only)
Sees the future  yes                     no
Core skill       understanding           generation
Training task    fill in masked words    predict the next word
Output           representations         text, token by token

The third option: encoder-decoder. The original 2017 Transformer used both together for sequence-to-sequence tasks like translation. The encoder reads the full source sentence into a representation; the decoder generates the target one token at a time and uses cross-attention to look back at the encoder output while it writes:

"je suis fatigue"  ->  [ENCODER] -> representation
                                      |  (cross-attention)
                                      v
                      [DECODER] -> "I" -> "am" -> "tired"

Models like T5 and BART use this full encoder-decoder setup. Analogy: the encoder is like reading a sentence with your eyes free to scan the whole line; the decoder is like writing a sentence word by word, knowing only what you have written so far.

2.5b The encoding process and the decoding process

Encoder and decoder are the components; encoding and decoding are the processes they perform. Here is each in detail.

Encoding (building a representation)

Goal: turn a sequence of input tokens into a set of context-aware numeric vectors that capture meaning. Encoding reads the entire input at once and produces a representation the rest of the system can use — it does not produce words.

Step by step:

1. Tokenize the input text into subword tokens.
2. Embed each token into a vector of size d_model, then add its positional encoding so order is preserved.
3. Contextualize through N encoder layers. In each layer, bidirectional self-attention lets every token gather information from every other token (left and right), followed by a feed-forward network, with residual connections and layer norm.
4. Output one vector per input token — each now encodes that token in context. The word "bank" ends up with a different vector in "river bank" than in "savings bank".

Input tokens  ->  embeddings + positions  ->  [N encoder layers]  ->  contextual vectors
"the river bank"                                                      [v_the, v_river, v_bank*]
                                                 (* v_bank reflects "river" context)

Key properties of encoding:

• Parallel — all tokens are processed at once (no left-to-right loop), so it is fast.
• Bidirectional — each token sees the whole sequence, giving the richest possible context.
• Output is a representation, not text — the vectors feed a downstream head (classifier, span predictor) or a decoder.

Where it is used: encoder-only models (BERT) are pure encoding — the final vectors go straight to a task head for classification, NER, retrieval, or extractive QA.

Decoding (generating output)

Goal: produce an output sequence one token at a time. Decoding is inherently sequential and autoregressive: each new token is predicted from everything generated so far (and, in a seq2seq model, from the encoder representation via cross-attention).

Step by step (one generation step):

1. Take the tokens generated so far (plus a start token at the beginning).
2. Run them through N decoder layers. Masked self-attention ensures each position sees only earlier positions — never the future.
3. In an encoder-decoder model, cross-attention lets the decoder look back at the encoder output to stay faithful to the input.
4. The final layer produces logits over the whole vocabulary — a score for every possible next token.
5. A softmax turns logits into probabilities; a decoding strategy picks the next token.
6. Append the chosen token to the sequence and repeat until an end-of-sequence token (or length limit).

step 1:  [START]                  -> predict "I"
step 2:  [START] I                -> predict "am"
step 3:  [START] I am             -> predict "tired"
step 4:  [START] I am tired       -> predict [END]  (stop)

Decoding strategies (how step 5 chooses a token) strongly affect output quality:

Strategy      How it picks                       Effect
------------  ---------------------------------  --------------------------
Greedy        always the highest-probability     deterministic, can be dull
Beam search   keeps top-k partial sequences      better for translation
Top-k         sample from the k most likely      adds controlled variety
Top-p         sample from smallest set whose     balances quality/diversity
(nucleus)     probability mass >= p
Temperature   scales logits before softmax       low = safe, high = creative

Key properties of decoding:

• Sequential / autoregressive — token t depends on tokens 1..t-1, so it cannot be fully parallelized at inference time.
• Causal (masked) — the future is hidden, which is what makes genuine prediction possible.
• Output is text — logits over the vocabulary, sampled one token at a time.

Where it is used: decoder-only models (GPT) are pure decoding driven by a prompt; encoder-decoder models (T5, BART) encode the input once, then decode the output step by step using cross-attention.

2.5a Is the Transformer a "superset" of encoder and decoder?

Almost — but it is more accurate to call the Transformer the parent category than a simple sum. The original 2017 Transformer did contain both an encoder stack and a decoder stack:

Transformer (2017, "Attention Is All You Need")
 - Encoder stack   (bidirectional self-attention)
 - Decoder stack   (masked self-attention + cross-attention)

But "superset" implies Transformer = encoder + decoder always. That holds only for the original model. Today "Transformer" names the architecture family, and most models use just one part of it:

Encoder-only     -> BERT        (uses just the encoder stack)
Decoder-only     -> GPT, LLaMA  (uses just the decoder stack)
Encoder-decoder  -> T5, BART    (uses both, like the original)

All three are Transformers — so a decoder-only GPT is a Transformer without an encoder. The thing they genuinely all share is not "encoder + decoder" but self-attention (plus feed-forward layers, residual connections, layer norm, and positional encoding). Encoder and decoder are two variants of a Transformer block that differ only in their attention rules (bidirectional vs. masked + cross-attention).

            Transformer (architecture: self-attention based)
           /            |                 \
   Encoder-only   Decoder-only      Encoder-decoder
     (BERT)        (GPT)              (T5, BART)

So: the original Transformer model is encoder + decoder together; the Transformer architecture is the parent category, with encoder-only, decoder-only, and encoder-decoder as its three forms.

2.6 Three model families

Encoder-only      BERT, RoBERTa     understanding (classify, NER, QA)
Decoder-only      GPT, LLaMA        generation (chat, completion)
Encoder-decoder   T5, BART          seq2seq (translate, summarize)

Part 3 — BERT in depth

BERT = Bidirectional Encoder Representations from Transformers (Google, 2018). It is the encoder stack only, pre-trained to understand language, then fine-tuned for specific tasks.

3.1 Why "bidirectional" matters

GPT-style models read left-to-right, so when encoding a word they only know what came before. BERT sees both directions simultaneously. For understanding tasks, right-hand context matters as much as left:

"I made a deposit at the [bank]."     -> financial institution
"I sat on the [bank] of the river."   -> riverside

Only full both-side context disambiguates "bank." BERT's bidirectionality is its defining strength.

3.2 Special tokens and input format

• [CLS] — prepended to every input; its final hidden vector is used as the aggregate representation for classification.
• [SEP] — separates two segments (sentence A / sentence B) and ends the input.
• Each token's input = token embedding + segment embedding (A or B) + position embedding.

3.3 Pre-training objective 1 — Masked Language Modeling (MLM)

Randomly select ~15% of tokens and replace them, then predict the originals from both-side context. This is what forces bidirectionality (a left-to-right model could trivially copy the answer from the future). The 15% are handled as:

• 80% replaced with [MASK]
• 10% replaced with a random token
• 10% left unchanged

(The 20% non-[MASK] trick prevents the model from only learning to fill in the literal [MASK] token, which never appears at fine-tuning time.)

Input:  "the man went to the [MASK] to buy milk"
Target: predict "store"

3.4 Pre-training objective 2 — Next Sentence Prediction (NSP)

Given sentence A and sentence B, predict whether B truly followed A in the corpus (50% real, 50% random). This teaches inter-sentence relationships, useful for QA and natural language inference. (Later work, e.g. RoBERTa, found NSP largely unnecessary and dropped it for better results — worth knowing.)

3.5 Fine-tuning — the transfer-learning payoff

Pre-training is expensive and done once on huge unlabeled corpora (BooksCorpus + Wikipedia). Then for your task you add a small output layer and train briefly on a small labeled dataset — the heavy lifting is already in the pre-trained weights.

Task                   Use this output
---------------------  --------------------------------------------
Classification         final [CLS] vector -> softmax
Named entity (NER)     each token's final vector -> per-token label
Question answering     predict start & end token positions of answer
Sentence-pair tasks    [CLS] vector with A [SEP] B input

3.6 Model sizes

Model        Layers  Hidden (d_model)  Heads  Parameters
-----------  ------  ----------------  -----  ----------
BERT-base    12      768               12     110M
BERT-large   24      1024              16     340M

3.7 BERT vs GPT at a glance

Aspect        BERT                      GPT
------------  ------------------------  --------------------------
Architecture  Encoder-only              Decoder-only
Direction     Bidirectional             Left-to-right (causal)
Trained on    MLM + NSP                 Next-token prediction
Best at       Understanding (classify)  Generation (write text)
Typical use   Fine-tune per task        Prompt / few-shot

3.8 Practical: fine-tuning for a Q&A / FAQ dataset — BERT or GPT?

A common real-world question: "I have some Q&A or FAQ data — should I fine-tune BERT or GPT?" It depends on what you want the system to do with that data.

Use BERT (encoder) when the answer already exists and you want to find/match it:

• Extractive QA — the answer is a span inside a document; BERT predicts the start/end positions. Best when answers must be grounded and verbatim.
• FAQ matching / retrieval — embed questions (e.g. Sentence-BERT) so similar ones sit close, then do similarity search to find the right FAQ entry. Cheap to run and does not hallucinate.
• Intent classification — route a question to one of N known FAQ categories.

Use GPT (decoder) when you want to generate a fluent, conversational answer:

• Generative QA / chatbot — produces natural answers in its own words and handles open-ended phrasing. Risk: can hallucinate if the answer isn't grounded.

Recommended approach for a typical FAQ/Q&A set:

1. Try RAG (Retrieval-Augmented Generation) before fine-tuning. Embed your FAQ entries with a BERT-style embedding model, retrieve the top matches for a user question, and either return them or feed them to an LLM to phrase the answer. No training, easy to update, stays grounded.
2. Fine-tune when you have enough quality pairs (roughly hundreds to thousands), need a specific tone/format, or want lower latency/cost: grounded extractive answers → fine-tune BERT; branded generative answers → fine-tune a smaller GPT-style model.

Rule of thumb:  BERT = find/understand the answer
                GPT  = write the answer

For most FAQ bots: RAG + a BERT-family embedding model is the
best starting point; add fine-tuning only if retrieval alone
isn't enough.

Part 4 — Putting it together

NLP                 the field (work with human language)
 - Transformer       the architecture (self-attention, parallel)
     - Encoder-only  -> BERT   (understanding)
     - Decoder-only  -> GPT    (generation)
     - Enc-decoder   -> T5/BART (seq2seq)

• NLP sets the goals and tasks.
• Transformers made modern NLP work by replacing recurrence with self-attention — parallelizable and able to model long-range dependencies.
• BERT specialized the encoder half for deep bidirectional understanding, pre-trained with masked-language modeling and transferred to many tasks via lightweight fine-tuning.

Mental model to remember

• Attention = a learned, content-based lookup: Query asks, Keys match, Values answer.
• Encoder = understand; Decoder = generate.
• BERT = fill-in-the-blank pre-training that produces reusable, context-aware representations.