How Transformers
Actually Work

1. The Attention Mechanism

Attention is the core operation of every Transformer. Before Transformers, sequence models like LSTMs processed tokens one at a time, left to right. The attention mechanism lets every token look at every other token simultaneously, computing a weighted sum based on relevance.

Q, K, V: Three Separate Projections

Each input token has an embedding vector. We multiply it by three learned weight matrices to produce three different vectors:

Why three separate matrices? Because the role a token plays when asking a question (Query) is different from the role it plays when being asked about (Key), which is again different from the information it provides (Value). A single projection cannot capture all three roles. This factorization gives the model flexibility to learn asymmetric relationships: token A attending to token B does not require B to attend equally to A.

The Attention Formula

Let's break down every component:

2. Numerical Walkthrough: 3 Tokens

Let's trace through the full attention computation with actual numbers. We have 3 tokens with d_k = 4 (tiny, but the math is identical to d_k = 64 in real models).

3. Multi-Head Attention

A single attention head can only capture one type of relationship at a time. Multi-head attention runs h parallel attention heads, each with its own W_Q, W_K, W_V projections, then concatenates and projects the results.

What Different Heads Learn

Research has shown that different heads in trained Transformers specialize in different linguistic patterns. This is not engineered — it emerges from training:

Concatenation and Output Projection

If the model dimension is d_model = 768 and we have h = 12 heads, each head works with d_k = d_model / h = 64 dimensions. After computing attention independently, the h output vectors (each 64-dim) are concatenated back to 768-dim and multiplied by W_O(768 × 768) to produce the final output. The total compute is the same as a single head with full dimensionality, but the model gets 12 different "perspectives" on the input.

4. The Full Transformer Block

A single Transformer layer ("block") wraps multi-head attention with feedforward networks, residual connections, and layer normalization. BERT-base stacks 12 of these blocks; GPT-3 stacks 96.

5. Positional Encoding

Attention is permutation-invariant: if you shuffle the input tokens, the attention computation produces the same output (just shuffled). The model has no inherent notion of word order. "The cat sat on the mat" and "mat the on sat cat the" would produce identical attention scores. Positional encodings inject order information.

Sinusoidal Positional Encoding (Original)

Each dimension of the positional encoding oscillates at a different frequency. Low dimensions change rapidly (high frequency), encoding fine position differences. High dimensions change slowly (low frequency), encoding coarse position. This design has a key property: the encoding of position pos + k can be represented as a linear function of the encoding at pos, allowing the model to learn relative position patterns easily.

6. From Tokens to Sentence Embeddings

A Transformer produces one output vector per token. But for tasks like semantic search, classification, or clustering, you need a single vector for the entire sentence. How do you collapse a sequence of vectors into one?

7. Scale: How Size Affects Quality

Transformer quality scales predictably with model size. The "Scaling Laws" research (Kaplan et al., 2020) showed that loss follows a power law with parameters, compute, and data. Doubling parameters gives diminishing but consistent returns.

8. Working Code: Attention in NumPy

Here is a complete, runnable implementation of scaled dot-product attention and multi-head attention in NumPy. No frameworks, no magic — every operation is explicit.

import numpy as np

def softmax(x, axis=-1):
    """Numerically stable softmax."""
    e_x = np.exp(x - np.max(x, axis=axis, keepdims=True))
    return e_x / e_x.sum(axis=axis, keepdims=True)

def scaled_dot_product_attention(Q, K, V):
    """
    Scaled dot-product attention.
    Q, K, V: arrays of shape (seq_len, d_k)
    Returns: (output, attention_weights)
    """
    d_k = K.shape[-1]

    # Step 1: Compute raw attention scores
    scores = Q @ K.T                # (seq_len, seq_len)

    # Step 2: Scale by sqrt(d_k)
    scores = scores / np.sqrt(d_k)  # prevents gradient vanishing in softmax

    # Step 3: Softmax to get attention weights
    weights = softmax(scores)       # each row sums to 1

    # Step 4: Weighted sum of values
    output = weights @ V            # (seq_len, d_k)

    return output, weights

# --- Demo with 3 tokens, d_k = 4 ---
X = np.array([
    [1.0, 0.0, 1.0, 0.0],   # "The"
    [0.0, 1.0, 0.0, 1.0],   # "cat"
    [1.0, 1.0, 0.0, 0.0],   # "sat"
])

# Using identity projection (Q = K = V = X)
output, weights = scaled_dot_product_attention(X, X, X)

print("Attention weights:")
print(np.round(weights, 3))
# [[0.506, 0.186, 0.307],
#  [0.186, 0.506, 0.307],
#  [0.307, 0.307, 0.387]]

print("\nOutput:")
print(np.round(output, 3))
# [[0.813, 0.494, 0.506, 0.186],
#  [0.494, 0.813, 0.186, 0.506],
#  [0.694, 0.694, 0.307, 0.307]]

def multi_head_attention(X, n_heads, d_model):
    """
    Multi-head attention from scratch.
    X: (seq_len, d_model)
    n_heads: number of attention heads
    d_model: model dimension (must be divisible by n_heads)
    """
    seq_len = X.shape[0]
    d_k = d_model // n_heads  # per-head dimension

    # Initialize random projection matrices (normally these are learned)
    np.random.seed(42)
    W_Q = np.random.randn(n_heads, d_model, d_k) * 0.1
    W_K = np.random.randn(n_heads, d_model, d_k) * 0.1
    W_V = np.random.randn(n_heads, d_model, d_k) * 0.1
    W_O = np.random.randn(d_model, d_model) * 0.1

    head_outputs = []

    for h in range(n_heads):
        # Project input to Q, K, V for this head
        Q = X @ W_Q[h]   # (seq_len, d_k)
        K = X @ W_K[h]   # (seq_len, d_k)
        V = X @ W_V[h]   # (seq_len, d_k)

        # Scaled dot-product attention
        output, _ = scaled_dot_product_attention(Q, K, V)
        head_outputs.append(output)  # (seq_len, d_k)

    # Concatenate all heads: (seq_len, n_heads * d_k) = (seq_len, d_model)
    concat = np.concatenate(head_outputs, axis=-1)

    # Final output projection
    result = concat @ W_O  # (seq_len, d_model)

    return result

# --- Demo: 3 tokens, 4 heads, d_model = 8 ---
X = np.random.randn(3, 8)  # 3 tokens, 8-dim embeddings
output = multi_head_attention(X, n_heads=4, d_model=8)
print(f"Input shape:  {X.shape}")    # (3, 8)
print(f"Output shape: {output.shape}")  # (3, 8)
print("\nEach token now contains information from all other tokens,")
print("aggregated through 4 different attention perspectives.")

def layer_norm(x, eps=1e-5):
    """Layer normalization over the last dimension."""
    mean = x.mean(axis=-1, keepdims=True)
    std = x.std(axis=-1, keepdims=True)
    return (x - mean) / (std + eps)

def gelu(x):
    """Gaussian Error Linear Unit activation."""
    return 0.5 * x * (1 + np.tanh(
        np.sqrt(2 / np.pi) * (x + 0.044715 * x**3)
    ))

def feed_forward(x, d_model, d_ff):
    """Position-wise feed-forward network."""
    np.random.seed(123)
    W1 = np.random.randn(d_model, d_ff) * 0.02
    b1 = np.zeros(d_ff)
    W2 = np.random.randn(d_ff, d_model) * 0.02
    b2 = np.zeros(d_model)

    hidden = gelu(x @ W1 + b1)  # (seq_len, d_ff)
    output = hidden @ W2 + b2    # (seq_len, d_model)
    return output

def transformer_block(x, n_heads, d_model, d_ff):
    """
    One complete Transformer encoder block.
    x: (seq_len, d_model)
    """
    # Sub-layer 1: Multi-head attention + residual + norm
    attn_out = multi_head_attention(x, n_heads, d_model)
    x = layer_norm(x + attn_out)     # Add & Norm

    # Sub-layer 2: Feed-forward + residual + norm
    ff_out = feed_forward(x, d_model, d_ff)
    x = layer_norm(x + ff_out)       # Add & Norm

    return x

# --- Full block: 3 tokens, 4 heads, d_model=8, d_ff=32 ---
X = np.random.randn(3, 8)
output = transformer_block(X, n_heads=4, d_model=8, d_ff=32)
print(f"Transformer block: {X.shape} -> {output.shape}")
# Transformer block: (3, 8) -> (3, 8)
# Same shape in, same shape out — stackable!

9. Key Papers

The Transformer story is told through a remarkably small set of papers. These are the ones every ML practitioner should read, in roughly chronological order.