Attention Is All You Need (2017)

Original paper: Vaswani et al., Attention Is All You Need, NeurIPS 2017 — arXiv:1706.03762

Problem

Before this paper, the best sequence models (translation, language modeling) used recurrent networks (RNNs/LSTMs) that process a sentence one word at a time. That's slow to train (can't parallelize across the sequence) and struggles to connect words that are far apart.

Key idea

Replace recurrence entirely with self-attention: for every word, look directly at every other word in the sequence and learn how much each one matters for understanding this word — regardless of distance. This is the Transformer architecture.

The core computation is:

\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V

Where $Q$ (query), $K$ (key), and $V$ (value) are learned projections of the input. Intuitively: for each word's query, compare it against every word's key to get attention weights, then use those weights to blend the values.

Method, at a glance

Stack of encoder layers (self-attention + feed-forward) and decoder layers (self-attention + cross-attention + feed-forward).
Multi-head attention: run several attention computations in parallel with different learned projections, so the model can attend to different kinds of relationships at once.
Since there's no recurrence, positional encodings are added to the input so the model knows word order.

Results

Transformers beat the prior state of the art on English-to-German and English-to-French translation, while training significantly faster because the whole sequence can be processed in parallel.

Why it matters

This architecture is the foundation of essentially all modern large language models (GPT, BERT, Claude, and others). Understanding self-attention is close to a prerequisite for understanding how today's AI systems work.

This is a summary for learning purposes — always read the original paper for full details and cite it properly if you build on it.