Attention Is All You Need
Ashish Vaswani, Noam Shazeer, Niki Parmar and 5 others
- Claims checked
- 4
- Overclaim caught
- 1
- Models compared
- 10
- Runs used
- 0
A reading group is writing a review of one of the most cited papers in machine learning. Follow the draft, ask Bloub, check every citation and inspect the data, all in one project.
Ashish Vaswani, Noam Shazeer, Niki Parmar and 5 others
Review draft · Transformer reading group
Vaswani et al. replace recurrence and convolution in sequence transduction with attention alone. We summarise the architecture, check the headline translation results against the paper’s own tables, and note what the evidence does not yet show.
The Transformer keeps the familiar encoder–decoder shape but builds both halves from stacked self-attention and position-wise feed-forward layers: six layers each, a model width of 512 and eight attention heads in the base configuration [1]. Every head computes scaled dot-product attention:
Attention(Q, K, V) = softmax(QKᵀ / √dₖ) · V
Because no position waits for the previous one, training parallelises across the whole sequence. Word order is supplied by fixed sinusoidal position encodings added to the embeddings.
On WMT 2014 English-to-German, the big model reached 28.4 BLEU, more than 2 BLEU above the best earlier results, ensembles included [1]. It outperformed every previously reported model on both translation tasks. Training took 3.5 days on eight P100 GPUs.
Model | EN-DE BLEU | EN-FR BLEU | Training FLOPs |
|---|---|---|---|
ConvS2S Ensemble | 26.36 | 41.29 | 7.7 × 10¹⁹ / 1.2 × 10²¹ |
Transformer (base) | 27.3 | 38.1 | 3.3 × 10¹⁸ |
Transformer (big) | 28.4 | 41.0 | 2.3 × 10¹⁹ |
Self-attention compares every position with every other, so its cost per layer grows with the square of sequence length. The paper’s translation sentences are short; long documents are left to future work. Convolutional models had already reduced sequential computation before this paper [2], so the contribution is removing recurrence and convolution entirely, not parallelism itself.
The paper, the draft, the data and the audit, kept together and private to the project.
Our close reading of the paper, written together with live co-editing.
The source paper. Text and references extracted for evidence checks.
BLEU scores and training cost for ten models, transcribed from Table 2.
Four claims checked: two supported, one contradicted, one waiting for a source.
Every reference is checked against Crossref and OpenAlex, and every claim against the source text. Saved example results; no live lookup runs here.
It outperformed every previously reported model on both translation tasks.
Attention Is All You Need
Ashish Vaswani et al. · 10.48550/arXiv.1706.03762
Title, authors, year and arXiv record match.
True for English-to-German. On English-to-French, the ConvS2S ensemble scored 41.29 BLEU against the Transformer’s 41.0. The paper itself claims the best single-model score there, not the best overall.
| Model | EN-FR BLEU |
|---|---|
| Deep-Att + PosUnk | 39.20 |
| GNMT + RL | 39.92 |
| ConvS2S | 40.46 |
| MoE | 40.56 |
| Deep-Att + PosUnk Ensemble | 40.40 |
| GNMT + RL Ensemble | 41.16 |
| ConvS2S Ensemble | 41.29 |
| Transformer (base) | 38.10 |
| Transformer (big) | 41.00 |
Ten models, transcribed from Table 2 of the paper. Choose a task to compare translation quality with training cost.
Transformer Single model Ensemble
The base model got there with about 23× less training compute than the ConvS2S ensemble.
| Model | EN-DE | EN-FR |
|---|---|---|
| ByteNet | 23.75 | — |
| Deep-Att + PosUnk | — | 39.2 |
| GNMT + RL | 24.6 | 39.92 |
| ConvS2S | 25.16 | 40.46 |
| MoE | 26.03 | 40.56 |
| Deep-Att + PosUnk Ensemble | — | 40.4 |
| GNMT + RL Ensemble | 26.3 | 41.16 |
| ConvS2S Ensemble | 26.36 | 41.29 |
| Transformer (base) | 27.3 | 38.1 |
| Transformer (big) | 28.4 | 41.0 |
An independent example. Manuscriva is not affiliated with the paper’s authors or their employers. The review text is our own; figures come from the published paper.
Upload your draft and sources, invite your co-authors, and let Manuscriva check the claims before your reviewers do.