Manuscriva
Start free
Interactive demo

Watch Manuscriva review a landmark paper.

A reading group is writing a review of one of the most cited papers in machine learning. Follow the draft, ask Bloub, check every citation and inspect the data, all in one project.

NeurIPS 2017

Attention Is All You Need

Ashish Vaswani, Noam Shazeer, Niki Parmar and 5 others

Claims checked
4
Overclaim caught
1
Models compared
10
Runs used
0
Read the paper on arXiv
Transformer reading group
AMKOLB3 editors Read-only sample
Review draftSaved

Review draft · Transformer reading group

Attention without recurrence: a close reading of the Transformer

Abstract

Vaswani et al. replace recurrence and convolution in sequence transduction with attention alone. We summarise the architecture, check the headline translation results against the paper’s own tables, and note what the evidence does not yet show.

The architecture

The Transformer keeps the familiar encoder–decoder shape but builds both halves from stacked self-attention and position-wise feed-forward layers: six layers each, a model width of 512 and eight attention heads in the base configuration [1]. Every head computes scaled dot-product attention:

Attention(Q, K, V) = softmax(QKᵀ / √dₖ) · V

Because no position waits for the previous one, training parallelises across the whole sequence. Word order is supplied by fixed sinusoidal position encodings added to the embeddings.

Results

On WMT 2014 English-to-German, the big model reached 28.4 BLEU, more than 2 BLEU above the best earlier results, ensembles included [1]. It outperformed every previously reported model on both translation tasks. Training took 3.5 days on eight P100 GPUs.

Model

EN-DE BLEU

EN-FR BLEU

Training FLOPs

ConvS2S Ensemble

26.36

41.29

7.7 × 10¹⁹ / 1.2 × 10²¹

Transformer (base)

27.3

38.1

3.3 × 10¹⁸

Transformer (big)

28.4

41.0

2.3 × 10¹⁹

What the evidence does not show

Self-attention compares every position with every other, so its cost per layer grows with the square of sequence length. The paper’s translation sentences are short; long documents are left to future work. Convolutional models had already reduced sequential computation before this paper [2], so the contribution is removing recurrence and convolution entirely, not parallelism itself.

References

  1. Vaswani et al., 2017. doi:10.48550/arXiv.1706.03762
  2. Gehring et al., 2016. doi:10.48550/arXiv.1705.03122

An independent example. Manuscriva is not affiliated with the paper’s authors or their employers. The review text is our own; figures come from the published paper.

Now bring your own paper.

Upload your draft and sources, invite your co-authors, and let Manuscriva check the claims before your reviewers do.

Start with 3 free runs