(1) One thing after another, only better

Arguably the most influential paper on AI published in the last 10 years, “Attention is All You Need”* provided the necessary new thinking to unstick how neural net ideas were applied, introducing the Transformer.

For strategists, two things to note: first, the simplicity of the new idea, i.e. how self-attention can be used to generate a representation of a sequence, a powerful abstraction; and second, that it took a team of brilliant colleagues, not just an individual, to advance science. Referencing 9 of the 11 authors, Google highlighted their “equal contribution.” Each may not be a household name, but their contributions are used everyday.

“Self-attention, sometimes called intra-attention is an attention mechanism relating different positions of a single sequence in order to compute a representation of the sequence. Self-attention has been used successfully in a variety of tasks including reading comprehension, abstractive summarization, textual entailment and learning task-independent sentence representations.

“End-to-end memory networks are based on a recurrent attention mechanism instead of sequence-aligned recurrence and have been shown to perform well on simple-language question answering and language modeling tasks.

“To the best of our knowledge, however, the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.”*

* Vaswani, A. et al, “Attention is All You Need,” arXiv:1706.03762 [cs.CL] (original June 12 2017); https://doi.org/10.48550/arXiv.1706.03762...