Demystifying Gen AI Models - Transformers Architecture : 'Attention Is All You Need'

The Transformer architecture demonstrated that carefully designed attention mechanisms — without the need for sequential recurrence — could model language and sequences more effectively and efficiently.

1. Transformers Replace Recurrence

• Traditional models such as RNNs and LSTMs processed data sequentially.
• Transformers use self-attention mechanisms to process all tokens simultaneously, enabling parallelisation, faster training, and better handling of long-range dependencies.

2. Self-Attention is Central

• Each token considers (attends to) all other tokens to gather contextual information.
• Attention scores are calculated between every pair of input tokens, capturing relationships irrespective of their position.

3. Multi-Head Attention Enhances Learning

• Rather than relying on a single attention mechanism, the model utilises multiple attention heads.
• Each head independently learns different aspects of relationships (such as syntax or meaning).
• The outputs from all heads are then combined to produce richer representations.

4. Positional Encoding Introduced

• As there is no recurrence, positional information must be introduced manually.
• Positional encodings (using sine and cosine functions of varying frequencies) are added to input embeddings to maintain the order of the sequence.

5. Encoder-Decoder Structure

• The model is composed of two main parts:
• Encoder: A stack of layers that processes the input sequence.
• Decoder: A stack of layers that generates the output, one token at a time (whilst attending to the encoder outputs).

6. Layer Composition

Each encoder and decoder layer includes:

• Multi-Head Self-Attention
• Feed-Forward Neural Network (applied to each token independently)
• Residual Connections and Layer Normalisation to stabilise training.

7. Scaled Dot-Product Attention

• Attention scores are calculated using dot products between queries and keys, scaled by the square root of the dimension to prevent excessively large values, before being passed through a softmax.

8. Simpler, Yet More Powerful

• Despite removing recurrence, the Transformer outperformed more complex architectures such as stacked LSTMs on translation tasks (for instance, English-German).
• Training is considerably quicker (thanks to parallelism), particularly on long sequences.

9. Key Achievement

• Transformers became the state-of-the-art model for many natural language processing tasks — paving the way for later innovations such as BERT, GPT, T5, and others.

The latest breakthrough in generative AI models is owed to the development of the Transformer architecture. Transformers were introduced in the Attention is all you need paper by Vaswani, et al. from 2017.