Attention

Understand the term. Connect the concept.

A practical dictionary of 512 AI, machine learning, mathematics, and programming terms from HerbDev’s AI Vocabulary app. Start with a definition, explore an example, then follow the concepts behind it.

Linked concepts open in a new tab. Audio reads the definition and example.

Transformers · Level 3 of 5

A mechanism that weights information from candidate sources based on relevance.

Weights are computed from representations and need not be faithful explanations.

Example

A decoder emphasizes selected input positions when producing a token.

Listen to the definition and example

Audio transcript

Attention. A mechanism that weights information from candidate sources based on relevance. Weights are computed from representations and need not be faithful explanations. For example: A decoder emphasizes selected input positions when producing a token.

Explore this concept

A useful analogy

A learned mixing desk that assigns different volumes to available information.

Why it matters

This helps you trace how token representations exchange and transform information.

Technical detail

Scores from queries and keys are normalized and used to weight values. In scaled dot-product attention, output = softmax(QKᵀ/√d)V.

Common misconception

Attention weights are not automatically causal explanations for a prediction.

Start with

Related concepts

Quick recall question

Try answering before looking back at the definition.