10
8 minute interactive lab
The attention mechanism
Which tokens should influence this one?
YOU’LL LEAVE KNOWINGAttention builds a context-specific mixture of information.
THE EXPERIMENT
Let every token choose its context.
Move the controls. Watch the internal state respond. The explanation follows the behavior—not the other way around.
Runs the real small-scale equationThe displayed values are calculated from the formula shown in this lesson.
SCALED DOT-PRODUCT ATTENTIONChoose the token asking for context
“it” compares its query with every key. The dot products are divided by √d and normalized with softmax; all displayed weights sum to 100%.
largest non-self weight for “it”token · score 1.32
Attention builds a context-specific mixture of information.
Each token creates a query and compares it with the keys of other tokens. The resulting scores become weights used to blend their values into context tailored to the current token.
IN DEVELOPER TERMS
context = softmax(query · keys) × valuesGO DEEPER
Check the model against the source.
Connectionism optimizes for intuition, then points you to the rigorous treatment.
See what concepts are being built next.