Tool 04
Watch scaled dot-product attention connect tokens through Q, K, and V.
Attention lets each token "look at" every other token and decide how much to borrow from each. It solves the core problem: when predicting the next word, which earlier words actually matter? This mechanism is what makes transformers context-aware.
In "The cat sat on the mat because it was tired," the word "it" probably refers to "cat" — not "mat." A model needs a way to connect distant words.
Before attention, RNNs processed tokens sequentially and struggled with long-range dependencies. Attention lets every token directly access every other token in one step.
Each token's vector is projected into three versions:
Query (Q): "What am I looking for?" Key (K): "What do I contain / advertise?" Value (V): "What information do I pass if selected?"
Attention scores = how well each Query matches each Key. Those scores weight the Values to produce the output.
The formula (in plain terms):
scores = (Q · Kᵀ) / sqrt(d_k) weights = softmax(scores) output = weights · V
Dividing by sqrt(d_k) prevents scores from getting too large (which would make softmax too sharp). Softmax turns scores into a probability distribution — each row sums to 1.
Rows are query tokens (who is looking). Columns are key tokens (who is being looked at). Brighter cells = higher attention weight.
Look for diagonal patterns (tokens attending to themselves) and off-diagonal spikes (pronouns linking to nouns, verbs linking to subjects). Real models have many heads — each head can learn different patterns.
Start with short sentences containing pronouns — watch where "it" or "they" attends. Try lists ("apple, banana, cherry") and see if later items attend to earlier ones.
Longer sentences produce bigger heatmaps — scroll horizontally and look for non-obvious connections.