Attention
Also: attention mechanism, scaled dot-product attention
Each query vector is compared with every key vector; the scaled scores pass through softmax to become weights, and the output is the weighted sum of the value vectors. Cross-attention takes queries from one sequence and keys and values from another.
Attention(Q, K, V) = softmax(QKᵀ / √dₖ) V
Example
If a query’s scaled scores against three keys are [2, 1, 0], softmax gives weights ≈ [0.67, 0.24, 0.09], so the output is mostly the first value vector.