Multi-head attention
Also: MHA, attention heads
The model dimension is split across h heads, each with its own query, key and value projections; their outputs are concatenated and mixed by a final linear layer. One head might follow syntax while another tracks coreference.
MultiHead(X) = Concat(head₁, …, head_h) W_O, headᵢ = Attention(XW_Qᵢ, XW_Kᵢ, XW_Vᵢ)
Example
The original Transformer uses d_model = 512 and h = 8 heads, so each head works in 512 / 8 = 64 dimensions and the total cost matches one full-width head.