Positional encoding
Also: positional embedding, position embedding
Self-attention treats its input as an unordered set, so order must be injected explicitly, either with fixed sinusoids of different frequencies or with learned per-position vectors. Rotary embeddings (RoPE) instead rotate queries and keys by a position-dependent angle.
PE(pos, 2i) = sin(pos / 10000^(2i/d)), PE(pos, 2i+1) = cos(pos / 10000^(2i/d))
Example
For position 1, the first two sinusoidal dimensions are sin(1) ≈ 0.841 and cos(1) ≈ 0.540; at position 0 they are 0 and 1. Higher dimensions oscillate more slowly.