KV cache
Also: key-value cache, KV-cache, KV caching
During autoregressive decoding, each layer saves the key and value vectors of every previous token; a new token computes only its own query, key and value and attends over the cache. It trades memory, which grows linearly with sequence length and batch size, for far less compute.
bytes = 2 × layers × seq_len × n_kv_heads × head_dim × bytes_per_value × batch
Example
A 7B model with 32 layers, 32 KV heads of size 128 in FP16 needs 2·32·32·128·2 bytes = 0.5 MiB per token, so a 4,096-token context takes 2 GiB of cache per sequence.