Model merging has moved from a research curiosity to a routine step in open-model workflows. Teams that fine-tune open-weight models now combine the results by operating on the weights themselves: adding “task vectors”, averaging fine-tunes into a “soup”, or merging LoRA adapters with interference-aware methods such as TIES and DARE. The tooling is mature enough to be unremarkable. mergekit turns a short YAML recipe into a new checkpoint, Hugging Face PEFT ships merge methods for adapters, and merged checkpoints are a routine sight on the Hugging Face Hub.
Our read: weight-space programming is real, cheap and reproducible inside one model family, and still mostly empirical outside it. It composes capabilities that some fine-tune already learned; it does not manufacture new ones. Simple merges are production practice. Surgical weight editing and treating weights as training data are still research.
From Loss-Surface Geometry to a YAML File
The idea that you can do arithmetic on a trained network rests on one empirical observation about the loss surface, and it took about four years to go from that observation to everyday tooling.
2019–2020: Fine-tunes share a basin
Frankle et al. showed that networks sharing their early training are often joined by a straight, low-loss path in weight space (linear mode connectivity). Neyshabur et al. found the same for models fine-tuned from one pre-trained checkpoint. If two models sit in the same basin, the points between them are usable models too.
2022: Soups and task vectors
Model soups (Wortsman et al., March) averaged many fine-tunes of one model and often beat the single best fine-tune at no extra inference cost. Task arithmetic (Ilharco et al., December) named the unit of composition: a task vector τ = θfine-tuned − θbase. Add one to gain a skill, negate it to suppress a behavior, add several to build a multi-task model.
2022: Surgical edits
ROME and MEMIT (Meng et al.) went the other direction: instead of adding whole fine-tunes, they locate where a fact is stored and rewrite a few weight matrices to change it. Precise in demos, less dependable at scale.
2023 onward: Interference-aware merging and tooling
TIES-Merging (Yadav et al.) trims small deltas and resolves sign conflicts before adding; DARE (Yu et al.) randomly drops and rescales delta parameters. Both shipped in mergekit alongside linear averaging, task arithmetic and SLERP, and in Hugging Face PEFT for LoRA adapters. That is the point where merging stopped needing a research codebase.
Where It Fits: Four Uses
1. Combining specialist fine-tunes
Fine-tune one base on code and, separately, on math. With τcode = θcode − θbase and τmath = θmath − θbase, the merge θbase + τcode + τmath is often competitive on both without training on their union, usually keeping most (not all) of each specialist’s score. The appeal is organizational as much as technical: separate teams can fine-tune independently and combine at the end.
2. Mixing and swapping LoRA adapters
A LoRA adapter is already a compact weight delta, so adapters can be added, scaled or swapped per request. Merging two adapters into one, or folding an adapter into the base for serving, is the most common weight-space operation in production today.
3. Soups for free accuracy
When a hyperparameter sweep produces many fine-tunes of the same model, averaging the good ones is a cheap way to gain accuracy and robustness without paying ensemble inference costs: the soup is still one model.
4. Removing a behavior
Negating a task vector learned on, say, toxic text reduces that behavior while largely preserving the rest. It is a quick mitigation, not a guarantee: the behavior is suppressed along one direction in weight space, not proven gone.
What Is Settled and What Is Not
- Standard practice: task arithmetic, linear and spherical (SLERP) interpolation, model soups, and adapter merging with TIES or DARE, all within one base-model family.
- Active research, mixed reliability: ROME- and MEMIT-style surgical edits and weight-space unlearning.
- Early: hypernetworks that generate weights, and meta-models trained on collections of trained networks (“model zoos”) to predict generalization or produce new weights.
- Theory behind practice: that fine-tunes land in one low-loss basin is well supported empirically; why, and when it fails, is still being formalized.
What to Watch
- Cross-base merging. Models trained from different starting points are not in one basin. Permutation-alignment work such as Git Re-Basin narrows the gap, but merging unrelated models remains largely open.
- Evaluation hygiene. Merge recipes are cheap to search, which makes it easy to overfit a public benchmark. Leaderboard gains from merges deserve a held-out check.
- Editing at scale. Whether many surgical edits can coexist without degrading unrelated behavior is the test that would move editing from demo to tool.
- Weights as data. If meta-models trained on model zoos become reliable, “which model should I merge?” becomes a prediction problem rather than a search.
The durable takeaway: for teams that fine-tune open models, merging is now a cheap default to try before training on combined data. It is a complement to training, not a replacement for it.
Go deeper on MachinaLearning
- Glossary: Weight space programming (emerging-area entry with the task-vector diagram)
- Paper Gallery: Editing Models with Task Arithmetic
- Lesson: Model Optimization (LoRA adapters as weight deltas)
- Lesson: Gradient Descent (minima, basins and why merges work)
- Related glossary terms: fine-tuning, transfer learning, ensemble learning
Sources & Further Reading
- Ilharco et al., “Editing Models with Task Arithmetic” (2022)
- Wortsman et al., “Model soups” (2022)
- Yadav et al., “TIES-Merging” (2023)
- Yu et al., “Language Models are Super Mario” (DARE, 2023)
- Meng et al., “Locating and Editing Factual Associations in GPT” (ROME, 2022)
- Frankle et al., “Linear Mode Connectivity and the Lottery Ticket Hypothesis” (ICML 2020)
- Ainsworth et al., “Git Re-Basin” (2022)
- mergekit · Hugging Face PEFT: model merging · Hugging Face blog: Merge LLMs with mergekit