A paper argues that the standard practice of training every LoRA expert at the same rank before merging is unnecessary.
The paper examines a common constraint in model merging, the technique of combining separately fine-tuned experts into a single model without retraining. According to the paper, standard practice trains every expert at the same LoRA rank, which couples the rank choice across experts even though each domain has its own best rank. The authors argue this coupling is a property of the merge method, not of merging itself, and support the claim with a systematic experiment across domains, ranks, model scales and merge algorithms.
The experimental setup, as described in the paper, involved training 36 experts on three domains — mathematics, instruction following and multilingual — at four ranks and three model scales, then merging them with six methods, both at uniform ranks and at each domain's own best rank. The results split by method family. Giving experts different ranks costs averaging and utility-based selection nothing, and gains them up to 1.8 percentage points of multi-domain accuracy. Conflict-resolution methods, by contrast, are inconsistent and can lose up to 4 percentage points. The paper traces the cause to how components are selected: merging keeps only a fixed share of the pooled spectral components, and methods that choose them by size discard many that would have reduced loss. The instruction adapter has the smallest singular values, so magnitude-based methods drop it almost entirely, and IFEval falls by 16 to 25 percentage points. Choosing components by their effect on loss instead keeps all three experts and leaves accuracy nearly flat across an 8x rank range — 0.7 percentage points, against 6.7 for TIES at 4B scale.
The paper names existing merge methods, including TIES, among those tested. The framing is that the uniform-rank convention has been inherited from the behavior of specific merge operators rather than being an inherent requirement of merging. The paper's conclusion is conditional: rank can be chosen per expert whenever the merge operator does not select components by magnitude. The document does not situate the work relative to other literature beyond the merge methods tested, so its relationship to earlier research on rank allocation cannot be assessed from the document provided.
The evidence comes from a single source, the paper itself, and no independent second source was found to corroborate the findings. The document does not state the authors' affiliations, the publication venue or the peer-review status, so those cannot be reported. The headline numbers — the 1.8 percentage point gain, the 16 to 25 point IFEval drop — should therefore be read as the authors' own reported results, not independently verified findings.