A replication study re-implements DNO-Prct on two open-weight instruct models and reports length-controlled win rate improvements, with some of the gain retained using open-weight annotators.
A replication study re-implements DNO-Prct on two open-weight instruct models and reports length-controlled win rate improvements, with some of the gain retained using open-weight annotators.
A study posted on arXiv examines Direct Nash Optimization, a method that aligns a language model to a general preference function by approximating the Nash equilibrium of a two-player preference game. Its practical variant, DNO-Prct, iterates on-policy sampling, rating of samples by an LLM annotator together with a teacher gold response, large-margin preference-pair selection, and a DPO-style contrastive update. The authors re-implemented DNO-Prct independently and tested it on Llama 3.1 8B Instruct and Mistral 7B Instruct v0.1, using GPT-4o or a larger open-weight model of the same family as the annotator.
On AlpacaEval 2.0 with GPT-4o as judge, three iterations raised the length-controlled win rate against GPT-4o from 31.7 percent to 45.3 percent for Llama and from 6.8 percent to 18.4 percent for Mistral. The tuned 8B model scored above Llama 3.1 70B Instruct on every reported win rate. When same-family open-weight annotators replaced GPT-4o, the runs retained 62 percent of the length-controlled gain for Llama and 82 percent for Mistral, which the authors present as evidence that the rating step does not require a proprietary model.
The excerpt cuts off mid-sentence in its description of an ablation configuration, so the reported effects of removing the gold responses, removing the data look-back, or accepting only teacher-preferred pairs are not available here. The results also come from a single benchmark with GPT-4o as judge, which the same judge family used during training, a setup that can inflate measured gains. Independent evaluation on other benchmarks and with held-out judges would be needed before treating the numbers as general. The full paper should be checked for the ablation outcomes.
Source: Aligning Language Models with General Preferences: An Empirical Study of Direct…