A paper by Shourov Joarder and three other authors proposes multi-reward reinforcement learning to make large language model training more stable without external supervision. The authors report that combining two internal reward signals improves robustness on mathematical reasoning and code-generation benchmarks, with performance approaching that of supervised methods.
The approach builds on reinforcement learning from internal feedback (RLIF), which draws training signals from the model itself. That differs from reinforcement learning with verifiable rewards (RLVR), which often relies on human annotations or gold-standard solutions.
The paper identifies a weakness in existing RLIF methods: they typically use a single internal reward. According to the authors, such dependence can give rise to reward hacking, entropy collapse and weakened reasoning structure.
How multi-reward reinforcement learning balances training
The proposed framework combines two complementary signals: an answer-level reward using cluster voting and a completion-level reward using token-wise self-certainty. Rather than relying on one internal measure, it uses both to guide training.
To combine them, the researchers apply GDPO-based normalization, reducing imbalances between the reward scales. The normalization addresses how the two signals are balanced; a separate technique addresses training collapse.
That technique, KL-Cov regularization, targets low-entropy token distributions responsible for disproportionate entropy reduction. The authors describe it as preserving exploration and preventing collapse late in training.
Across the mathematical reasoning and code-generation benchmarks, the paper reports greater stability and robustness than previous unsupervised reinforcement learning approaches. Its multi-reward reinforcement learning framework also achieves results close to those of supervised RLVR, according to the authors.
The paper states that the authors plan to release the code.
Article produced with the assistance of artificial intelligence and reviewed by the editorial team.
