Точка Синхронізації

Researchers have introduced a new methodology called Difficulty-Estimated Policy Optimization (DEPO) on the arXiv preprint server this week to address critical efficiency flaws in the training of Large Reasoning Models (LRMs). As models like DeepSeek-R1 increasingly rely on scaling inference-time compute through Group Relative Policy Optimization (GRPO), developers found that standard training techniques often stall when faced with tasks of extreme difficulty levels. By introducing a difficulty-aware mechanism, the researchers aim to stabilize the learning process and ensure that gradient signals remain robust even when the model encounters problems that are either significantly below or above its current reasoning capabilities. The core issue identified in the study is the phenomenon of gradient signal attenuation within the GRPO framework. In standard group-based reinforcement learning, the model improves by comparing multiple outputs for the same prompt to determine an 'advantage' score. However, when a problem is too trivial, all outputs are identical and correct; conversely, when a problem is too complex, all outputs tend to be equally incorrect. In both scenarios, the lack of variance between group members causes the advantage signal to disappear, leaving the model's learning updates vulnerable to random noise and computational inefficiencies. To overcome these hurdles, the DEPO approach incorporates an estimation of task difficulty to calibrate the optimization process. By accounting for the inherent complexity of a prompt, the algorithm can maintain a meaningful learning signal where traditional GRPO would otherwise fail. This advancement is particularly relevant for the next generation of LRMs, which are designed to 'think' longer during inference to solve complex mathematical or logical problems. The research suggests that refining how these models interpret feedback across varying difficulty spectra is essential for the continued scaling of artificial intelligence reasoning capabilities.