F-GRPO: Don't Let Your Policy Learn the Obvious and Forget the Rare

A team of AI researchers published a new technical paper on the arXiv preprint server on February 11, 2025, introducing Filtered Group Relative Policy Optimization (F-GRPO) to address critical sampling failures in Reinforcement Learning with Verifiable Rewards (RLVR). The researchers identified a systemic flaw where standard group-sampling methods cause Large Language Models to over-prioritize common, obvious solutions while neglecting rarer but correct reasoning paths. By refining how rewards are calculated within small sample groups, the authors aim to prevent models from 'forgetting' complex or infrequent logic during the fine-tuning process. The core of the problem lies in the computational limitations of Group Relative Policy Optimization (GRPO), a popular technique used in training models like DeepSeek-R1. When group sizes are restricted due to hardware constraints, the mathematical estimation of 'advantage' becomes skewed. If a group of generated responses contains mostly common errors and only a few rare, high-quality solutions, the training algorithm may fail to provide enough signal to reinforce those rare successes. Consequently, the model begins to converge on mediocre, frequent patterns while losing the ability to generate sophisticated, out-of-distribution reasoning. To bridge this gap, the F-GRPO framework introduces a filtered approach to group sampling that ensures rare-correct trajectories are not overshadowed by the statistical noise of mixed-quality responses. The researchers derived new probability calculations to better estimate advantages when operating under hardware-constrained environments. This development is significant for the broader AI field, as it suggests that smaller, more efficient training runs can achieve state-of-the-art reasoning performance if the underlying policy optimization algorithm is mathematically tuned to value quality over sheer frequency.