FP4 Explore, BF16 Train: Diffusion Reinforcement Learning via Efficient Rollout Scaling

A research team has proposed a novel method called "FP4 Explore, BF16 Train" to significantly reduce the computational cost of aligning large text-to-image diffusion models with human preferences via reinforcement learning (RL), as detailed in a paper published on arXiv on April 4, 2026 (ID: 2604.06916v1). This work addresses a critical bottleneck in the field, where scaling up the number of model evaluations, or "rollouts," is known to improve alignment quality but becomes prohibitively expensive for massive models like the 12-billion-parameter FLUX.1. The core innovation lies in a dual-precision training strategy. During the exploration phase, where the model generates many images to be evaluated by a reward model, the researchers use extremely low 4-bit floating-point (FP4) precision. This drastically cuts the memory and compute required for these forward passes. For the actual weight update step, the method switches back to the more stable Brain Floating Point 16 (BF16) format, preserving training stability and final model quality. This hybrid approach decouples the cost of exploration from the cost of learning. This technique directly tackles a major challenge in diffusion model alignment. Post-training with RL from human feedback (RLHF) or similar methods is a powerful way to make AI-generated images more aesthetically pleasing, safe, and aligned with user intent. However, each improvement cycle requires the model to generate thousands of images for assessment, a process that becomes a massive computational hurdle for state-of-the-art models with billions of parameters. The proposed method offers a pathway to achieve better alignment with far fewer resources, potentially making advanced model tuning more accessible. The implications are significant for the development of more controllable and higher-quality generative AI. By lowering the barrier to extensive RL-based fine-tuning, this work could accelerate progress in creating diffusion models that are not only more capable but also more reliably aligned with complex human values and safety standards, all while managing the soaring costs of AI research.