Unleashing Low-Bit Inference on Ascend NPUs: A Comprehensive Evaluation of HiFloat Formats

Researchers have published a comprehensive evaluation of HiFloat formats for low-bit inference on Ascend NPUs in a new arXiv paper released on February 26, 2026, addressing the growing need for efficient precision in large language models. The study introduces HiFloat, comprising HiF8 and HiF4 formats specifically designed for Huawei's Ascend Neural Processing Units, examining their performance across weight-activation and KV-cache tasks. Through rigorous benchmarking, the researchers discovered that traditional INT8 formats perform best with narrow-range data, while floating-point formats like HiFloat excel when handling high-variance data. The evaluation also revealed that in 4-bit regimes, HiF4's hierarchical approach offers significant advantages over existing solutions, potentially enabling more efficient deployment of increasingly large AI models. This represents a significant advancement in the optimization of AI inference hardware and software efficiency as computational demands continue to grow. As large language models scale to unprecedented sizes, the trade-off between precision and computational efficiency becomes increasingly critical. The HiFloat formats provide a promising approach to balance these competing demands, particularly for specialized hardware like Ascend NPUs. This research contributes to the broader field of AI optimization, offering practical solutions for real-world deployment challenges in the era of massive neural networks.