Rational Neural Networks have Expressivity Advantages

Researchers published a groundbreaking study on February 12, 2026, demonstrating that neural networks with trainable low-degree rational activation functions offer significant advantages in expressiveness and parameter efficiency compared to modern activation functions like ELU, LeakyReLU, and ReLU, establishing new theoretical foundations for neural network architecture design. The study, currently available on the arXiv preprint server, represents a significant advancement in neural network theory, systematically comparing rational activation functions against a comprehensive list of modern alternatives including ELU, LeakyReLU, LogSigmoid, PReLU, ReLU, SELU, CELU, Sigmoid, SiLU, Mish, Softplus, Tanh, Softmin, Softmax, and LogSoftmax. Their analysis revealed that rational functions can achieve the same error targets with fewer parameters, potentially leading to more efficient neural network architectures across various computational domains. The researchers established approximation-theoretic separations, proving that networks built from standard activation functions require more resources to achieve the same level of accuracy as those using rational activation functions, with implications for future developments in artificial intelligence and machine learning.