Revisiting the Shape Convention of Transformer Language Models

Researchers have published a new technical study on the arXiv preprint server in February 2025 challenging the traditional structural design of dense Transformer language models to improve computational efficiency and performance. The paper, indexed as arXiv:2602.06471v1, re-evaluates the long-standing architectural convention where each model layer consists of an attention module followed by a Feed-Forward Network (FFN). This investigation was prompted by recent mathematical evidence suggesting that the standard 'narrow-wide-narrow' Multi-Layer Perceptron (MLP) configuration may not be the most effective way to approximate complex functions in large-scale AI systems. For years, the industry has relied on a consistent shape convention for Transformers, typically allocating the majority of a model's parameters to the MLP component using expansion ratios between 2 and 4. This traditional approach forces the data through a bottleneck, expanding it into a higher-dimensional space before compressing it back down. However, the researchers propose a shift toward residual 'wide-narrow-wide' or 'hourglass' MLP structures. This alternative design aims to leverage superior function approximation capabilities that have been overlooked during the rapid scaling of modern Large Language Models (LLMs). By revisiting these foundational architectural choices, the study suggests that the current path of AI development may be optimized further by changing how layers process information. The transition to an hourglass-style MLP could potentially allow models to achieve better results with the same or fewer parameters, addressing the growing demand for more efficient AI hardware utilization. This research represents a significant pivot in deep learning theory, questioning whether the 'standard' Transformer recipe inherited from the original 2017 designs remains the optimal blueprint for the next generation of artificial intelligence.