The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models

In a recent study published as a paper on arXiv, researchers conduct a comprehensive review of existing literature that evaluates the syntactic capabilities of Transformer-based language models, specifically focusing on their interpretability in understanding syntactic knowledge. The paper, entitled 'The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Models,' systematically examines 337 scholarly articles, which collectively report on 1,015 results related to model performance across various syntactic phenomena. This extensive review highlights the breadth of methodologies and datasets used in this area of research, underscoring the maturity and sophistication of current interpretability strategies. A significant finding of this review is the prevalent focus on the English language within the field. Despite the promise of diverse research methods and phenomena, the study reveals a tendency to concentrate on English, which limits the generalizability of results across different languages. Moreover, the review points out the predominance of a single model type, BERT, which, while seminal, suggests a lack of exploration into the broader ecosystem of Transformer-based models. This indicates an opportunity for researchers to expand their focus to encompass a wider variety of languages and a broader spectrum of models. The reviewed articles assess the syntactic competence of these models through a vast array of syntactic phenomena, including relatively straightforward tasks such as part of speech tagging. While these simpler tasks provide a valuable entry point for evaluating syntactic knowledge, they may not adequately capture the full complexity of language models' understanding. Consequently, future research is encouraged to tackle more challenging syntactic phenomena to deepen insights into model capabilities and limitations. In conclusion, the study serves as both a resource and a call to action for the computational linguistics community. By cataloging current research efforts, it provides a structured overview of the field's achievements and identifies areas ripe for exploration. Particularly, the study advocates for a more balanced approach that incorporates more non-English languages and models beyond BERT, in order to enrich and diversify the understanding of language model interpretability.