The Use of AI Tools to Develop and Validate Q-Matrices

A research team publishing via the arXiv preprint server released a study in May 2025 detailing the effectiveness of large language models in developing and validating Q-matrices for cognitive diagnostic modeling (CDM). The researchers conducted this investigation to determine if artificial intelligence could automate the traditionally labor-intensive process of mapping assessment items to specific cognitive attributes, using a reading comprehension test as their primary case study. By providing multiple AI models with the same training materials previously given to human experts, the study sought to streamline complex educational measurement workflows that typically require significant time and specialized expertise. The core of the research involved a comparative analysis where AI-generated Q-matrices were measured against an established, validated benchmark created by Li and Suen in 2013. In cognitive diagnostic modeling, a Q-matrix serves as a fundamental framework that links test items to the underlying skills or knowledge components required to solve them. Traditionally, this matrix is meticulously handcrafted by subject matter experts, a process prone to subjectivity and high resource consumption. The study explored whether modern generative AI could replicate this expert logic with sufficient accuracy to be used in high-stakes educational data analysis. Preliminary findings from the study focused on the level of agreement between different AI models and the degree to which they converged on the validated human-made standards. While specific performance metrics varied across different language models, the research underscores a growing trend in using machine learning to handle the structural components of psychometrics. This advancement suggests that AI could eventually serve as a reliable supplementary tool or even a primary architect for structural diagnostic models, significantly reducing the bottleneck in creating sophisticated educational assessments and adaptive learning systems.