Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-training

A team of researchers introduced a groundbreaking data processing framework called 'Data Darwinism' on the arXiv preprint server on February 13, 2025, to address the critical lack of systematic methodologies for enhancing the quality of training data used in artificial intelligence foundation models. The project aims to overcome the inherent 'learnability gap' often found in complex raw scientific literature, which frequently hampers the performance of large language models. By categorizing data through a tiered evolutionary lens, the researchers seek to create a self-sustaining cycle where advanced AI models are utilized to refine and generate superior datasets for the next generation of systems. At the heart of this research is the introduction of a ten-level taxonomy, ranging from L0 to L9, which conceptualizes the co-evolution of data and models. This hierarchy allows developers to track the transition from raw, unprocessed information to highly structured, machine-optimized knowledge. To prove the efficacy of this framework, the authors constructed 'Darwin-Science,' a massive 900-billion-token corpus designed specifically for scientific pre-training. This corpus represents the practical application of levels L0 through L5 of the taxonomy, demonstrating how raw text can be systematically elevated to more effective training material. The researchers emphasize that simply increasing the volume of data is no longer sufficient for pushing the boundaries of AI capabilities. Instead, the 'Data Darwinism' approach identifies that raw scientific text often contains noise or structural complexities that prevent models from learning effectively. By bridging this gap through their processing framework, the team provides a roadmap for shifting from quantity-driven data collection to a quality-focused evolutionary process. This development is expected to have significant implications for how future foundation models are trained, particularly in specialized fields like medicine, physics, and engineering.