Evaluating Retrieval-Augmented Generation Variants for Natural Language-Based SQL and API Call Generation

Researchers Huynh and Lin published a technical paper on the arXiv preprint server on February 12, 2025, evaluating various Retrieval-Augmented Generation (RAG) strategies to improve how Large Language Models (LLMs) convert natural language into SQL queries and REST API calls within enterprise environments. The study addresses the growing need for reliable natural language interfaces that allow non-technical users to interact with complex corporate databases and software systems. By testing different RAG variants, the authors sought to overcome the current limitations LLMs face when navigating domain-specific schemas and handling both data retrieval and record modification tasks simultaneously. The research highlights a critical gap in current AI implementation: while general-purpose models like GPT-4 are proficient at writing generic code, they often struggle with the intricate, proprietary structures of enterprise systems. The paper explores how providing the model with relevant context—such as specific database schemas or API documentation—through a retrieval mechanism can significantly reduce syntax errors and logical hallucinations. This is particularly vital for "modification tasks," where incorrect API calls or SQL commands could lead to unintended data loss or system instability. Furthermore, the evaluation compares how different RAG architectures influence the accuracy of structured operation generation. By benchmarking these variants, the researchers provide a roadmap for developers aiming to build more robust AI agents capable of bridging the gap between human language and machine-executable commands. The findings suggest that a joint approach, which treats retrieval and modification as interconnected processes, is essential for achieving the level of precision required for professional-grade enterprise applications.