Overview
NL2SQL systems convert natural language questions into SQL queries. Evaluating them is tedious: you need to run generated queries against a real database, compare results, and understand *why* a query failed — not just whether it did.
This project built an automated evaluation agent that handles scoring, error classification, and feedback generation end to end.
Problem
Manual NL2SQL evaluation bottlenecks the development loop. Engineers write a question, see a wrong SQL output, manually inspect the schema and query, diagnose the issue, and write a fix. For hundreds of test cases, this is unsustainable.
Existing automated metrics (exact match, execution accuracy) tell you a query failed but not why. Improvement requires structured, actionable feedback.
Agent Design
The evaluation agent runs as a pipeline:
Query execution — run generated SQL against the target database, capture results or errors.
Result comparison — compare execution output against the expected answer. Handle edge cases: equivalent queries that produce the same result in a different order, partial matches on multi-row results.
Error classification — for failed queries, classify the failure type: schema misunderstanding, wrong join, incorrect filter, aggregation error, syntax error, etc.
Feedback generation — produce a natural language explanation of the error that can be fed back to the NL2SQL system or surfaced to the developer.
Benchmark scoring — aggregate results across test cases, track performance over time, flag regressions.
Results
The agent improved internal NL2SQL benchmark performance by approximately 10% through the feedback loop: structured error signals made it easier to identify systematic failure patterns and target them directly.
Evaluation that previously required manual inspection per test case now runs automatically, reducing the iteration cycle from hours to minutes.