BIRD-SQL Benchmark Tests LLM Text-to-SQL Skills

SELECT account.account_id
FROM account
WHERE account.type = 'OWNER';

Turning a plain-English question into a working SQL query is one of the more practical tests of what a large language model can do with structured data. The BIRD-SQL benchmark, documented at https://bird-bench.github.io/, was built to measure exactly that, with a focus on the messy realities of production databases rather than clean academic examples.

What the Benchmark Covers

BIRD stands for “BIg Bench for LaRge-scale Database Grounded Text-to-SQL Evaluation.” It is described as a cross-domain dataset that examines the impact of extensive database contents on text-to-SQL parsing.

The dataset contains 12,751 unique question-SQL pairs drawn from 95 databases that total 33.4 GB of data. Those databases span more than 37 professional domains, including blockchain, hockey, healthcare, and education. The size and breadth are meant to push models beyond the small, tidy schemas used in earlier work.

The benchmark was accepted at NeurIPS 2023 as a Spotlight paper, and the dataset is licensed under CC BY-SA 4.0.

Where the Difficulty Comes From

The project highlights three sources of difficulty that separate BIRD-SQL from cleaner benchmarks.

The first is large and dirty values. Real-world database values typically keep their original and frequently messy format, so a parser has to analyze non-standard formats before it can reason about them.

The second is external knowledge. Some queries cannot be answered from the schema alone. The project gives the example that account.type = 'OWNER' can be inferred from outside evidence, such as the rule that a loan requires the account type to be the owner. A model has to connect that domain knowledge to the data to produce the right query.

The third is efficiency. BIRD is described as the first text-to-SQL benchmark designed to encourage parsers to produce SQL queries that are not only correct but also efficient. To capture this, it uses a Reward-based Valid Efficiency Score, abbreviated R-VES.

How Models Compare to People

The headline result is the gap between machines and people. Human performance on the benchmark, measured with data engineers and database students, reaches 92.96% execution accuracy. The human R-VES, the efficiency metric, sits at 83.26.

Leading systems on the overall leaderboard remain below that human mark. The reported top entries by test-set execution accuracy include AskData with GPT-4o at 81.95% (September 2025), Agentar-Scale-SQL at 81.67% (December 2025), and Xiaomi Text2SQL at 80.83% (May 2026). Even the strongest of these trails human accuracy by roughly 11 percentage points.

What the Results Suggest

The numbers point to steady progress paired with a persistent gap. Models can now handle a large share of realistic text-to-SQL tasks, but they still fall short of skilled people on databases full of irregular values and unstated domain rules.

For teams considering text-to-SQL features, the benchmark is a reminder that accuracy on clean demos does not transfer cleanly to production data. External knowledge requirements and messy values are where systems tend to slip, and the inclusion of an efficiency metric reflects that a query can be correct yet still poorly written. Validation and human review remain sensible safeguards while the gap to human-level accuracy stays open.