Why BM25 Still Beats Embeddings Out of Domain

Neural embeddings are often treated as a strict upgrade over older keyword search. A widely cited benchmark suggests the picture is more complicated. The BEIR study (Thakur et al., 2021), published at the NeurIPS 2021 Datasets and Benchmarks track, found that the classic lexical algorithm BM25 remains a robust baseline that many modern dense retrieval models fail to beat when they are tested on data they were not trained on. The paper is available at https://arxiv.org/abs/2104.08663.

What the BEIR Benchmark Tested

BEIR, short for Benchmarking-IR, was built to measure how well retrieval systems generalize rather than how well they perform on a single familiar dataset. The authors assembled 18 publicly available datasets covering a range of retrieval tasks and domains, then evaluated 10 retrieval systems spanning lexical, sparse, dense, late-interaction, and re-ranking approaches.

The emphasis on out-of-distribution evaluation matters. Many retrieval models report strong numbers on the dataset they were trained and tuned for. BEIR instead asked a harder question: how do these systems behave in a zero-shot setting, where the model meets a new domain with no additional training?

BM25 as a Robust Baseline

The central finding is that BM25, a keyword-matching algorithm based on term frequency and document statistics, holds up surprisingly well against more sophisticated neural approaches in zero-shot settings. The paper describes BM25 as a robust baseline. Dense and sparse retrieval models, despite being more computationally efficient, often underperformed other approaches in the benchmark. The authors read this as evidence of considerable room for improvement in those models’ ability to generalize beyond their training distribution.

This is the counterintuitive result worth internalizing. A method that requires no model training and no embedding computation stayed competitive with neural systems that demand substantial infrastructure, simply because it does not depend on having seen similar data before.

Where Neural Methods Pulled Ahead

The benchmark does not conclude that keyword search wins everywhere. According to the paper, re-ranking and late-interaction models delivered the best average zero-shot performance across the datasets. The catch is cost: the authors note that this stronger performance comes at high computational expense compared with simpler retrieval.

So the trade-off runs along two axes. Lexical BM25 is cheap and generalizes reliably but does not capture meaning beyond shared terms. The strongest neural results come from re-ranking and late-interaction methods that are accurate but computationally heavy. The plain dense embedding models that sit between these extremes were the ones that most often struggled to generalize.

Reading the Result in Practice

For teams choosing a retrieval approach for an unfamiliar corpus, BEIR is a reminder to measure against a strong baseline before assuming embeddings are required. BM25 is inexpensive to run and, in this benchmark, difficult for off-the-shelf dense models to beat without domain-specific training.

The broader lesson reaches past search. A newer technique does not automatically obsolete an established algorithm, especially when the newer method depends heavily on training data that matches the task at hand. BEIR’s out-of-domain framing exposes exactly that dependency, and it shows why a decades-old keyword method continues to earn its place as a baseline that any new system has to clear.