Skip to the content ForoHackers
Log inJoin

Why BM25 still beats a lot of clever things: term frequency saturation

Data scienceInformation retrievalSearch engines
#1060 · two_phase_commitment · · Report

If you query a text index and the ranking looks sensible, there is a good chance it is BM25 underneath. The formula looks scary but it encodes three simple ideas.

score(D, Q) = sum over terms q in Q of
    IDF(q) * f(q, D) * (k1 + 1)
    ---------------------------------------------
    f(q, D) + k1 * (1 - b + b * |D| / avgdl)
  1. IDF: rare terms matter more. A match on "ocsp" tells you more than a match on "the".
  2. Saturation: the term frequency f grows the score, but with diminishing returns. As f goes to infinity, the fraction goes to k1 + 1. Mentioning a word 50 times is not 50 times as relevant as mentioning it once.
  3. Length normalisation: a long document naturally contains more terms, so b (usually 0.75) shrinks the score of documents longer than average.

Typical defaults are k1 between 1.2 and 2.0 and b = 0.75. Both are worth tuning on your own queries, with some relevance judgements to measure against, not by feel.

It is a baseline that is cheap, explainable, and hard to beat by much without labelled data. Treat it like a database index: boring and enormously useful.

#1606 · regex_regret · · Report

And the one-line summary of why grep -c is not a search engine: no IDF, no saturation, no length normalisation, and no ranking at all. It tells you whether, not how well.

#1607 · tabs_over_spaces_42 · · Report

Stupid question: if b = 0 the length term vanishes. When would you ever want that?

#1608 · two_phase_commitment · · Report

When documents are all about the same length anyway (titles, tags, short product records), or when length itself is a signal of quality. For long free text you almost always want it above zero. The honest way to find out is to run both against your own queries and look at the difference.

Log in or join to reply.