Skip to the content ForoHackers
Log inJoin

Information retrieval: start here

Information retrieval
#407 · Ophuscado · · Report

Information retrieval is the science of finding the documents, passages or items that satisfy a person's need from a large collection, usually text. Unlike a database query with an exact answer, a search result is a ranked guess, so the field is about modelling relevance and measuring how good the ranking is.

Key concepts

  • Inverted index: for each term, the list of documents containing it; it makes search over millions of documents fast.
  • Text analysis: splitting text into tokens, lowercasing, removing noise and reducing words to a base form.
  • Ranking models: term frequency, inverse document frequency and length normalisation, as in TF-IDF and BM25.
  • Vector search: documents and queries as embeddings, compared by similarity rather than by shared words.
  • Evaluation: judge a system against queries with known relevant results, not by impression.

Vocabulary

  • Precision and recall: the share of results that are relevant, and the share of relevant items that were found.
  • Query expansion: adding related terms to a query to reach more relevant documents.
  • Stemming: cutting words to a common root, so "running" and "runs" match.
  • Relevance feedback: using the user's reactions to improve the results.

What to study, in order

  1. Build an inverted index over a folder of text files, and answer a one-word query.
  2. Add tokenisation and stemming, and support queries of several words.
  3. Implement TF-IDF ranking, then BM25, and compare the orders they give.
  4. Learn precision, recall and mean average precision, and score your system on a small set of judged queries.
  5. Add embeddings and compare semantic search with keyword search on queries where each fails.
  6. Read about query understanding and learning to rank.

Related tags

#search-engines #nlp #data-structures #databases

Log in or join to reply.