Skip to the content ForoHackers
Log inJoin

Information retrieval: start here

Information retrieval
#2913 · · Quote · Report

Information retrieval is the science of finding the documents, passages or items that satisfy a person's need from a large collection, usually text. Unlike a database query with an exact answer, a search result is a ranked guess, so the field is about modelling relevance and measuring how good the ranking is.

Key concepts

  • Inverted index: for each term, the list of documents containing it; it makes search over millions of documents fast.
  • Text analysis: splitting text into tokens, lowercasing, removing noise and reducing words to a base form.
  • Ranking models: term frequency, inverse document frequency and length normalisation, as in TF-IDF and BM25.
  • Vector search: documents and queries as embeddings, compared by similarity rather than by shared words.
  • Evaluation: judge a system against queries with known relevant results, not by impression.

Vocabulary

  • Precision and recall: the share of results that are relevant, and the share of relevant items that were found.
  • Query expansion: adding related terms to a query to reach more relevant documents.
  • Stemming: cutting words to a common root, so "running" and "runs" match.
  • Relevance feedback: using the user's reactions to improve the results.

What to study, in order

  1. Build an inverted index over a folder of text files, and answer a one-word query.
  2. Add tokenisation and stemming, and support queries of several words.
  3. Implement TF-IDF ranking, then BM25, and compare the orders they give.
  4. Learn precision, recall and mean average precision, and score your system on a small set of judged queries.
  5. Add embeddings and compare semantic search with keyword search on queries where each fails.
  6. Read about query understanding and learning to rank.

Related tags

#search-engines #nlp #data-structures #databases

You are not logged in, so this goes up anonymously: no account, no name and no address is stored with it, and it may carry no link and no image. Log in or join to post under your name. Our web server's logs keep the address and time of every request for up to 14 days; to keep yours out of them, use Tor. To reach the staff in confidence, use the drop. Sign as a returning source (a key kept only in this browser)

Sending takes a few seconds here: your browser first works out a proof of work against spam.

Reply