Information retrieval is the science of finding the documents, passages or items that satisfy a person's need from a large collection, usually text. Unlike a database query with an exact answer, a search result is a ranked guess, so the field is about modelling relevance and measuring how good the ranking is.
Key concepts
- Inverted index: for each term, the list of documents containing it; it makes search over millions of documents fast.
- Text analysis: splitting text into tokens, lowercasing, removing noise and reducing words to a base form.
- Ranking models: term frequency, inverse document frequency and length normalisation, as in TF-IDF and BM25.
- Vector search: documents and queries as embeddings, compared by similarity rather than by shared words.
- Evaluation: judge a system against queries with known relevant results, not by impression.
Vocabulary
- Precision and recall: the share of results that are relevant, and the share of relevant items that were found.
- Query expansion: adding related terms to a query to reach more relevant documents.
- Stemming: cutting words to a common root, so "running" and "runs" match.
- Relevance feedback: using the user's reactions to improve the results.
What to study, in order
- Build an inverted index over a folder of text files, and answer a one-word query.
- Add tokenisation and stemming, and support queries of several words.
- Implement TF-IDF ranking, then BM25, and compare the orders they give.
- Learn precision, recall and mean average precision, and score your system on a small set of judged queries.
- Add embeddings and compare semantic search with keyword search on queries where each fails.
- Read about query understanding and learning to rank.