Skip to the content ForoHackers
Log inJoin

Search engines: start here

Search engines
#409 · Ophuscado · · Report

A search engine is the full system that turns a question into a ranked list of results: it finds the content, indexes it, understands the query and returns answers in a fraction of a second. It may search the whole web, a company's documents or one website, and each case shares the same architecture.

Key concepts

  • Crawling: following links or reading a source to discover and fetch content, politely and without overloading sites.
  • Indexing: parsing content, analysing text and writing it into an inverted index, split into shards.
  • Query processing: parse the query, fetch candidate documents from the index and score them.
  • Ranking signals: text match, freshness, link structure such as PageRank, and quality signals.
  • Serving at scale: caches, replicas and sharding keep results fast under heavy load.

Vocabulary

  • Crawler or spider: the program that fetches pages to be indexed.
  • robots.txt: the standard file where a site states what crawlers may fetch.
  • Shard: a slice of the index stored on one machine.
  • Autocomplete: suggestions offered while the person is still typing.

What to study, in order

  1. Information retrieval basics: inverted index, TF-IDF and BM25.
  2. Write a small crawler that respects robots.txt and a delay, and runs only on your own site or a test site.
  3. Index what it collected and add a search page with ranked results and snippets.
  4. Learn PageRank and compute it on a small graph by hand and in code.
  5. Add spelling correction, autocomplete and filters, and observe how each affects results.
  6. Use an open source search library, shard an index across two processes and measure latency.

Related tags

#information-retrieval #web-standards #distributed-systems #nlp

Log in or join to reply.