A search engine is the full system that turns a question into a ranked list of results: it finds the content, indexes it, understands the query and returns answers in a fraction of a second. It may search the whole web, a company's documents or one website, and each case shares the same architecture.
Key concepts
- Crawling: following links or reading a source to discover and fetch content, politely and without overloading sites.
- Indexing: parsing content, analysing text and writing it into an inverted index, split into shards.
- Query processing: parse the query, fetch candidate documents from the index and score them.
- Ranking signals: text match, freshness, link structure such as PageRank, and quality signals.
- Serving at scale: caches, replicas and sharding keep results fast under heavy load.
Vocabulary
- Crawler or spider: the program that fetches pages to be indexed.
- robots.txt: the standard file where a site states what crawlers may fetch.
- Shard: a slice of the index stored on one machine.
- Autocomplete: suggestions offered while the person is still typing.
What to study, in order
- Information retrieval basics: inverted index, TF-IDF and BM25.
- Write a small crawler that respects robots.txt and a delay, and runs only on your own site or a test site.
- Index what it collected and add a search page with ranked results and snippets.
- Learn PageRank and compute it on a small graph by hand and in code.
- Add spelling correction, autocomplete and filters, and observe how each affects results.
- Use an open source search library, shard an index across two processes and measure latency.
Related tags
#information-retrieval #web-standards #distributed-systems #nlp