Loading...
Loading...
Distributed, RESTful search and analytics engine for addressing complex queries.
Understand how a distributed search engine stores, scales, and queries massive amounts of text data in milliseconds without doing full table scans.
| Term (Dictionary) | Postings List (Doc IDs) |
|---|---|
| brown | Doc0 |
| fox | Doc0 |
| quick | Doc0 |
Instead of scanning documents, Elasticsearch looks up the exact term in a sorted dictionary to instantly find matching documents.
Why doesn't Elasticsearch just use a standard SQL database for text search?
You already know from the indexing module how a sorted B-tree answers prefix matches yet cannot help a wildcard on both sides, and from the sharding module how slices of a dataset can be searched in parallel. Imagine an e-commerce store with 500 million products. A shopper types "wireless gaming mouse". The naive query scans every title for that substring (a LIKE with wildcards on both sides, which cannot use a normal sorted index). At 0.1ms per row, 500M rows cost 50,000 seconds of single-thread work, saved only by parallelism and still seconds slow. Elasticsearch (a search engine that pre-builds word-to-document lists) flips the work to write time and answers in milliseconds.
Think of the index at the back of a textbook, the one comparison we will use here: instead of rereading every page for each question, you look up each word once and intersect the page lists. In the lab above, run the substring scan versus the indexed search on the same catalog and watch seconds collapse to milliseconds, because one path reads every document and the other reads two short lists.
If tables are familiar, here is the mapping. A cluster is the whole search system. An index is one collection of JSON documents (one document is a single JSON object like one product). A field is one key inside it. An inverted index is the word-to-ids dictionary that replaces substring scanning.
| Table idea | Search name | What it holds |
|---|---|---|
| Database | Cluster | The cooperating machines running search |
| Table | Index | One collection of related JSON documents |
| Row | Document | One JSON object, for example one product |
| Column | Field | One key-value pair inside the document |
| LIKE '%query%' | Inverted Index | Sorted word dictionary pointing at document ids |
At write time the engine splits each title into tokens (lowercased word pieces with plurals folded, so MICE becomes mouse) and appends the document id to each token's list. At read time it tokenizes the query the same way, fetches one sorted list per token, and intersects them. Work the toy: wireless lives in docs 1 and 3, mouse in docs 1 and 2, so wireless plus mouse leaves only doc 1.
"wireless mouse" never opens the three documents. It fetches two short lists and intersects: [1, 3] and [1, 2] = [1]. In the lab, add a fourth document sharing both words and watch the intersection grow by exactly one.The query pipeline normalizes language before it touches the dictionary, because wireless MICE and wireless mouse must meet the same lists. An analyzer (the tokenizer plus lowercasing plus stemming pipeline) produces the tokens, the dictionary lookup fetches the lists, and BM25 (the ranking formula that rewards rare-word matches and penalizes long documents) orders the survivors. The edge case is over-stemming: folding mouse and mice together helps shoppers but folding gaming and game together can blur intent, which is why analyzers stay per-field and tunable.
Shopper types "Wireless MICE". Nothing is searched yet.
Analyzer emits ["wireless", "mouse"] after lowercasing and folding the plural.
Dictionary seeks, each near constant time, return the two id lists, then one linear merge intersects them.
BM25 scores survivors: rare wireless-mouse matches outrank documents that merely mention mouse fifty times.
No single server holds 500 million documents.
An index splits into shards (independent slices of the document set) spread across nodes (member servers). A query fans out to all shards at once, each intersects locally, and the coordinator merges the top hits. Four shards evaluate in roughly a quarter of the single-shard time, and replica shards (extra copies of each slice on other machines) let a dead node pass unnoticed. The trade is freshness plus merge cost: newly written documents appear after a short refresh delay, and very wide fan-outs pay more in merging than they save in scanning. In the lab, grow shards from 1 to 4 and watch latency fall, then kill a node and watch the same query survive on replicas.
Each shard searches its slice simultaneously, so added documents cost added machines, not added seconds.
Every slice keeps copies elsewhere. One node dies mid-query and the coordinator simply reads the copy.
Split the dictionary, search the pieces in parallel, survive dead nodes. But word lists only answer what contains these words. When your questions become what happened, in order, across billions of events, which store shape would you reach for instead of an inverted index?
Try this in the playground
Open a template and build it yourself — then take a quiz.