The Dictionary Guide to Spark: Read, Sort, Define

Imagine being handed an impossible job: write down every word in the English language. Not just what each word means today, but when it was first used, how its meaning changed over the centuries, and a real sentence, from a real book, to prove every claim.

In 1879, a Scottish schoolmaster named James Murray took on exactly that job. It became the Oxford English Dictionary (OED), and it was expected to take about ten years.

There was one enormous problem. To find the earliest use of every word, someone had to read practically everything ever printed in English. If Murray had sat in a library and read from A to Z, he would have died before he finished B.

So he didn’t read. He asked the public to.

His appeal went out across the English-speaking world. Read the books you already own, it said, and whenever a word strikes you as rare, old-fashioned, new, or used in an odd way, write it down. Each find went on a slip of paper about six inches by four: the word in the top left corner, then the date, the title, and the full sentence it appeared in. Then the reader posted the slip. Not the book.

Thousands of people were suddenly reading at once, each through their own small shelf. The reading happened where the books already were.

The slips came back by the sack, around a thousand a day. Murray sorted them in what he called the Scriptorium, a corrugated-iron shed he first put up at Mill Hill School, where he taught, and then rebuilt in his back garden when he moved to Oxford. Its walls held 1,029 pigeonholes. His assistants filed each slip under its word, threw out duplicates, laid each word’s slips out by date, chose the best, and wrote the entry.

It didn’t take ten years. The last volume came out in 1928, thirteen years after Murray died, with 414,825 words and 1,827,306 quotations. But it got finished, because thousands of people read in parallel and a small team sorted what they sent.

Computers learned the same trick for the same job. In 2004, Google published a paper describing MapReduce, the system it had used to rebuild the index behind its web search: split the reading across thousands of machines, sort what they find by key, and assemble the results. An open-source version, Hadoop, grew out of Nutch, a web search engine. MapReduce had one expensive habit, though: it wrote everything to disk between steps. In 2009, Matei Zaharia started Spark at UC Berkeley’s AMPLab to keep that work in memory instead. It became an Apache project in 2014, and today it’s the default engine for large batch jobs, from building search indexes to training data for machine learning.

So this post isn’t stretching an analogy. It’s the same job, done twice, a century apart: read a mountain of text, and build an index anyone can search.

The first half is the Scriptorium: what each part of Spark is for, and whether you need it at all. The second half is a Thursday morning, a Bible verse the search engine swears doesn’t exist, and a small working engine called scriptorium that finds it.

Reading the appeal

The OED needed the public because no one person could read that much. Before you reach for a cluster, check that you have the same problem. Three questions decide it.

Does it fit on one machine?

The corpus in this post is a sample of eighteen Project Gutenberg books, from Emma to Moby-Dick, plus the King James Bible and three Shakespeare plays. It’s about 12 MB of text: 49,037 passages and 2.2 million words. On a laptop, Spark’s local mode builds the whole index in about 45 seconds, and most of that is Spark starting up and handing text to Python.

The whole of Project Gutenberg is more than 75,000 books. That’s tens of gigabytes of plain text, which still fits comfortably on one machine. A single server running Lucene will index it without complaint, and a single-node engine like DuckDB or Polars will crunch it without a cluster at all.

You need a cluster when the reading list is more like the web. Each Common Crawl snapshot holds billions of pages. No single machine reads that in a useful amount of time.

So, in the spirit of the maple syrup post, which ran a lakehouse on DuckDB: this post runs Gutenberg on Spark to learn the shape of the thing, not because Gutenberg needs it.

Are you building the index, or answering questions from it?

These are different jobs. Spark is the reading programme: a thorough batch build that runs overnight. OpenSearch is the published dictionary: it answers questions in milliseconds and takes new documents all day long.

In most real systems they meet in the middle. A Spark job cleans, enriches or builds the documents and loads them into OpenSearch, which then keeps its own index up to date. The search half of scriptorium is a small stand-in for OpenSearch, so you can see what it does with the index Spark built.

What happens when a machine dies halfway through?

On one machine, you start again. The reading programme didn’t have that option, and it lost things. Slips for the letter H turned up in Italy. The slips for words beginning with Pa went missing for twelve years and were eventually found in County Cavan, in Ireland, where someone had been using them as kindling.

Spark assumes machines will die, and plans for it. It remembers how every piece of its work was produced, so a lost piece can be recomputed from its inputs. It retries tasks that fail, and it can rerun slow ones somewhere else.

	Murray at his desk	The reading programme
Size	Fits on one machine	Too big for one machine, or too slow on it
When something fails	Start the run again	Lose one reader’s pile, hand it to someone else
What it costs to run	One process	A driver, executors, a cluster manager, object storage, and someone on call
Good fit for	Most corpora, including all of Gutenberg	The web, logs, anything that grows faster than one machine

One honest note before we go on. Spark isn’t the newest idea in data processing, and for anything that fits on one machine, single-node engines are now often faster and always simpler. But when the data really doesn’t fit, Spark is still what most teams reach for, on Databricks, Amazon EMR, Google Dataproc or their own Kubernetes clusters. And its core ideas, partitions, shuffles and retries, are the ones every distributed engine has to answer somehow.

From parlour to pigeonhole

The reading programme and a Spark job have the same stages, in the same order.

Every reader gets a shelf. Spark splits its input into partitions and gives each one to a task, running on a worker process called an executor. In the Hadoop era, the books really did stay home: data lived on the same machines that processed it. Today the books usually live in object storage, like Amazon S3, and executors fetch them over the network. What still happens at the reader’s desk is the first round of work. Everything that can be done inside one partition is done there, before anything is sent anywhere.

Murray’s address books. In 2014, the lexicographer Sarah Ogilvie found three of Murray’s handwritten address books in the Oxford University Press archive. They recorded who his readers were and where they lived. Spark has the same thing in the driver, the process that plans the job, hands out tasks, and keeps track of which task produced what and where it went. Lose a reader and the driver sends their shelf to someone else. Lose the driver and the whole job is lost.

Every slip in the same format. A reader wrote the word, the date, the title and the sentence. In our search engine, the first stage turns each passage into rows: a word, the passage it came from, and how many times it appears there. The rule for what counts as a word, including whether “Café” and “cafe” are the same one, is called the analyzer. Keep that in mind, because it comes back.

The pigeonholes. To gather every row for the same word in one place, Spark has to move rows between machines. This is the shuffle. Each row is hashed on its key and sent to the partition that owns that hash. Murray split the work by letter instead. We’ll see why that’s a poor choice for machines.

Weeding duplicates at the desk. In the Scriptorium, duplicate slips were thrown out after they reached Oxford, which means someone had already paid the postage. Spark weeds at the reader’s desk. Before a shuffle, each task combines what it can on its own, so a count of “the” leaves each task as one row, not thousands. Hadoop made you write this step yourself, as a combiner. Spark does it automatically for aggregations.

The editors. After the shuffle, each task receives every row for its share of the keys and writes the entries. Ours turn each word’s rows into a postings list: every passage the word appears in. A collection of postings lists is an inverted index, and it’s the core of every search engine.

The reader who never writes back. Some of Murray’s readers took a book and were never heard from again. When a Spark task fails, it’s retried, up to four times by default. When one is merely slow, Spark can start a second copy on another executor and keep whichever finishes first. This is speculative execution, and it’s off by default. Deciding which machines the executors run on is the job of a cluster manager, usually Kubernetes or YARN, much like someone had to decide which reader got which book.

The lost slips. Then there’s bondmaid. Its slips fell down behind some books in the Scriptorium, and nobody noticed. The first volume, covering A and B, was published in 1888 without it. Murray later wrote that at least thirty people had seen the pages at various stages, and not one of them spotted the gap. He called it absolutely inexplicable. The word finally appeared in the 1933 supplement.

Modern storage has mostly solved the Pa problem. Object stores keep several copies of everything, so a dead disk is rarely your problem any more. What they haven’t solved is the bondmaid problem. The slips weren’t destroyed. They were somewhere nobody looked, and the volume shipped without an error. Counting what you publish is how you notice.

Where the analogy breaks

Murray’s readers chose what to write down. They were asked for words that struck them, so each reader was quietly filtering. A search index can’t do that. Every word in every passage has to go in, or phrase searches and rare words stop working.

The dictionary also never ranked anything. An editor chose the best quotation by judgement. A search engine has to put ten thousand matching passages in order in a few milliseconds, which takes arithmetic, not judgement. We’ll get to it.

And the OED divided its pigeonholes by word. OpenSearch divides its shards by document. That difference turns out to matter more than anything else in this post.

Thursday, 7:20 a.m.

Parlour is a made-up reading app. It launched with a shelf of eighteen public-domain classics from Project Gutenberg, with the full catalogue to follow. Every night, two Spark jobs run on a small cluster. One rebuilds the reading list from the books, and the other rebuilds the search index from the reading list.

On Thursday, September 24, the first support email arrives before anyone has had coffee. A theology student is writing an essay on Leviticus. She searched for bondmaid and got one passage, from Galatians. The verse she actually needs, Leviticus 19:20, isn’t in the results. She’s sure it was there yesterday.

At 8:05 an editor preparing a reading guide for Julius Caesar reports that searching for “Cæsar,” the First Folio spelling, finds nothing at all. At 8:40 someone on the team notices something odder. “Hast seen the White Whale?” appears three times in Moby-Dick, word for word, and search ranks the copies differently.

The pipeline was built quickly and has been fine for months. Here are the parts that matter:

# the nightly schedule
#   01:00  spark-submit parlour_reading_list.py   (usually done by 01:40)
#   02:00  spark-submit parlour_index.py          (reads whatever is in the folder)

# parlour_reading_list.py: every book read as text
books = spark.read.text("s3://parlour/books/", wholetext=True)

# the search service, which splits queries its own way
words = query.lower().split(" ")

Each line looks harmless. Together they produced four problems:

1. Reading a folder before its writer had finished. On Wednesday night a slow mirror held up the reading list job, and at 02:00 it was still writing. On object storage, a Spark job’s output doesn’t appear all at once: its files become visible one at a time as the job commits. The index job listed the folder, found six of the seven files, and indexed what it found. It succeeded. The index was missing 7,093 passages, 14% of the corpus, and Leviticus 19:20 was one of them.
2. Bytes read in the wrong encoding. Two of the eighteen files, Julius Caesar and Chesterton’s The Ball and the Cross, are stored in ISO-8859-1, an older encoding. Spark’s text reader assumes UTF-8, and bytes that aren’t valid UTF-8 quietly become the replacement character, �. So “Cæsar” became “C�sar,” which the analyzer split into two words: c and sar. Four passages were garbled this way, and nothing complained.
3. Two rules for the same word. The index job folded accents, so “café” was stored as cafe. The search service only lower-cased, so a reader typing “café” looked up café, which isn’t in the index. Twelve passages mention a cafe; typed with the accent, none of them appear.
4. Each shard with its own scale. The index is split into four shards, and each one ranked results using statistics about its own quarter of the corpus. One copy of “Hast seen the White Whale?” sits on shard 2 and two on shard 3, so identical text got different scores.

None of these are exotic, and each has a counterpart in the Scriptorium: slips that were somewhere nobody looked, a slip nobody could read, one word filed under two spellings, and editors who judged how rare a word was by looking only at their own pile.

So let’s build it properly. I’ve called it scriptorium.

A Scriptorium you can carry

scriptorium is PySpark, the Python API that most Spark code is written in today. The Spark parts use DataFrames, Spark’s table-like API, and the search side is plain Python with PyArrow reading Parquet.

The same scripts run in two places. In Spark’s local mode, the driver and the executors all run inside one process on a laptop, and every number in this post comes from running it that way. On a cluster, you hand the same file to spark-submit and point it at object storage instead of a local folder. Nothing in the code changes.

The books are the eighteen-book Gutenberg sample that NLTK distributes. If you want the full catalogue, use one of Project Gutenberg’s mirrors. The main site asks not to be crawled.

The search side is deliberately small. It loads every shard into memory, and it’s there to show what OpenSearch does with an index, not to replace it. Here’s how each piece maps to the real thing:

In scriptorium	At scale
Local mode on a laptop	spark-submit to Kubernetes, EMR, Dataproc or Databricks, reading from object storage
Index shards as Parquet	Lucene segments inside OpenSearch shards, written through the bulk API, for example with OpenSearch’s Spark connector
Sharding by hash(passage_id)	OpenSearch routing, which also takes a Murmur3 hash of the document ID, modulo the number of primary shards
The corpus-wide stats table	The dfs_query_then_fetch search type, which collects term statistics from every shard before scoring
The CURRENT file	Build a new index, then move an index alias to it in one atomic request
search.py’s thread pool	The coordinating node, which sends a query to every shard and merges the answers
Canaries	Search regression tests and synthetic monitoring

Read, sort, define: the build, layer by layer

The code lives in my blog-tools repository, next to sumbisori and sugarshack from the last two posts. It needs Python and Java 17 or later, and the README takes you from an empty machine to every result below.

git clone https://github.com/Kunal-Vyas/blog-tools.git
cd blog-tools/scriptorium
pip install -e .

Here’s the whole thing, and what each step hands to the next:

1. The reading list

The first job turns each book into passages: a paragraph, split further if it runs past 1,200 characters. Like an OED slip, a passage is short enough to quote and carries its source with it. Leviticus 19:20 becomes passage bible-kjv:2636.

This is also where the encoding gets fixed, once, before anything else touches the text. Instead of spark.read.text, which assumes UTF-8, the job reads each book as raw bytes and decodes them itself:

def decode(content: bytes) -> tuple[str, str]:
    """Strict UTF-8 first; ISO-8859-1 only for files UTF-8 rejects.

    Every byte sequence is valid ISO-8859-1, so it never fails. That's exactly why it
    has to be the fallback: tried first, it would garble real UTF-8 without a word.
    """
    try:
        return content.decode("utf-8", errors="strict"), "utf-8"
    except UnicodeDecodeError:
        return content.decode("iso-8859-1"), "iso-8859-1"


@F.udf(BOOK)
def read_book(content, book):
    text, encoding = decode(bytes(content))
    return passages.title(text, book), encoding, passages.split(text)

The order matters. Every sequence of bytes is valid ISO-8859-1, so decoding as ISO-8859-1 never fails, and a file that’s really UTF-8 would come out garbled without a single error. So the strict UTF-8 decoder goes first. Only a file it rejects gets the fallback.

The rest of the job writes the passages as seven Parquet files, and then, last of all, a small manifest recording how many it wrote:

raw = spark.read.format("binaryFile").option("pathGlobFilter", "*.txt").load(src)
book = F.regexp_extract("path", r"([^/]+)\.txt$", 1)
books = raw.select(book.alias("book"), read_book("content", book).alias("b"))

# ...

rows = (books.select("book", F.posexplode("b.passages").alias("i", "text"))
        .select(F.concat_ws(":", "book", (F.col("i") + 1).cast("string")).alias("passage_id"),
                "book", (F.col("i") + 1).alias("n"), "text"))
(rows.repartitionByRange(FILES, "book", "n")           # seven files, in reading order
     .write.mode("overwrite").parquet(str(out / "passages")))
books.select("book", "b.title", "b.encoding").write.mode("overwrite").parquet(str(out / "books"))

# The manifest is written last, and counts what was written: the build checks against it.
written = spark.read.parquet(str(out / "passages")).count()
(out / "_manifest.json").write_text(json.dumps({"passages": written}) + "\n")

18 books {'iso-8859-1': 2, 'utf-8': 16}, 49,037 passages

That manifest is the reading list’s equivalent of a signature at the bottom of the page. Anything that reads the folder can check its count against it. The build refuses to start without it, and the publisher checks the count in step 7.

You can reproduce Thursday’s partial read without a slow mirror. lose_a_file copies the reading list without the Parquet file that holds a given passage, which is exactly what the index job saw at 02:00:

python -m scriptorium.demo.lose_a_file lake lost-lake bible-kjv:2636

removed part-00001-….snappy.parquet: 7,093 passages

2. One rule for what counts as a word

Problem 3 happened because the index and the search service each had their own idea of a word. The fix is a single function that both of them use:

import re
import unicodedata

_WORD = re.compile(r"[^\W_]+")          # runs of letters and digits, in any script
_LIGATURES = str.maketrans({"æ": "ae", "Æ": "Ae", "œ": "oe", "Œ": "Oe", "ß": "ss"})


def tokens(text: str) -> list[str]:
    """Lower-case, fold accents and ligatures, split on anything that isn't a letter or digit."""
    folded = unicodedata.normalize("NFKD", text)                         # é becomes e + an accent mark
    folded = "".join(c for c in folded if not unicodedata.combining(c))  # drop the accent marks
    folded = folded.translate(_LIGATURES)                                # æ doesn't decompose: fold by hand
    return _WORD.findall(folded.lower())

A few details are worth pointing out:

• Decomposing, then dropping accents. Unicode normalization (NFKD) splits “é” into “e” plus a separate accent mark, and the next line throws the marks away.
• Ligatures by hand. “æ” is a letter in its own right, not an “a” with a mark on it, so normalization leaves it alone. Without the explicit replacement, “Cæsar” and “Caesar” would stay different words.
• No stemming. This analyzer doesn’t reduce words to their roots, so bondmaid won’t find bondmaids. OpenSearch’s english analyzer would. It’s a choice, and the important thing is that indexing and search make the same one.
• It runs in Python. Spark itself runs on the JVM, so every passage has to cross into a Python worker and back to be analyzed. At this size that doesn’t matter. At scale, you’d reach for Spark’s built-in functions, or an Arrow-based pandas UDF that moves data in batches.

3. The readers

The build turns every passage into one row per word, then counts each word within its passage:

passages = spark.read.parquet(str(reading_list / "passages"))

# The readers: one slip per word per passage, carrying its count. Spark counts
# inside each task before the shuffle, so "the" nine times becomes one row, not nine.
slips = (passages
         .select("passage_id", F.explode(tokens("text")).alias("word"))
         .groupBy("passage_id", "word").agg(F.count("*").alias("tf"))
         .cache())

That groupBy is where counting at the desk happens. Every passage lives in one partition, so each task counts its own passages before the shuffle. Spark’s own metrics for the run show the difference: 2,199,207 words went in, and 1,612,161 rows crossed the shuffle, one per distinct word in each passage. A passage that uses “the” nine times sends one row, not nine.

.cache() keeps those rows in memory, because the next two steps both start from them. This is the habit Spark was built for: MapReduce would have written them to disk and read them back twice.

4. The pigeonholes

Every row needs a shard. Here are two ways to choose one:

def by_passage(shards: int):
    """OpenSearch's way: a passage and all its words live on one shard."""
    return F.pmod(F.hash("passage_id"), F.lit(shards))


def by_letter(shards: int):
    """The OED's way: words are split among editors by their first letter."""
    first = F.ascii(F.col("word")) - F.ascii(F.lit("a"))           # a=0 ... z=25
    return (F.when((first >= 0) & (first < 26), F.floor(first * shards / 26))
             .otherwise(0).cast("int"))                           # a-g, h-m, n-t, u-z for 4

Murray’s way looks tidy, and it goes wrong for two reasons. English words don’t start evenly across the alphabet: I ran the same build both ways, and the busiest letter range got 3.2 times as many rows as the quietest. A stage is only as fast as its slowest task, so the whole build waits for the a–g editor. And some single words are enormous. “the” appears in 36,563 of the 49,037 passages, and dividing by word sends that entire pile to a single shard, which will also be the only shard that can answer any query containing it.

Dividing by passage is what OpenSearch does. Each passage, and every word in it, goes to one shard, chosen by a hash of its ID. Spark’s hash function is Murmur3, the same family of hash OpenSearch uses to route documents. The work is even, and each shard is a complete small index of its own passages.

The price is that every query has to go to every shard, because any of them might hold a match. That’s a good trade: shards answer in parallel. But it has a second, quieter cost, which step 6 deals with.

5. The editors

Now each shard gathers every row for each of its words into one sorted postings list:

# The pigeonholes and the editors: pick a shard for every slip, then gather each
# word's slips on each shard into one sorted postings list.
shard = by_letter(shards) if letter else by_passage(shards)
index = (slips.withColumn("shard", shard)
         .groupBy("shard", "word")
         .agg(F.sort_array(F.collect_list(F.struct("passage_id", "tf"))).alias("postings"))
         .withColumn("df", F.size("postings")))
index.write.mode("overwrite").partitionBy("shard").parquet(str(out / "index"))

partitionBy("shard") writes each shard into its own folder, index/shard=0 to index/shard=3, so a search server can load exactly one. Each row holds a word, its postings, and df, the document frequency: how many passages on that shard contain the word. Here’s “whale” across the four shards:

Shard	Passages on the shard	Passages containing “whale”
0	12,427	194
1	12,196	175
2	12,055	196
3	12,339	207

Each shard knows how common “whale” is among its own passages, and nothing about the other three. That matters for ranking.

6. One scale for everyone

Ranking depends on how rare a word is. Rare words are strong evidence of relevance, and common words are weak evidence. That rarity comes from the document frequency, and in step 5 every shard measured it on its own quarter of the corpus. That’s problem 4.

The fix is to count, once, how many passages in the whole corpus contain each word. It starts from the same cached rows:

# One scale for everyone: how many passages in the whole corpus contain each word.
stats = slips.groupBy("word").agg(F.count("*").alias("df"))
stats.write.mode("overwrite").parquet(str(out / "stats"))

This is where weeding at the desk earns its keep. Each of the eight tasks counts its own rows for each word before anything crosses the network, so the shuffle carried 139,872 rows instead of 1,612,161. That’s 91% less. It’s more than the 41,837 distinct words in the corpus because each task still sends its own count for every word it saw. The tasks after the shuffle finish the sum.

With one scale, identical passages get identical scores, wherever they live:

Shard 2 has 150 passages containing “white,” out of 12,055. Shard 3 has 154 out of 12,339. So on shard 2 “white” looks a little rarer, and its passages score a little higher. With four shards and a corpus this size the differences are small, but they’re already enough to change which passage comes first. With many shards, or documents that aren’t spread evenly, they get bigger.

OpenSearch has exactly this problem. By default each shard scores with its own statistics, which is usually close enough on large indexes. When it isn’t, the dfs_query_then_fetch search type makes an extra round trip to gather statistics from every shard first. scriptorium computes them once, at build time, because the index doesn’t change between builds.

7. Publishing the volume

The first volume of the OED went to press without bondmaid because everyone checked the pages, and nobody checked the count. The publisher checks both.

Builds are written to their own folder, and readers find the live one through a single small file, CURRENT. The publisher audits a finished build, and only if every check passes does it point CURRENT at it:

# 2. Every passage the reading list's writer wrote was read. Reading a folder while
#    its writer is still at work succeeds, and quietly misses whatever isn't there yet.
expected = json.loads((lake / "reading-list" / "_manifest.json").read_text())["passages"]
corpus = json.loads((out / "corpus.json").read_text())
report(corpus["passages"] == expected, "passages",
       f"{expected:,} in the manifest, {corpus['passages']:,} in the index")

# 3. Every passage was decoded. Bytes that weren't valid UTF-8 become U+FFFD,
#    and the words around them become unfindable.
report(corpus["garbled"] == 0, "encoding",
       f"{corpus['garbled']:,} passages with undecodable bytes")

# 4. The shards and the stats agree: for every word, the shards' counts add up to
#    the corpus-wide count.
index = pq.read_table(out / "index", columns=["word", "df"])
by_word = index.group_by("word").aggregate([("df", "sum")])
stats = pq.read_table(out / "stats")
joined = by_word.join(stats, "word", join_type="full outer")
disagree = pc.sum(pc.fill_null(pc.not_equal(joined["df_sum"], joined["df"]), True)).as_py()
report(disagree == 0, "conservation",
       f"{by_word.num_rows:,} words in the shards, {stats.num_rows:,} in the stats, {disagree} disagree")

And the publish itself:

def publish(lake: Path, build: str) -> None:
    """Write the new pointer beside the old one, then swap them in one step.

    os.replace is atomic on a local disk. On object storage, write CURRENT as a single
    small object: a reader gets the old version or the new one, never a mix.
    """
    tmp = lake / "CURRENT.tmp"
    tmp.write_text(f"builds/{build}\n")
    os.replace(tmp, lake / "CURRENT")

On object storage, this is the part people get wrong. Renaming a folder there isn’t one step, it’s a copy and a delete for every file inside it. That’s the same reason Thursday’s index job could see half-written output. But replacing one small object is a single step: a reader sees the old version or the new one, never half of each. Table formats like Delta Lake and Apache Iceberg commit the same way, by writing a new metadata file and swapping one pointer to it.

There’s one more check, the canaries: a short list of words that must be findable in specific passages, including bondmaid in Leviticus 19:20 and Cæsar in Julius Caesar. Here’s a good build, then Thursday’s two broken ones, rebuilt on purpose:

=== 2026-09-24
  PASS   passages      49,037 in the manifest, 49,037 in the index
  PASS   encoding      0 passages with undecodable bytes
  PASS   conservation  41,837 words in the shards, 41,837 in the stats, 0 disagree
  PASS   canaries      4 of 4 found
Published builds/2026-09-24

=== 2026-09-25-lost          (one file missing, as on Thursday)
  BLOCK  passages      49,037 in the manifest, 41,944 in the index
  PASS   encoding      0 passages with undecodable bytes
  PASS   conservation  40,277 words in the shards, 40,277 in the stats, 0 disagree
  BLOCK  canaries      not found: bondmaid in bible-kjv:2636
Not published. CURRENT still points at the last good build.

=== 2026-09-25-raw           (every file read with spark.read.text, as on Thursday)
  PASS   passages      49,037 in the manifest, 49,037 in the index
  BLOCK  encoding      4 passages with undecodable bytes
  PASS   conservation  41,840 words in the shards, 41,840 in the stats, 0 disagree
  BLOCK  canaries      not found: caesar in shakespeare-caesar:25; cafe in chesterton-ball:849
Not published. CURRENT still points at the last good build.

Each check catches something different:

• Passages is the count nobody made in 1888. It’s the only check that catches a job reading less than its writer wrote.
• Encoding catches garbled text, whatever word it happens to break.
• Conservation compares the shards with the corpus-wide stats: for every word, the shards’ counts must add up to the total. It passes on both broken builds, which is correct. Both halves read the same bad input and agree about it. It exists to catch a bug in one half, not bad data going into both.
• Canaries are the closest thing to a real reader’s complaint. But they only catch what you thought to check. Notice that the encoding check caught four garbled passages, and the canaries only knew about two of them.

A blocked build is a good outcome. Readers keep yesterday’s index, and someone gets a message at breakfast instead of an email from a theology student.

8. Answering the post

The searcher reads CURRENT, loads the four shards, and scores passages with BM25, the same formula OpenSearch uses by default:

def search(self, query: list[str], scale: dict | None) -> list[tuple[float, str]]:
    """Score this shard's passages. scale=None means: judge rarity from this shard alone."""
    n = scale["passages"] if scale else len(self.length)
    avgdl = scale["tokens"] / scale["passages"] if scale else self.tokens / max(1, n)
    scores = {}
    for word in query:
        postings = self.postings.get(word, [])
        if not postings:
            continue
        df = scale["df"][word] if scale else len(postings)
        idf = math.log(1 + (n - df + 0.5) / (df + 0.5))           # Lucene's BM25 idf
        for pid, tf in postings:
            dl = self.length[pid]
            s = idf * tf * (K1 + 1) / (tf + K1 * (1 - B + B * dl / avgdl))
            scores[pid] = scores.get(pid, 0.0) + s
    # Each shard sends back only its own best: that's all the merge needs.
    return sorted(((s, pid) for pid, s in scores.items()), reverse=True)[:TOP]

BM25 multiplies two things for each query word. The first is how rare the word is (idf). The second is how often it appears in the passage, with diminishing returns: the second “whale” adds less than the first. K1 controls how fast those returns diminish, and B controls how much a long passage is marked down for having more room to mention things. 1.2 and 0.75 are OpenSearch’s defaults too.

The query goes through the same analyzer as the passages did, and to every shard at once. Each shard sends back only its own top five, which is all the merge needs:

python -m scriptorium.search lake bondmaid

1. 10.44  bible-kjv:23258          The King James Bible
   4:21 Tell me, ye that desire to be under the law, do ye not hear the law? 4:22 For it is written, that Abra...
2. 10.15  bible-kjv:2636           The King James Bible
   19:20 And whosoever lieth carnally with a woman, that is a bondmaid, betrothed to an husband, and not at al...
(4 shards, global idf, 2.1 ms)

Leviticus 19:20 is back. So are “café” and “Cæsar,” which now find the same passages as “cafe” and “Caesar.” Add --local-idf to see the per-shard scores from the chart in step 6.

If you’d rather start from the working version than from the listings above, clone scriptorium and follow the README.

Knowing when to write to the public

On the cost side: a JVM and a Python process on every executor, a driver that must not be lost, a cluster manager, and a publisher whose only job is to count. On a corpus the size of Gutenberg, one machine would be simpler, and nobody should feel bad about choosing it.

In exchange, you get a build that shrugs off a failed task, work spread evenly over as many machines as you have, and an index no single machine could have produced in time. That’s the trade Murray made: he gave up control of the reading to get it done at all, and spent his own effort on sorting, editing and checking.

It’s worth being precise about which parts did what. Spark’s retries protected the job. The partitioning kept the work even. But what protected the readers was the publisher, a count and a few known words checked before anything went live. The first volume of the OED had thirty careful people reading its proofs. It didn’t have a count.

The tools keep changing. Spark replaced MapReduce, object storage replaced HDFS, and something will eventually replace Spark. But read, sort, define, then check before you publish is still how large indexes get built, the same way Murray’s method outlived Murray.