Measuring a search nobody thought was broken

Search can lose a large share of its matches while pages still load, tests remain green and the first results look plausible. Users cannot see what was omitted, so the usual product signals are weak evidence of recall.

This is the fourth article in a series on how search works inside freedam. The first covered retrieval: manufacturing a searchable document, fusing BM25 with pgvector, guaranteeing recall under caps. The second took text out of the problem and looked at perceptual hashing. The third put it back and asked which language the index is in. All three quote numbers. This one is about where those numbers came from.

We built a sixty-case relevance panel and ran it manually against a replica of a customer library containing 22,944 assets. The first serious run found fifteen hard failures. Over the next eight runs, the panel corrected a plausible diagnosis, measured the cost of a fix and exposed a gap in its own case set.

Replacing "pretty good" with a measurement

No search problem had been reported, and people were finding enough assets for the library to feel reasonably good in normal use.

The first serious run of the panel returned exit code 1: forty-five passes, fifteen hard failures, zero errored cases, across sixty cases on a 22,944-asset replica of a real retail catalogue.

The first run of the panel, by category. The shape is the finding: retrieval was perfect for queries that are also stored strings, and fell apart for everything a retailer would actually type.

This happens because people usually judge a search from its first few results. Whether another 178 matching assets were reachable is difficult to infer from the interface, and few users inspect positions deep enough to reveal a candidate-limit problem. Perceived quality therefore tells us little about recall beyond the top of the ranking.

For example, the panel found that galaxy a52 case returned twelve results even though 178 assets carried that device value. The discrepancy became visible once the expected set was written down and compared with the production path.

What a useful relevance case contains

A relevance case combines a claim about the corpus with an expectation about search behaviour, so the ground truth needs to survive normal changes to the replica.

Each case carries an id, a category, the query as a retailer would type it, ground truth, and assertions split into hard and soft. Hard assertions fail the run, while soft targets are reported without changing the verdict, a limitation discussed in the section on fault-testing the instrument. The interesting field is ground truth, and specifically the fact that it is almost never a list of asset ids.

The two lanes never touch. The search path resolves the query; the resolver resolves the specification straight against the tenant database through the asset model's own query builder, so soft-deleted and archived rows are outside both.

We derive most ground truth because the replica is periodically replaced from production and asset IDs do not survive that process. A case pinned to 178 identifiers would error after a re-snapshot, making the panel expensive to maintain and less likely to be run.

So a derived case declares what it means rather than what it matched last time. Four specification types cover the panel: a metadata field equal to a value, one matching a prefix or substring, membership of a named collection, and a token-level title match. The resolver runs the specification against the database at the start of each run and hands the evaluator a fresh id set.

The schema also allows curated ground truth as an explicit ID list for cases where relevance is a judgement rather than a database property. The current panel uses none because those cases require maintenance after a re-snapshot, and expressing an expectation as a specification often reveals a rule that can be derived instead.

Collection names are not unique, and metadata keys are unique only within an asset class, so both resolvers can legitimately match several records. Rather than pick the first and silently narrow the expected set, the panel marks the case as an error with exit code 2. An explicit error is easier to correct than a passing case based on the wrong ground truth.

Twelve cases have no ID ground truth because they cover did-you-mean hints, suggestions, autocomplete strings and negative queries. Since usefulness is partly subjective, each assist case records a rationale based on the live dropdown captured from the replica and reviewed by a person.

Recording the conditions of each run

Two things decide whether a relevance number means anything, and both are properties of the instrument rather than of the search: what configuration produced it, and what the candidate cap did to it before anybody looked.

Every run prints a settings snapshot before the first case executes: candidate limits, hybrid.rrf_k, fusion weights and whether typo and prefix fallbacks are enabled. A misspelling result depends on typo correction, just as recall depends on the candidate limit, so recording these values distinguishes a code change from a settings change. Runs 1 through 8 all used hybrid.limit 1000, hybrid.rrf_k 60, weights 1.0 and 1.0, with typo correction enabled.

The snapshot also showed prefix.enabled as null on the tenant used for the first three runs, meaning no settings row existed. The code default was true, so prefix matching still ran, but the report made clear that the behaviour came from a default rather than an explicit tenant choice. We now record both effective and stored values where that distinction matters.

The candidate cap also limits what a case can assert. On this tenant, a query with more than 1,000 plausible matches reports 1,000 because of the candidate window, not because the corpus contains exactly that number. A ground-truth set of 2,189 assets therefore cannot require full membership from that path; doing so would create a permanent failure that makes new regressions harder to notice.

Downgrading the assertion is a concession to the cap. Excusing the failure is not. The guard changes how the question is asked and refuses to change the answer.

Saturation does not excuse every missing match. The product guarantees that an asset whose title contains the query remains findable regardless of rank or cap, so a missing direct title match is still a failure. The report labels it not findable within candidate window, preserving both the verdict and its likely cause.

One sampling detail is worth copying. The sample is drawn by hashing rather than by a seeded random draw, so the same case draws the same fifty every run, and sampled recall 0.76 in run 1 against 0.76 in run 2 is a real comparison rather than two draws from the same urn.

Grouping failures by mechanism

The fifteen failures covered seven panel categories but reduced to six underlying mechanisms.

Sorting failures by category tells you where they show up. Sorting them by mechanism tells you how many fixes you need, which is a different and much smaller number.

The first mechanism was a useful example of why index terms need direct inspection.

Device compatibility values in this catalogue look like Galaxy A52/A52S and iPhone 12/12 Pro. PostgreSQL's English parser classifies a slash-joined value as a single token of type file, so Galaxy A52/A52S is indexed as the lexeme a52/a52s and a search for a52 cannot match it. The text was in the document; a substring search found it; the tsvector did not contain it. Across the corpus, 178 documents held the raw text A52 and twelve held the lexeme, those twelve arriving via space-separated titles.

The symptom initially suggested missing metadata or a stale index, but the metadata was present and rebuilding reproduced the same lexemes. We fixed the vocabulary normalisation so slash-joined labels emit their segments alongside the original value.

The typo dictionary used a different tokeniser that already split on slashes, so it reported a52 in 178 documents while BM25 could reach only twelve. This mismatch made the dictionary an unreliable proxy for terms stored in the BM25 index, so the panel now compares against the retrieval path itself.

Run 2 measured the tokenisation fix and the scope change separately, in one four-point design. galaxy a52 case went from 12 results to 178 with zero assets unfindable, iphone 12 case from sampled recall 0.06 to 1.00, and the two scoped cases from 1 of 28 and 10 of 24 to full membership. Between the scope-off and scope-on arms, nothing else in the panel moved at all: not a verdict, not a total, not a recall figure, not an MRR. The design constraint that unscoped searches pay zero additional cost stopped being an assertion and became a measurement.

Testing a plausible diagnosis

The tokenisation fix had a side effect. It first shipped applying its rule to the entire normalised document rather than to vocabulary labels only, so it split every slash-joined token from any source: EXIF shutter speeds like 1/100, voltages, URLs, EU directive references like 2009/125/EC, ordinary conjunctions. Roughly half the corpus grew, and documents carrying a slash averaged 3,758 characters against 917 for those that did not.

BM25 normalises by document length, so every document that grew lost a little score on every term it contained and every document that did not grow gained rank for free. Sure enough, five cases regressed on recall@10 between run 1 and run 2, two of them from a perfect score to zero. The analysis wrote itself: the expansion inflated document lengths, inflated lengths demoted the ground truth, and narrowing the expansion to vocabulary labels would reverse it while keeping the recall recovery.

We recorded that prediction before run 3, then narrowed the expansion, rebuilt the corpus and measured the affected cases.

The attribution was not lazy. It named a real mechanism, in the right subsystem, and it was wrong four times out of five. Tens of characters added to a document of hundreds is a rounding error next to a three-to-eight-times length ratio between a video clip's document and a product image's.

The narrowing did exactly what it was designed to do at the corpus level. The lexeme a52 still matched 178 documents; the lexeme 2009, which had been leaking out of 2009/125/EC, matched zero. And four of the five predicted reversals did not move by a hair.

The real mechanism was visible once someone stopped looking at tokens and started looking at lengths. The query greenland was being won by eleven Product Video documents of 155 to 493 characters, each essentially the title repeated by its field weight plus a little codec metadata. The first product image, which is what the case's ground truth consists of, arrived at rank 12 at 751 characters against a ground-truth average of 1,280. A three-to-eight-times length ratio decides an ordering on its own, and the document at the boundary contained no slash at all, so neither version of the expansion had ever touched it.

The diagnosis was coherent and specific enough to survive review, but the per-case prediction made it testable. Measurement showed that four of the five expected reversals did not occur, which redirected the investigation toward document length rather than token expansion.

The narrowing shipped regardless, because it was still the right change. It also cost a case. compat-iphonexxs-01, the query iphone x case, went from sampled recall 1.00 to 0.74 and flipped from pass to hard failure, because its result set saturates at the cap in every run and the over-broad expansion had happened to lengthen its competitors just enough to admit more iPhone X and Xs assets. Take the pollution away, the competitors get shorter, score better, and crowd back in. The panel went from 48 passes to 47, and the trade-off went into the record alongside the fix.

Ranking inside a capped window is close to zero-sum, so an improvement can displace another result. When a change shows only gains, we check whether the case set covers what moved down. Known regressions remain in the report with their rationale so they are distinguishable from new failures.

The useful habit is to record per-case predictions before the run, including cases expected not to move. Run 2 matched ten predictions, while run 3 matched one of five; both were informative because the negative controls exposed whether the proposed mechanism was correct.

A panel can only see what its cases cover

Run 4 was a canary for a different feature: the title-findability guarantee, which promises that an asset whose title contains the query is findable by that query regardless of rank or cap. Control arm off, treatment arm on, same corpus, same build, same settings. One caveat travels with every number from here on: run 4 measured a fresh replica of production, which has deliberately not been re-indexed since the tokenisation fix shipped, so its corpus is the pre-fix one. Both arms share it, which is all an A/B needs, but the run-4 verdicts are not comparable case by case with run 3 even where the totals coincide.

The result was flat on verdicts, 47 passes in both arms, and unambiguous on metrics. Mean recall@10 fell from 0.6667 to 0.6333, mean recall@50 from 0.7446 to 0.7338, and three cases regressed. Not one case improved on any relevance metric.

Taken alone, the result suggested that the feature reduced relevance without adding value. However, none of the sixty cases asked whether an asset beyond the cap remained findable by its title, which was the feature's contract. The existing cases used metadata-derived ground truth, so the panel could measure the cost but not the intended benefit.

Run 4 therefore identified a coverage gap, and run 5 added three title-findability cases.

Two cases the feature is supposed to fix, and one it is supposed to leave alone. Without the third, a clean sweep would show that the cases were written to the answer rather than that the guarantee works.

The control case is the one that looks like padding. Its title band is 188 assets: below the 1,000 candidate cap, and below the 200-asset threshold at which membership assertions downgrade to sampling. Nothing is ever cut, so it passes with the guarantee off and with it on, identical in both arms.

One case uses a title band below the candidate cap and must pass with the feature both off and on. This control distinguishes the intended guarantee from a broader change that reorders results already reachable without it. It remained green and unchanged while the two beyond-cap cases measured the new behaviour.

The addition also let the panel see something the sixty cases never could. title-product-01 returned 2,587 results against a ground truth of 2,156, meaning 431 non-title matches survived alongside a band more than twice the size of the cap. An earlier version of the guarantee had numbered the band and the ranked pool together, so the band took every position under the cap, got re-admitted anyway by its own clause, and silently deleted the ranked matches it displaced. Those 431 assets are that fix, visible on a real corpus.

The panel also showed that the first attempt to demote title-band rows had no effect: run 6 was byte-identical to run 5b. The ordering had been changed inside a subquery, but the two consumers that determine user-visible order did not preserve it. Applying the demotion in both consumers removed the measured regressions across all sixty-three cases while the two beyond-cap title cases continued to pass.

Query-log replay measures change, not correctness

Everything so far comes from one instrument with a fixed blind spot. Its cases probe failure modes we already suspected and its ground truth is derived from metadata, so it answers "is the right thing findable" precisely but cannot answer "would a real user notice". The title-findability experiment above showed why that distinction matters.

The second instrument replays production queries without ground truth. It compares two arms on the same corpus and reports what moved, so it measures change rather than correctness: an identical top ten is unchanged but not verified, and a larger result set contains more reachable assets without proving they are relevant.

Neither instrument is a superset of the other. The panel knows what the right answer is and does not know what anybody asked; the replay knows what everybody asked and has no idea what the right answer is.

The source contained 140 searches covering 99 distinct terms over ten weeks on one production tenant. We extracted only the term, occurrence count, search type and result count; user and session identifiers were not included because the comparison did not need them.

Of the 99 distinct terms, 55 were keyword searches and 44 were serialised advanced-filter trees. We excluded the trees because those without a full-text leaf never reach the hybrid path and therefore could not change in this comparison. The remaining searches included SKUs, filenames, brand phrases, descriptions, case variants, a misspelling, a Danish term and one paste containing 79 SKUs.

The top ten remained identical for all 55 searches. Five totals grew, each for a query already at the 1,000-candidate cap, so the change affected only beyond-cap reachability: greenland moved from 1,000 to 1,014, iphone to 1,010, and phone and blue products to 1,005. No query lost results, and the two empty queries remained empty.

This is evidence about the change's reach, not its quality. The curated panel deliberately concentrates on sensitive mechanisms, while the replay reflects queries customers happened to make; the two instruments answer different questions.

It did find one thing the panel could not. With the band merged in, the reported candidate limit became null on 52 of the 55 queries while the truncation flag stayed set on all of them, so the interface goes from "may be truncated, limit 1000" to "may be truncated, depth not describable by one number". That is honest, and it is a copy decision to make before rollout rather than discover afterwards.

The first latency pass incorrectly suggested that the title band made search faster because the arms ran sequentially and the second inherited warm caches. Comparing warm runs against warm runs left about five milliseconds of mean difference, with the slowest individual queries eight to ten milliseconds worse. Both arms therefore need the same cache conditions.

The limit is the sample. Fifty-five distinct terms from one tenant over ten weeks, skewed towards SKU lookups, is a small window on exactly the query shape least likely to be affected by the thing being tested.

Verifying that the checks can fail

The instrument itself also needed fault testing. We found four checks that appeared in the test or monitoring output but produced the same signal when the behaviour they watched was broken.

A computed exit code that nothing consumed. The runner carefully returned 0 for pass, 1 for hard failures and 2 for configuration errors. The trait that invoked it discarded the integer and reacted only to a thrown exception, and the runner was deliberately designed never to throw, so every real invocation exited 0 regardless of how many cases failed. The suite's own tests passed, because the test runtime takes an earlier branch that does propagate the return value. The only exercised path was the only correct one.

Metrics that report but cannot gate. Soft targets emit a note saying the first relevant result arrived at rank 17 against a target of 10, and notes do not move an exit code. After the document-builder change, five cases lost top-ten relevance, two falling from perfect to zero, and four of them still reported pass.

A monitor that reported silence while the job it watched died, because it filtered for progress markers only and a crash produces no marker. A tie-break test that stayed green after the tie-break was stripped out, because the assertion happened to hold for reasons unrelated to the behaviour it named.

In each case, a failure signal existed but the next layer did not consume it according to the same contract.

For each gate, we now ask what it would output if the watched behaviour broke, then introduce a controlled fault and confirm that the result turns red. The two defects found by this method were easier to locate than the two discovered through code reading.

The exit-code defect would have made full replica runs report success regardless of hard failures. Soft targets remain diagnostic rather than gating, so the report now labels them clearly; otherwise a displayed metric can easily be mistaken for an enforced guarantee.

What we would tell anyone building this

The transferable version, stripped of our specifics.

Assume you are wrong about your own search until an instrument disagrees. Perceived quality has no resolution below the top three results, so a system can lose most of a result set and still feel fine to everyone who uses it. Fifteen failures in sixty cases, and not one support ticket.
Express ground truth as a specification, not a list of ids. A panel that dies when the corpus is replaced stops being run, and a suite nobody runs is deleted within the month. Curated cases are future maintenance events; keep them near zero.
Make the report state the configuration it measured, and print stored values rather than effective ones. A recall figure without its candidate cap is a rumour, and the gap between stored and effective tells you whether anybody actually decided.
Downgrade the assertion under a cap, never the verdict. Sample instead of asserting full membership, but keep a missing direct match a hard failure, or the instrument goes blind to exactly the defect class it was built for.
Sort failures by mechanism before you fix any of them. Five failures in one category were three bugs needing three fixes, and only one was about retrieval limits. Fixing a cause moves several cases at once; fixing a case moves one and breaks another.
Write the prediction down before the run, per case, including what should not move, and expect a bill. The negative predictions expose a misdiagnosis. Ranking is close to zero-sum inside a cap, so a change that shows only improvements usually means nothing in your case set points at what you displaced.
Check whether your instrument can see the thing you are asking it about, and include a control that must pass with the feature off. Zero better and three worse was a true measurement and a false conclusion, because not one case exercised the contract under test. A case set where everything only passes with the feature on shows the cases were written to the answer.
Run two instruments with different blind spots, and ask of every gate what it would output if the thing it watches broke. A curated panel knows the right answer and not what anyone asked; a query log knows what everyone asked and not the right answer. Then break your checks on purpose and confirm red, because reading finds these only if you were already suspicious.

Where this lives in the product

None of this is a feature. It is the instrument that decides which features ship, and its output is visible only in the behaviour of ordinary searches: the gallery, the REST API, the TypeScript SDK, and any AI agent connected over MCP. The fixes it arbitrated are the ones described in the first article: the recall guarantees under caps, the candidate selection, the completeness reporting that lets a truncated result say so. The same subject at product altitude is search and discovery, and the fact that all of it runs inside one PostgreSQL instance is what makes it reproducible on a replica at all, which is also what makes freedam practical to self-host.

The limits deserve stating as plainly as the results, because an instrument that oversells itself is the exact failure this article is about. The panel is manually run and deliberately not in continuous integration: it needs a replica of a real customer corpus, an embedding sidecar and about ninety seconds per run, and its numbers mean nothing against any other corpus. Its queries are English only, because the catalogue it was built for is English, so it measures none of the multilingual behaviour in the previous article. Its cases resolve their ground truth freshly on every run and survive a re-snapshot, but a changed corpus still changes the numbers: runs 4 through 8 measured a fresh replica of production, un-reindexed since the tokenisation fix, so they are not comparable case by case with run 3 even where the totals happen to match.

Next in this series: faces and person search, where the interesting problems stop being about retrieval quality and start being about who is allowed to assert that two faces are one person.