What it actually takes to build a good image search

In February 2000, Jennifer Lopez wore a green Versace dress to the Grammys, and for a few weeks afterwards it was the most searched thing on Google. Google could only return ten blue links, some of which led to pages that might contain a photograph. Eric Schmidt later wrote that it was the most popular query they had ever seen, and that it was the reason they built an image search at all. Google Images launched in July 2001, indexing about 250 million pictures.

I like this story because of the part people leave out: Google Images did not search the pixels. It indexed the filename, alt text, caption and words near an <img> tag, then matched the query against that text. The dress was findable because thousands of pages had written "Jennifer Lopez", "Versace" and "Grammys" next to it.

Twenty-five years and several generations of computer vision later, image search still begins by creating something searchable from or around the image, which explains many of the practical problems that follow.

I run freedam, a digital asset manager for the photographs, videos and documents a company needs to find again. We have spent more engineering time on search than on any other part of it, so this article collects what we learned for people building one or trying to understand why theirs misses things. There is also a more technical version with the SQL and parameters; this one stays with the shape of the problem.

A picture has no words in it

A row in an orders table contains a customer name, a support ticket contains a body and a PDF may have a text layer. A JPEG has three colour channels and some EXIF, but no terms that a text query can match directly.

An image search therefore needs a searchable stand-in for the picture. In practice, those stand-ins fall into four broad groups, and each one covers queries that the others do not.

Words someone typed. A filename, title, caption or the text around an image. This is what Google Images launched on, and what AltaVista's Photo Finder had launched on three years earlier. It is cheap and often effective because people are good at naming things, but it cannot help when a folder contains only files named DSC_0001 through DSC_4817.

Words from a vocabulary. Tags, categories and taxonomies are precise and filterable, which is why stock agencies and corporate libraries rely on them. They are also expensive to maintain, and the vocabulary tends to drift away from the language people use to search.

Words a machine wrote. This includes OCR from the image, captions from a vision model and transcripts from video. Luis von Ahn's ESP Game used a human version in 2004: two strangers saw the same picture, scored points when they entered the same word, and their agreement became a label. Google licensed it as Google Image Labeler in 2006 and ran it for five years. Models can now produce these labels at scale, although the output is noisy and lacks business context: a model may recognise a sneaker without knowing it is SKU 4471-RD.

No words at all. An image can be represented as numbers and compared with other numerical representations. IBM's QBIC did this in 1995 by letting people sketch a colour layout. TinEye in 2008 and Google's search-by-image in 2011 made the reverse form familiar: provide a picture and find copies of it. CLIP later put text and images into the same numerical space, so a query such as a dog on a skateboard could find an image nobody had tagged with those words. This is the basis of most recent "AI search" demos.

Nothing on this list replaced what came before it. Google Images still uses surrounding text and stock agencies still pay people to tag images, because each representation answers questions the others cannot. Our first mistake was to treat one of them as the main answer rather than combine them.

What the research was chasing in the meantime

The research running alongside those products spent roughly thirty years on one question: how to reduce the gap between visual features and the meaning expressed by a query. Recent models made substantial progress, although the result is narrower than many demos suggest.

The phrase "content-based image retrieval" was coined in 1992, by Toshikazu Kato at Japan's Electrotechnical Laboratory, for the idea that you could search pictures by what was in them rather than by what someone had written about them. IBM's QBIC was the first serious system to try. It could match colour histograms, textures and shapes, and you queried it by sketching. AltaVista's Photo Finder in 1998, built with a company called Virage, actually shipped a "visually similar" button to the public, and the public, overwhelmingly, typed words instead.

The reason was named in 2000, in a survey paper by Smeulders and colleagues that every researcher in the field has cited since: the semantic gap. The gap between what a computer can extract from pixels (this region is greenish, this edge is vertical) and what a person means by a query (a dress, a celebrity, an awards ceremony). Colour histograms live on one side of that gap. Every query anyone actually types lives on the other. For the next twenty years the research programme was, essentially, "close the gap".

It closed in stages, and each stage ended up in a product. Local features (SIFT, 1999) and the "Video Google" trick of 2003, which treated visual patches as words and indexed them like text, are the direct ancestors of TinEye and Google's reverse image search: same-picture matching at web scale, which is a different and easier problem than meaning. ImageNet in 2009 made a million labelled images available, and AlexNet in 2012 showed that a neural network trained on them produced features that beat every hand-designed descriptor. Then in 2013 a paper from Google called DeViSE did something that now seems obvious and then seemed strange: it trained the image features to land near the word vectors for their labels, so that an image of a thing and the name of the thing sat close together in one space, and a model could recognise categories it had never been trained on. CLIP in 2021 was that idea scaled to 400 million image-text pairs scraped from the web. The gap that had been the field's central problem since 1992 was, for descriptive queries, mostly closed.

The important qualifier is descriptive. CLIP can find a dog on a skateboard, but it cannot reliably find SS26-HERO-03, understand that the current search is limited to one collection, or report that a candidate limit excluded 953 results. Those are product and retrieval problems, handled by a different set of tools: BM25 for keyword ranking, reciprocal rank fusion for combining result lists, and HNSW for fast nearest-neighbour search over vectors. Semantic retrieval is useful, but it remains one component of the search engine.

What people actually expect

Before the architecture, I want to spend some time on the users, because the expectations are where most of the design constraints come from, and they are not what a search engineer would choose.

Most people learned the interaction from Google Images, where results arrive quickly, the grid rarely ends and there is no single correct answer to sunset. If one of the first few images is useful, the search has done its job.

A company's photo library is a different animal, and the Google mental model is actively harmful there, in ways that took me years to see.

In a company library, the answer is usually more specific: the sunset from the Lisbon shoot, with the bridge, used in last year's annual report. There may be one correct result or twelve, so a grid of 400 plausible sunsets is not particularly useful.

Many library queries are not descriptions at all. The logs contain 4471-RD, SS26-HERO-03, invoice numbers and model names, which are common in a DAM and poorly suited to semantic search because the nearest neighbours of a SKU are usually other SKUs.

People also search inside a folder, collection or campaign. "Find the hero shots in this shoot" is a common request, and candidate limits can make this type of search incomplete without making the result page look broken.

"No results" can also be useful. If nobody has photographed the new store in Madrid, the user may need to commission the work, while filling the grid with photographs of Barcelona and Seville only hides that information.

Finally, people act on whole result sets: "Select all 1,240 results and apply this licence expiry." That is only safe when 1,240 really is the complete set.

People tend to judge search by its first few results, so missing matches are hard to notice. A query can lose two thirds of its relevant assets and still return a plausible page in ninety milliseconds. In one library that everyone, including us, considered reasonably good, measurement found hard failures in fifteen of sixty realistic queries, none of which had been reported.

For a company library, then, a good image search needs to find specific assets, handle identifiers as well as descriptions, work inside a scope, return an honest empty result and report whether candidate limits may have excluded matches. The rest of this article explains how we approached those requirements.

Indexing sets the limit

We initially spent too much time on ranking because it is the visible part of search, but many of our recall problems began at indexing. A query cannot recover a term that was never placed in the searchable representation of the asset.

We therefore compile a stored search document for each asset and language, rebuilding it when the asset changes. It includes the title, filename, description, creator, copyright, location, keywords, generated tags and captions, OCR, custom metadata, collection names, confirmed person names and video subtitles. Controlled-vocabulary terms also include their ancestors, so an asset tagged espresso machine can be found through small appliances or kitchen without the user knowing the taxonomy.

Two things about this document turned out to matter far more than I expected.

The first is weighting. A title should count for more than a line of OCR, and with a keyword index that has no notion of fields, we represent that weight by repeating important text in the document. This works but also changes document length. Because keyword scoring normalises for length, the repetition can favour shorter documents in a tie. In one place-name query, eleven product videos outranked the product photographs: the video documents contained 155 to 493 characters, while the richer image documents contained 746 to 1,280.

The second is tokenisation, which decides where one term ends and the next begins. We found a small but useful example in compatibility labels such as Galaxy A52/A52S: PostgreSQL treated the value as one path-like token, so a query containing a52 missed assets that visibly carried the label. Normalising separators before indexing fixed the case. The broader lesson is simply to inspect the terms produced by the parser when text is present but recall is unexpectedly low, because rebuilding the same tokens will not change the result.

Two ways to search, one database

Once you have the document, there are two useful ways to search it, and they are good at opposite things.

The first is keyword search using BM25. The name is not especially helpful (it stands for "Best Match", and 25 is the version number that stuck), but the idea is simple: a document scores higher when a query term appears more often, when the term is rare across the library, and when the document is short. The formula dates from 1994 and remains effective for exact terms such as 4471-RD. Its weakness is vocabulary, because when the photographer wrote autumn and the marketer searches for fall, there may be no shared term to rank.

The second is the CLIP-style approach: turn the document and image into points in a 768-dimensional space, put the query in the same space and return nearby points. This can find team celebrating outdoors when nobody used those words, but a vector index always returns k neighbours, including when none is a good match. A distance threshold is therefore part of the retrieval design rather than an optional ranking tweak.

We run both inside PostgreSQL on the same instance as the assets: BM25 through pg_textsearch, and vectors through pgvector with an HNSW index. This avoids a separate search service and its synchronisation pipeline.

The trade-off is straightforward. A new asset can become searchable in the transaction that inserts it, while permissions, embargoes and usage rights remain ordinary WHERE clauses instead of a second authorisation model. Self-hosted installations also have one fewer stateful service to operate. In return, query shape matters a great deal: in one case, referencing the same subquery twice changed a 6 ms query into a 7.3 second one because the planner lost the row bound required for a top-k index scan. Much of the work is therefore in checking plans and keeping the intended bounds visible to PostgreSQL.

The embeddings come from nomic text and vision models trained to share a space, running in a self-hosted sidecar so customer images do not leave the installation. Two operational details mattered. First, the models use task prefixes: documents are embedded as search_document and queries as search_query, so those inputs need coverage because an incorrect prefix reduces quality without producing an error. Second, we cache failed query-embedding attempts briefly, allowing search to fall back to lexical retrieval instead of contacting an unavailable sidecar on every keystroke.

How close is close enough?

The vector lane needs a cutoff beyond which a neighbour is no longer considered a match. We expected one threshold, but the useful value varied with query length and was not monotonic.

Query	Max cosine distance
1 word	0.35
2 words	0.42
3 to 5 words	0.45
6 to 10 words	0.48
more than 10 words	0.40

Both ends are strict, for opposite reasons. A single word like chair has a gigantic semantic neighbourhood: at a loose threshold it matches every interior photograph in the library, fills the page, and looks authoritative while being useless. A very long query, which in practice is someone pasting an entire product title, produces a vague, averaged-out embedding that sits weakly near everything; loosen the threshold there and you get the whole corpus back, in confidence order. The middle band, where the query is specific enough to mean something and short enough to keep meaning it, is where semantic search earns its keep.

Single-word queries get one additional rule: a result must also have a lexical hit. Without it, niche terms tend to fill the page with semantically adjacent images, making an empty result look like a large one. With the rule, madrid can return no results when the library contains no matching text.

Combining two rankings that do not speak the same language

A BM25 score and a cosine distance are not directly comparable. Normalising them onto a common scale makes the combination depend on the corpus, language and thresholds used to produce that result set.

We combine ranks instead. Each lane produces an ordered list; a document at position r contributes 1 / (60 + r), while a document absent from a list contributes nothing from that lane. Reciprocal Rank Fusion is simple, does not require score calibration, and lets a document that ranks well in both lists outrank one supported by only one retrieval method.

Candidate limits can hide matches

Everything so far concerns relevance, but candidate limits determine whether relevant assets are reachable at all.

Every retrieval system needs a bound. Rather than score 200,000 documents per keystroke, it may take the top 1,000 lexical candidates and top 200 vector candidates, then rank that smaller set. Unless the response reports the bound, the page and paginator can look normal while assets beyond the cut are absent.

For example, a collection contains 400 assets and 30 carry the query term, but the search returns four. The top 1,000 candidates were selected across the whole library before the collection filter was applied, so the other 26 fell outside the candidate set. The returned assets are relevant; the problem is that most eligible matches were never available for ranking.

The more general case is cap crowding: if 1,953 documents contain every term and only 1,000 candidates are kept, equally relevant rows may be excluded by a tie-break.

Our fix is a second, guaranteed retrieval for rows that must remain reachable, such as members of the collection being searched and assets whose title literally contains the term. These rows enter after the normally ranked scan positions, and every downstream limit admits them explicitly. The first page keeps the same order, while the additional rows remain available on later pages and in the total.

The guaranteed retrieval changes with scope size. For a collection of forty assets, we score all forty directly, which is complete by construction and costs about half a millisecond per member. Up to a couple of thousand members, we add an exact vector comparison over the scope rather than use the approximate index. Up to fifty thousand, we scan the ranked list more deeply, while larger scopes overlap the normal candidate pool enough that the extra work is less useful.

Reporting whether results are complete

Candidate limits lead to a practical question: can the system tell the user whether it has shown every match?

This matters well beyond user comfort. "Select all 1,240 and apply this rights policy" is only safe if 1,240 is provably the whole set. A search that silently truncated at 1,000 turns a bulk operation into a data-integrity incident that nobody notices for months.

So every freedam search returns a verdict alongside its results: complete, truncated, or unknown. It is computed by extra counting columns in the same query, which measure how many rows each candidate source actually produced against the cap that source was operating under. If a source hit its bound, the result is truncated, and the interface says so, and bulk operations on it warn you. Scoring every member of a small collection is complete by construction. A deep scan is complete only if it finished before its row bound. A new candidate source defaults to unknown until someone proves otherwise. And a result reached through a fallback, a typo correction or a prefix expansion, never inherits the verdict the original query earned.

This also changed how we review monitoring and tests. We had a monitor that stayed quiet after its job died, a tie-break test that remained green after the tie-break disappeared from the query, and a quality suite whose warnings could not change the exit code. For each check, we now ask what its output would be if the behaviour broke. If failure produces the same output as success, the check needs another observable assertion.

The other ways people ask

Text is one input. Three more matter for images specifically, and I will be brief because each has its own article.

You can search with a picture to ask whether it was already licensed or find the rest of a shoot. This path uses perceptual hashes and a multi-index lookup to generate candidates, then a vision embedding to verify them. The near-duplicate article covers the catalogue styles that weaken the standard algorithm.

You can search for a person. Faces get their own detector and embedding space in a sidecar that does not send images outside the installation. A face has no name until a person confirms it, so the system groups candidates and writes only confirmed names into the search document. The details are in the person-search article.

You can also search with a sentence. Our assistant does not rank assets; it compiles the sentence into the same filter tree produced by the advanced-search interface, with an explicit intent such as replace, refine, add or remove. That tree runs through the normal retrieval path under the same permissions and completeness reporting, so the user can inspect and correct the interpretation. The conversational-search article covers that boundary and its trade-offs.

How we know any of it works

We evaluate these decisions with sixty realistic queries and agreed ground truth, run through the production search path against a replica of a customer library of about 23,000 assets. The instrument reports hard pass/fail, recall and reciprocal rank, and it has read-only access so it cannot change settings or rebuild an index. The measurement article explains the method and its limitations.

The first serious run found fifteen hard failures in a search everyone had described as reasonably good. Later, we attributed five recall regressions to a tokenisation change, but measurement showed that four queries had identical recall and rank to fifteen decimal places. The actual cause was document length, as in the video-versus-photo case above. The instrument was useful because it separated a plausible explanation from the change that affected the result.

Back to the dress

Google found the dress in 2001 by reading the words around the picture. We find a product shot today by searching the text assembled around it and a vector representation produced from the image and its metadata. Semantic models help with descriptive queries, while the rest of the system still has to handle identifiers, access rules, scopes, candidate limits and honest totals.

If you are starting out, spend time on the indexed representation because a query cannot recover terms the tokenizer never emitted. Run lexical and semantic retrieval together and fuse their ranks rather than their scores. Tune vector thresholds by query shape, consider requiring lexical support for single words, and treat every candidate cap as a possible completeness issue. Most importantly, expose when the system cannot prove that it returned every match.

If you would rather see it than read about it, there is a demo with a seeded library. The interesting thing to do is search for something nobody would have tagged, and then search for something inside a folder and count.