A face is not a name

Most search errors in a digital asset manager are easy to correct with another query. Face recognition is different because a wrong result also makes a claim about a person. The earlier articles cover retrieval architecture, near-duplicate detection, multilingual indexes and search measurement; this one focuses on the additional constraints around identity.

If a library says that someone appears in a photograph, it may also imply that they were at a place, with other people, on a particular date. The cosine-distance query is straightforward; the harder requirement is making identity decisions auditable and explaining the evidence behind the threshold.

This article describes the pipeline and the measurement behind its default similarity threshold of 0.6, using a library of 22,944 assets. Without the underlying similarity distribution, that value would be an arbitrary constant.

Three jobs with different error costs

Before choosing a model or threshold, we separate three jobs that need different definitions of correctness.

The same index, the same distance metric, three incompatible definitions of a good result. The bottom row is the one that decides the architecture.

The first two rows are retrieval trade-offs that can use different thresholds. The third is an audit request: someone needs to know what the library records about a person and which parts were confirmed by a human.

Auditability therefore shapes the architecture alongside accuracy.

Separating detection from recognition

The terms are often used together, but they describe different operations.

Detection answers whether a region contains a face and where it is. It produces a bounding box and confidence without assigning an identity.

Recognition compares a detected face with known references and may assign it to a person, so an error is an incorrect identity claim rather than an inaccurate bounding box.

Systems that treat these as one pipeline stage end up with an architecture where a detection confidence and an identity confidence are the same number, and there is no point in the flow where a human can intervene, because there is no seam to intervene at. freedam keeps the seam. Detection is automatic and produces anonymous geometry. Recognition is a separate step against a set that a person built.

Detection remains anonymous, and identity enters only when a person links one detected face to a name.

The detector is a self-hosted InsightFace sidecar running the buffalo_l model pack at a 640 by 640 detection size on CPU. Its /detect endpoint accepts image bytes and returns boxes, detection scores and 512-dimensional ArcFace embeddings. Images stay inside the deployment, which is an important data-handling property for photographs of employees and customers.

The sidecar allows up to 50 faces per image, while the application keeps at most 20. Detection runs on its own queue with a five-minute timeout, three attempts and backoffs of 10, 60 and 300 seconds. A failure does not block the rest of ingestion, so assets remain available even when face processing finishes later.

Choosing the detector input

Face recall depends partly on the rendition passed to the detector, so this upstream input deserves the same attention as an index-time text decision.

The detector receives the W_1024 preview rather than the original file. DAM masters are often 40 or 80 megapixels, so using the preview bounds transfer and processing cost and remains below the sidecar's upload ceiling.

The likely cost is recall on small faces. The measured size distribution for detected faces on one library is shown below.

Measured on a replica of one library. This distribution covers detected faces only, so it cannot measure faces lost by using the preview.

We could not measure the recall cost directly because the replica contains the database but not the original files. A proper comparison would run the same images at full resolution and W_1024, so this article reports the detected-size distribution without claiming a recall figure for the faces the preview may have lost.

Downscaling cannot add visual detail, so the trade-off favours bounded processing at the likely expense of some small-face recall, although its size remains unmeasured.

The embedding is a biometric template

For each detected face the sidecar returns 512 floating-point numbers, normalised, from the ArcFace recognition head. In the database that is a vector(512) column alongside the bounding box, the detection probability, and a nullable recognition similarity, with an HNSW index over cosine distance restricted to rows that actually have an embedding.

Those 512 numbers are a biometric template. They are not directly viewable as a picture, but they can identify a person and need the access controls, retention policy and deletion lifecycle appropriate to biometric data rather than a generic cache column.

A cropped 150 by 150 thumbnail is stored too, with 20% padding around the box, JPEG quality 85 and metadata stripped, so a reviewer can see which face a row refers to without opening the original.

On the measured library, the faces table, HNSW index and out-of-line vector storage use 3.3 MB for 774 faces, or about 4.3 kB per face. This is small compared with the media itself, although the sensitivity of the data is unrelated to its storage size.

Why we do not cluster faces automatically

Consumer photo tools often cluster embeddings by similarity, present each cluster as an unnamed person and ask the user to supply names. We chose not to make clustering the starting point because it creates identity groupings before a person has asserted that the subject should exist in the library.

freedam does not do it, and the reason is the third row of figure 1.

Both approaches use the same embeddings and the same distance. They differ only in who is allowed to assert that two faces are one person.

There is an important qualification to that design.

The human decision happens once per person, not once per face. Somebody selects a face, gives it a name, and a Person is created. From that moment the person has a reference set, and subsequently detected faces are compared against it automatically: anything above the similarity threshold is assigned without further confirmation. There is also an opt-in bulk match at assignment time, which sweeps existing unassigned faces into the new person in one go.

A human does not confirm every face. A human creates a person from one confirmed face, after which the system extends that decision to similar faces automatically. This provides less assurance than per-face confirmation but keeps the first identity assertion outside an automatic clustering step.

That makes the similarity threshold especially important.

Measuring the 0.6 threshold

The threshold is a tenant setting, face_recognition.recognition_threshold, defaulting to 0.6 and clamped to the zero-to-one range. Faces at or above that cosine similarity to a person's reference face are assigned to that person.

The default is only meaningful with the distribution behind it. We therefore compared every possible pair among 774 faces in the measured library: 299,151 pairs without sampling.

Every pair of 774 real faces, exhaustively, no sampling. The four tallest bars are clipped to keep the right-hand mode legible; their true percentages are printed beside them.

The similarities form two populations. Pairs interpreted as different people cluster just above zero, with a median of 0.029, while same-person pairs cluster around 0.65 to 0.70. A trough near 0.30 contains 0.04% of all pairs.

Those labels are still an interpretation, so we looked for a structural source of known non-matches.

Pairs of distinct face detections in the same photograph provide non-matches without requiring names. There are 33 such pairs in this library, ranging from -0.021 to 0.167 with a mean of 0.057; none reaches 0.35 or approaches the 0.6 threshold.

This is a small sample from one library and visual style, but it provides independent support for the location of the lower-similarity population.

The same table also contains 1,647 pairs with a cosine similarity of exactly 1.0 across different assets, indicating repeated photographs. This provides an incidental connection with the near-duplicate work and also shows that repeated media can affect face-pair distributions.

What changes when the threshold moves

The distribution also lets us measure whether moving the threshold changes reachability or mostly changes the number of automatic assignments.

Reachability saturates almost immediately. Everything below the first few hundredths of the dial is spent on the density of the graph, not on whether anyone can be found.

At 0.6, 758 of 774 faces have at least one match in the library. Lowering the threshold to 0.35 raises that number to 766, while the number of pairs accepted as the same person increases from 23,691 to 31,162.

On this corpus, the threshold changes assertion volume much more than reachability. By the trough, most faces that have another match are already reachable, so lowering the value mainly creates additional automatic identity assignments and therefore more opportunities for false matches.

The 0.6 default sits above the trough on the rising edge of the same-person population. Compared with 0.35, it gives up eight reachable faces while declining 7,471 additional pair assignments. We therefore treat it as a limit on automatic assertions rather than a simple recall control.

The sixteen faces with no match at any threshold appear only once, so no similarity setting can connect them to another reference. The interface needs to leave room for such singletons rather than imply that every detected face can be identified automatically.

Growing reference sets can change earlier results

Assigning a face to a person makes that person's reference set larger, and a larger reference set can match faces that were previously unmatchable. A face photographed in profile might miss the one frontal reference it was compared against, then match comfortably once a profile shot has been added. So after any assignment, the system re-evaluates unassigned faces in the neighbourhood of the newly assigned one, and a separate command sweeps every unassigned face in the library against every known person.

As a result, adding a reference can change matches for images already in the library without a new upload or detection run. The identity is represented by a growing set rather than one fixed reference.

This helps find earlier photographs when a better reference becomes available, but it also means a data-protection review should account for retroactive matching. A conservative threshold matters here because a new reference can apply automatic assignments across the existing library.

Person names use the normal search engine

When a person has a name, that name is written into the asset's search document: the same per-language text document from the first article, built by the same builder, indexed by the same per-language BM25 index, fused with the same vector scores by the same reciprocal rank fusion. Names are de-duplicated case-insensitively, and the field carries a configurable weight like every other field, so an organisation that considers people more important than filenames can say so numerically.

Unnamed faces do not contribute to the search document or dictionary, so they cannot be found through a name query. This is enforced by the absence of an index entry rather than a query-time exclusion.

The consequence is that searching for a colleague's name uses the machinery this whole series has been describing, with every property it has: the recall guarantees under caps, the completeness reporting, the multilingual merge, the panel that measures it. There is no separate face-search subsystem to reason about, no second relevance model, no parallel ranking to keep consistent with the first one.

The boundary is therefore simple: detection produces anonymous geometry, a human introduces a name, and the existing text-search path retrieves that confirmed name.

What we would tell anyone building this

Separate detection from recognition in the architecture, not just the vocabulary. If there is no seam between anonymous geometry and asserted identity, there is nowhere for a human decision to live, and no way to answer "who decided this?".
Design for the withdrawal request, not the demo. The job with an expensive answer in both the false-match and the miss column is the one that fixes your architecture. Everything else is a threshold.
Publish the distribution or do not publish the threshold. A similarity default with no impostor and genuine distributions behind it is a magic constant, and it will be copied into someone else's system where it means something different.
Find labels your data gives you for free. Two faces in one photograph are two different people. That single structural fact produced a validation set with no annotation, no cost and no possibility of a labelling error.
Check whether your threshold is a recall control at all. Ours moves reachability by 1% and assertion volume by 32% over the same interval. Those are different products and the setting looks like the first one.
Know that a reference set makes identity retroactive. When adding a face changes what the system says about images uploaded years ago, that is a real behaviour with a real audience, and it should be a documented choice rather than a discovered one.
Decide early what a template's lifecycle is attached to. Giving biometric data its own independent lifecycle and tying it to something you already control are both defensible. Drifting into one of them without choosing is not.
Say what you cannot do. We could not measure the cost of detecting on a downscaled preview, so this article does not contain that number. An article that invented one would be worth less than an article that admits the gap.

Where this lives in the product

Face detection and person search run inside the same deployment as everything else: the InsightFace sidecar, the PostgreSQL instance holding the vectors, the queue workers. No image and no template is sent to a third-party service, which is a large part of what makes the whole system practical to self-host and is the difference between a feature a legal team can approve and one they cannot.

Once a person is named, they are searchable everywhere search reaches: the gallery, the REST API, the TypeScript SDK, and any AI agent connected over MCP. Who is allowed to run those searches is an access control question rather than a face question, and it is answered by the same permission model that governs every other asset in the library.

Every number in this article comes from one library, with all 299,151 pairs compared rather than sampled. A different mix of photography, lighting and framing may produce another distribution and threshold, so the transferable part is the method: compute the local pair distribution and use structurally known non-matches to validate its interpretation.

Next in this series: conversational search, where a language model is put in front of all of this and given exactly one job that has nothing to do with looking at pictures.