The chat that never sees your library

A common first version of conversational search sends a sentence and a sample of assets to a language model, then asks the model to choose the relevant ones. This can work in a small demo, but it does not scale with the library and cannot account for assets outside the context window.

freedam instead uses the model only to turn a sentence into a filter tree; the model does not receive assets or rank them. The tree is executed by the retrieval engine described in the architecture article, with BM25 and pgvector in PostgreSQL under the same access rules and result-count logic as the gallery.

This article covers the benefits and costs of that boundary, including state that survives transcript truncation, replies produced before retrieval runs and three current limitations in the rule-tree merge.

Separating interpretation from retrieval

"Search with AI" combines interpretation and retrieval, but the two jobs have different cost profiles.

Interpreting a query is approximately fixed cost. "Something calm for the newsletter header, landscape, not the ones we used last spring" contains a mood, an aspect ratio and a temporal exclusion, and extracting those constraints costs one model call whether the library contains four hundred assets or four hundred thousand.

Retrieving from the corpus is a high-volume job whose cost grows with the library, so it belongs in indexed database queries rather than a model context.

The boundary also affects result reporting. A retrieval engine can say that it found 1,247 matches or that a candidate cap makes the total a lower bound, using the completeness reporting shared by the rest of the search stack. A model given only a page of assets cannot distinguish an empty corpus from an incomplete sample, which makes its answer unsuitable for bulk operations.

We therefore let the model interpret language and let PostgreSQL retrieve from the library.

What the model produces

One turn produces exactly one JSON object, and it has three keys.

{
    "intent": "refine",
    "rule_group": {
        "type": "group",
        "operator": "and",
        "rules": [
            {
                "field": "full_text_search",
                "operator": "hybrid_search",
                "value": "sunset"
            },
            {
                "field": "asset.aspect_ratio_orientation",
                "operator": "equals",
                "value": "landscape"
            }
        ]
    },
    "explanation": "Narrowing to landscape shots with sunset colours."
}

rule_group is the same rule tree built by the gallery's advanced-filter editor, serialised by saved searches and compiled into SQL. Reusing that structure avoids a chat-specific query language and keeps conversational search on the same ranking and retrieval path as the rest of the product.

intent says what to do with the tree relative to the conversation so far. explanation is the only natural language the user ever sees, and the system prompt pins it to one short sentence in the user's own interface locale, which matters more than it sounds and is where the multilingual work shows up again: field names, operator names and intent values stay in English because they are identifiers, while the sentence around them is generated in French or Simplified Chinese.

We use temperature 0.1, a json_object response format and, when supported, OpenRouter's Exacto routing variant for providers vetted for structured output. The default is the mid-size open-weight openai/gpt-oss-120b model because the task is translation into a schema that is validated downstream.

Generating the prompt from the tenant schema

On a development tenant with 22,944 assets and 48 metadata definitions, the system prompt measures 36,790 characters across 646 lines. Of those, 16,646 characters, roughly 45%, are generated from the tenant's database for the current request.

The fixed part, 19,840 characters, is the instruction skeleton: the output schema, the six intents, the search-method priority order, the worked examples, and a list of things not to do that grew one entry at a time from watching real failures. The generated part is a catalogue of the tenant's business vocabulary, assembled per asset class, and it is what makes the same prompt behave completely differently for a furniture retailer and an aviation parts supplier.

Three decisions in the generated section are useful beyond this implementation.

Sample the values, not just the schema. For every text field, the prompt carries the twenty-five most frequent values actually present in the library. For every number and date field, the observed minimum and maximum. For every vocabulary field, up to fifty real term labels. A model that has been told a field named product_designer exists will guess at it; a model that has been shown that its values look like Henning Koppel and Louise Adelborg will map a surname onto it correctly on the first try.

Never make the model guess an identifier. Vocabulary terms are foreign keys, and a model asked to produce a term ID will invent one that parses. So the operator set includes vocab_text_equals and vocab_text_in, which take human text, and the resolution to an ID happens afterwards in the compiler using PostgreSQL trigram similarity with a 0.45 threshold and an exact case-insensitive match tried first. The model contributes the thing it is good at, which is deciding that "Alfredo" in this sentence is a designer. The database contributes the thing it is good at, which is knowing that the designer is Alfredo Häberli. This is the same division of labour as the whole architecture, applied one level down.

Measure the schema selector rather than assuming it is semantic. Six high-priority fields are always present, while a keyword matcher selects the rest from field labels, keys and keywords. On two realistic queries it contributed two and three fields out of 86, and the cap of 25 did not bind. This remains adequate because the prompt also contains six always-on fields and the full metadata catalogue, not because substring matching is a strong retrieval method.

Validating and retrying structured output

Valid JSON is only the first validation step. A well-formed tree may still reference asset.file_type, for example, even though that field does not exist.

So validation runs in two layers. The first checks structure: the three required keys, an intent drawn from the permitted six, a rule_group carrying type, operator and rules. The second walks the tree recursively and checks every leaf's field name against the same registry the gallery's filter editor reads. An unregistered field is a hard rejection, not a warning, because a filter on a nonexistent field either fails to compile or, worse, compiles into something that silently matches nothing.

When rejection happens, the errors do not become an exception. They become the next user turn:

ERROR: Your previous response had validation errors:
Invalid field 'asset.file_type': The field 'file_type' does not exist.
Use 'asset.mime_type' (for MIME types like 'image/jpeg') or
'asset.extension' (for file extensions like 'jpg') instead.

Please correct these errors and provide a valid response
following the exact schema.

The invalid response is appended as the assistant turn and the validation error as the next user turn, then both are sent back for one correction attempt. Common mistakes include remediation text, because naming the permitted replacement is more useful than only saying the field is unregistered.

Two retry loops are nested. Transport failures get one retry after a two second backoff. Validation failures get one retry with feedback. The worst case before the user sees a fallback message is therefore four model calls, and with a 45 second per-call timeout that is an upper bound that comfortably exceeds most reverse proxy patience. We have not measured how often the outer loop is reached in production, and the honest reading of that is that the bound is a design fact rather than an observed one.

After structural validation, leaf values receive domain-specific normalisation. Colour values may arrive as #f00, FF0000 or {"hex": "#ff0000"}, so the constructor expands short hex, normalises case and punctuation, and supplies the default tolerance when needed. This accepts equivalent spellings without spending another model round trip.

Keeping state in the rule tree

A simple design makes the transcript the state and asks the model to reconstruct the accumulated filters on every turn. That reconstruction can drift as the transcript grows, and it leaves no explicit object that the gallery can open, save or share.

In freedam the state is the rule tree, stored on the session, updated once per turn. The transcript is context, capped at the last ten messages, and if it were dropped entirely the current search would survive intact. The model is told the current tree in full on every turn, and is asked not to restate it, only to describe the delta and label the delta with an intent.

Because state is a tree, "open in gallery" can pass the accumulated filters to the normal rule editor, where users can inspect, save, share or automate them. The handoff only adapts the top-level shape. Production trees may contain several wrapper layers, so traversal and tests both need nested fixtures rather than assuming every filter is a direct child of the root.

The current merge has three known limitations, found during code review rather than from support reports.

Refining an existing field and operator does not replace its value. Deduplication compares the field and operator but ignores the value, so refining width > 1000 with width > 2000 keeps the first rule. This handles repeated filters but not value updates.

Removal requires an exact value match. "Drop the width filter" succeeds only when the model reproduces the stored field, operator and value exactly; otherwise the tree remains unchanged.

Removal only walks the top level. Nested groups, including the OR groups generated for palette searches, cannot be removed through the targeted path, although resetting the search still works.

None of these cases throws an error, and the generated explanation can still describe the requested change. They therefore need post-merge assertions that compare the requested operation with the resulting tree, following the fault-testing approach in the measurement article.

Remembering an ambiguous term

"Landscape" can describe an orientation or a subject, and both readings are common in a photo library. The system therefore asks for clarification instead of choosing one implicitly.

The handling is small and worth copying. The word is tracked in the session as a three-state memory, stored alongside the rule tree.

Two properties make it work. It is inferred from behaviour: the resolution is derived by inspecting the merged tree for an orientation filter or for the word appearing inside a search value, never from the model claiming which reading it chose. And it is sticky: once resolved, the interpretation is asserted in every subsequent prompt, so turn seven does not quietly flip the meaning that turn two established.

When the term is unresolved, the model returns the clarify intent, which changes no filters and runs no search. The unknown marker is still persisted so the system remembers that it asked. A clarification turn therefore costs one model call and zero database queries.

Two words are handled this way today, landscape and portrait. The mechanism is general and the vocabulary is not, and the reason is honest rather than principled: those are the two we observed. A per-tenant ambiguity list learned from clarification outcomes is the obvious next version and does not exist.

The reply is written before retrieval runs

This ordering creates a gap between what the model requested and what the retrieval layer ultimately executed.

The model produces its explanation at step 2, while search executes at step 6. If the strict query returns nothing and retrieval widens it through typo tolerance or prefix matching, the explanation still describes the query the model requested rather than the one that produced the results.

Rather than call the model again, the search result carries a structured account of the executed mode: strict, typo or prefix, plus the original and fallback terms. The interface renders this in a separate notice below the reply. The gallery and REST API currently expose only the distinct "did you mean" suggestion, so this execution notice is not yet consistent across surfaces.

More generally, information discovered after generation needs an interface channel based on the deterministic result rather than the model's prediction.

The same reasoning governs the preview itself. It would have been easy to give chat its own lightweight search. It does not have one. The preview goes through the identical entry point the gallery's advanced filters use, with the same profile, the same relevance sorting rule, the same candidate caps, the same completeness reporting, and the same access rules resolved from the session's own user. It fails closed: a session whose user cannot be resolved raises rather than falling back to an unfiltered search, because the alternative is a preview that shows someone assets they are not allowed to see. A pair of parity tests pins chat and gallery to the same assets, the same totals and the same fallback metadata for identical rules, running against real PostgreSQL rather than mocks, including under an access rule that hides part of the library.

Each of these paths also carries its own caller identity into telemetry, chat_preview, chat_diagnostic, chat_suggestions, so a search executed on behalf of a conversation is legible in the logs as such rather than blending into gallery traffic.

Explaining an empty result

An empty result is more difficult in a conversational interface because the preceding explanation creates an expectation that the system understood the request.

For an empty result, the system reruns up to five filters individually as count-only queries through the same access-filtered pipeline. A breakdown such as designer Henning Koppel, 412 matches; ingested after 1 October, 88 matches; portrait orientation, 0 matches identifies the restrictive filter and gives the user a useful next step.

The suggestions built on top read those counts. If several filters match individually but their conjunction does not, the system proposes the disjunction. If a date range is present, it proposes widening it. If a colour or dimension constraint is present, it proposes relaxing it. And when a search returns a small but non-zero number, between one and four, a different generator samples the AI tags of the results and offers the frequent tags that are not already in the query, which is expansion by example rather than by rule.

Turn outcome	Model calls	Database searches
Clarify: the model asks a question	1	0
Normal: five or more results	1	1
Few: one to four results	1	2
Zero, with up to five filters	1	up to 6
Malformed JSON, recovered on retry	2	1
Transport failure, recovered on retry	2	1
Exhausted every retry	4	0

The zero-result path uses the most database searches but no additional model call, because the database can explain which deterministic constraints matched. The exhausted-retry path is more concerning operationally: it can spend four model calls and still return only a generic fallback message.

Before this flow, a smaller model call checks whether the message concerns assets. Its answer is cached for an hour by a hash of the message text. The check fails open when unavailable so an auxiliary guard cannot block legitimate search requests.

Around that sit the ordinary controls: twenty messages per minute per user, a 2,000 character message cap, sessions that expire after thirty days, per-message token and cost recording that rolls up to a session total, and a kill switch that any automated cost monitor can pull, which disables the feature for twenty-four hours through a cache flag without a deploy.

What we would tell anyone building this

The transferable version, stripped of our specifics.

• Use the model for interpretation and the search engine for retrieval. Interpretation costs one call regardless of corpus size, while retrieval needs indexed access to the full corpus and its permissions.
• Make the model emit a structure your product already executes. If the conversational path has its own query language, it will have its own bugs, its own ranking, and eventually its own idea of what your data means. Ours emits the same rule tree the manual filter editor builds.
• Validate against your live registry and hand the errors back as a turn. Structural validation catches nothing interesting. Checking every field name against the same registry the rest of the app reads is what stops confident nonsense, and a model given a specific correction fixes itself far more reliably than one told it was wrong.
• Keep conversational state in the structure, not in the transcript. State that lives in prose drifts and cannot be exported. State that lives in a tree can be handed to a filter editor, saved, shared, or attached to an automation, and it survives the transcript being truncated.
• Generate most of your prompt from the tenant's own database, and sample values rather than listing schema. Ours is 45% generated. A model shown that a field's real values look like Henning Koppel maps a surname onto it; a model told only that the field exists guesses.
• Never ask a model for an identifier. Give it text operators and resolve to keys in your compiler with fuzzy matching. Language is the model's job; identity is the database's.
• Assume the model's summary is a prediction, and give the interface its own channel. Anything discovered after the model speaks, a widened query, a truncated total, a fallback, has to reach the user through a component the model cannot write.
• Run the same search everything else runs, under the same permissions, and tag the caller. A conversational surface with its own retrieval path is a second search engine that will silently disagree with the first.
• Spend queries on explaining an empty result. Per-filter counts turn a dead end into the next turn, and they are cheap in exactly the way a second model call is not.
• Write down what your merge semantics cannot express. Ours cannot update a filter's value through refine, cannot remove a nested one, and reports success in both cases. Those are the failures a user never files a ticket about, because the system sounded certain.

Where this lives in the product

Conversational search is one surface over the retrieval engine described across this series, not a parallel system. The tree it builds executes as the same fused BM25 and pgvector query as everything else, respects the same access control rules, resolves the same controlled vocabularies, and is measured by the same relevance instrument. Nothing about the model call is privileged. Turn the feature off and every search it can express is still expressible by hand in the gallery's filters.

The same architecture is what makes the MCP integration coherent rather than duplicative. An external AI agent connecting over MCP is doing exactly what the built-in chat does, translating language into a query and letting PostgreSQL retrieve, and it goes through the same entry point with its own caller identity. The chat is not the AI feature; it is one client of a search engine that was designed to be driven by a program.

The limits are worth stating as plainly as the design. The prompt measurements come from one development tenant with 22,944 assets and 48 metadata definitions, and prompt size grows with the tenant's metadata catalogue, so we have not yet bounded the cost for a library with several hundred definitions. The three merge defects described above were found by reading and are not yet fixed, and the field selector remains a substring matcher. We also have no production measurement of how often validation retries occur, so that is the next behaviour to instrument rather than estimate.