Retrieval without the vectors

The problem

An engineer, deep into an issue, suddenly needs a test user with just the right mix of attributes: a certain coverage type, an active prior authorization, and a benefit that has already been terminated. Like everyone else, they first dig through Confluence, hoping to unearth a documented match.

They find one but discover it is stale. The account's data or the feature it was meant to test has changed. So they dig again and check the next candidate.

If nothing usable surfaces, the next move is to ask around and ping teammates who might recall a suitable user. When that comes up empty, the last resort is filing a request with the data team—a process that can stall feature work for up to two weeks.

That ladder (search, repeat, ask around, file a formal request) repeats constantly, and not only for QA. A software engineer reproducing a bug needs a user that triggers it. A QA engineer writing test cases needs several, one per branch of the scenario. A product owner checking that what shipped is what they asked for needs one too. They work across many mobile and web apps, and every test scenario needs a user whose data matches a specific combination of clinical and enrollment attributes: coverage type, plan status, feature flags, network membership, claims history. The requirements are combinatorial because changing one attribute means the user who matched a minute ago no longer does.

Before Quade, that ladder was the entire system: Confluence pages that became outdated within a few months, colleagues who happened to remember where to look, and a data team blocking engineers with a two-week wait. The initial corpus had a few hundred test users with poorly indexed attributes.

The idea: give the model a SQL tool

The premise was simple: load test user data into a relational database, equip the model with a SQL tool, and let it answer natural-language questions directly.

The case for an agent over a plain search UI came down to the nature of the requests. The queries are multi-step and ambiguous. "A user like the one I used last sprint, but without the dental plan" is not a filter form but a conversation. A chat interface can ask clarifying questions, iterate on empty queries, and explain what it found, rather than just listing rows.

This was always a proof of concept, and that shaped the architecture as much as the idea itself. The goal: prove the core loop—natural language in, real answer out—with as few moving parts as possible. Building production infrastructure could wait until the idea proved itself.

The architecture was optimized for a hybrid React and Express service, with Claude running through AWS Bedrock as the reasoning layer and PGlite, an in-process, Postgres-compatible embedded database, as the data store. There was no separate database server to provision or manage. It was light enough to run entirely on a laptop or deploy to a cloud container without adding extra dependencies.

Letting the model write the schema

Before any of that could work, the data had to be in the database, and getting it there was its own problem. Test user attributes lived across about ten service endpoints per person, and no two of them agreed on much. One returned a flat object. Another wrapped the useful part four levels down in an envelope of metadata. A third returned an array of records where only the most recent one mattered, and did not sort them.

Historically this is the expensive part of a project like this, and it is expensive in a way that does not look like difficulty. Someone reads each response by hand, decides which fields are worth keeping, writes a table definition to hold them, then writes the script that maps one onto the other. None of it is hard. All of it is slow, and it has to be redone every time an upstream team changes a payload.

That work turned out to be something the model could do. Given a sample response and a description of what mattered, it would propose a schema, and then the script to flatten a response into it. The output needed review and sometimes correction, but reviewing a generated schema is a different order of task from writing one: minutes per source instead of an afternoon.

The hours saved are the smaller half of it. Once generating a schema is cheap, the calculation about which data is worth absorbing changes. A source that could never have justified two days of mapping work justifies twenty minutes, so it gets absorbed instead of skipped. Structured sources, semi-structured ones and free text all become tractable the same way, because the model reads whatever shape the data arrives in and proposes the columns that would hold it. The messiest input we took on later, a pile of free-text request descriptions, went through the same move and came out as the same structured attributes.

This is also the reason there is no vector store anywhere in the system.

Embeddings are the reflexive answer to "make this searchable", and they would have been the wrong answer twice over. The first problem is that similarity search has no notion of exact. A plan code, a member identifier, a feature flag name: these are tokens, and they are either right or wrong. Two plan codes differing by a single digit sit almost on top of each other in embedding space and describe entirely different products. Keyword matching is not a nice-to-have on this data, it is most of what the data is.

The second problem is worse, because it is quiet. When an embedding model considers two things closely related, it returns the near miss with exactly the confidence it would return a match. Nothing in the result distinguishes them. The requester gets an account that looks right, uses it, and finds out later: in a test that passes when it should have failed, or one that fails for a reason that has nothing to do with the code under test. That is the failure mode you least want in a tool whose entire job is to be trusted about which user matches.

The usual answer to the first problem is to stop choosing between them. Run a vector search and a keyword search side by side, and you get semantic reach where it helps and exact matching where it is required. That does work. It is also where the complexity starts compounding, and the compounding is the part worth looking at before committing to it.

Two retrieval systems return two result sets that do not agree, scored on scales with no relationship to each other. Something has to merge them, which means a fusion step and a set of weights expressing how much to trust each side. Then something has to rank what comes out, and ranking test users is not one dimension. Relevance to the query is an axis, but so is hierarchy among the attributes themselves: a mismatched coverage type disqualifies a candidate outright, while a different claims history is usually irrelevant to whether the account is usable. Encoding which attributes are load-bearing and which are decoration is a design problem in its own right, and the weights that express it need tuning, then retuning as the corpus grows and as new attributes arrive from new sources.

A query does not have this problem. A WHERE clause over the attributes someone asked for either returns rows or does not, and an empty result is unambiguous. It says the user you want does not exist in this corpus, which is a true and useful answer. The reason the model can be given a SQL tool and trusted with it is that SQL fails loudly, and similarity fails plausibly.

Before there was a reputation to defend

The first two weeks were unglamorous: an Express backend, a React UI shell, a codebase converted to TypeScript, and by the end of week two a chat endpoint wired to Bedrock with PGlite seeded from a snapshot on startup. It worked, which is not the same as being right. An agent that answers confidently is harder to catch out than one that visibly fails, and the failures that matter here look exactly like successes.

So the next thing built was a way to be told it was wrong. A tiny "Something went wrong?" link under each response opened a popup to a Teams channel where anyone could describe what had happened. The team wanted that signal from engineers using the tool daily rather than from internal test runs, and wanted it while the tool was still too new for anyone to have formed a view of it.

A few weeks later the human signal was joined by an automated one. A dynamic evaluator ran as a second model pass, grading the agent's answers against a set of known-good use cases. Baseline accuracy was helpful but not solid enough to stake a regression cycle on: the agent handled common query shapes reliably, a coverage type here, a plan status there, and degraded on rare attribute combinations and edge cases nobody had thought to test directly.

The score by itself was never the point. The evaluator tested the full stack through the same endpoint a real user hits, so it caught regressions from prompt changes, data gaps and model upgrades alike rather than just changes to its own narrow slice. When the team upgraded the underlying model, the suite ran before any real traffic switched over, which turned the upgrade into a deliberate decision instead of a bet. It sits off to the side of the live path, grading finished answers rather than standing between a question and its response.

Growing pains

Production has its own lessons, and most of them arrive without warning.

Memory limits were first encountered inside the container environment. PGlite loads the whole database into memory, and about 200 MB of data pushed straight into the container's ceiling. A few rounds of provisioning adjustments addressed this, but it was a clear early signal that the architecture's simplicity came with trade-offs.

Streaming shipped next, because responses were slow enough that a non-streaming UI felt broken. The user would watch an animated loading bubble that was as uninformative as a blank screen while an answer assembled elsewhere. Shipping it also revealed a second problem: a proxy in front of the service was silently buffering the entire stream before releasing it, defeating the point. The fix was a single header, X-Accel-Buffering: no, telling the proxy to pass each chunk straight through rather than hold it.

Streaming also changed what the UI could show while an answer was still forming. The agent's reasoning could stream like the final answer: each block of reasoning would appear with a pulsing placeholder, then be replaced by the next block as the agent moved on. A user could watch the reasoning unfold in real time instead of staring at a blank screen while waiting. Once the answer landed, that reasoning collapsed into something they could expand later to check how the agent arrived at it.

Timeout tuning came last. Agent calls chaining multiple SQL queries plus a model round-trip would occasionally run past a default sixty-second timeout. The team wanted to support genuinely long queries rather than just tolerate rare slow ones, so the window was deliberately extended to two minutes: long enough for slow queries, short enough that a stuck request wouldn't hang forever.

The data pipeline catches up

PGlite was fast, but the answers were only as good as what had been seeded into it. A separate harvester made those calls, pulling account details, activity history, entitlements, relationships, feature flags and more, one source at a time, per person.

A later change wired that harvester directly into the agent's database and evaluation system, so the full pipeline ran end-to-end: harvester, into the database, into the agent, out through the UI. A nightly automated job kept the data fresh without anyone triggering it. Getting the authentication right for that automation took several passes because credential management is more difficult inside a heavily permissioned enterprise environment.

At this point, the corpus had several hundred users, and a full harvest run took well over an hour.

Bringing in a bigger source

There was already a system for this: an internal tool built to manage the intake of user data requests. A team would fulfill each request by hand and log the resulting user back into the system, which meant it was quietly accumulating a real, growing list of users as a byproduct of its actual job. The team wanted to link that list directly to Quade, not just to increase the number of available users, but to expand the range of scenarios Quade could cover in the first place.

A new harvest module queried that system's API directly, used the model to summarize each user's profile from the most recent request made against it (turning free-text descriptions into structured attributes), and fed the results into the main pipeline.

The much larger number of users this brought in drove harvest time up sharply. Running everything sequentially, as the original pipeline had, was no longer viable: a full harvest needed to finish within a reasonable window, not grow linearly with each new user added. The harvest was parallelized around a bounded worker pool, so two steps that had been running one after another (the model doing the summarizing and the per-user API fetches) now ran concurrently.

The result was a harvest that dropped from well over an hour to roughly fifteen to twenty minutes, and a user corpus that grew well past where it had started, climbing further as more teams enrolled in the system feeding it.

That growth also outgrew the original feedback mechanism. A pop-up pointing to a Teams channel worked for a small pilot group; it did not scale to that audience. The "Something went wrong?" link was replaced with a thumbs-up and thumbs-down under each response, with an optional note; lower friction for the person leaving it and something structured enough to act on.

Graduating to a managed database

PGlite had been exactly right for a proof of concept with few dependencies. However, it became a constraint for different reasons as the service matured. Due to our method of deployment, PGlite did not scale well; a single file cannot be shared across multiple running instances, which limited horizontal scaling in production. As usage and data volume grew, we were forced to increase memory limits, highlighting that we would eventually need a slightly more sophisticated solution to support ongoing database growth.

The deployed database moved to Amazon RDS, a managed Postgres service. PGlite stayed for local development since both speak the same SQL dialect and the application code never had to change to support either.

The data load was redesigned around atomic schema swaps: a harvest loads into a fresh, timestamped schema, and only once complete does it get promoted to live in a single rename. A failed or partial load never touches the schema anything is reading from.

Keeping the old schemas around turned out to be worth more than the safety it was built for. When a harvest fails for a particular user, the previous schema still holds a good copy of them, so the request gets answered from slightly stale data rather than not answered at all. And because every load is preserved under its own timestamp, two of them can be compared directly. If a user's attributes changed shape, or went missing entirely, the difference between last night's schema and the one before it says so, which is a much shorter path to an explanation than reconstructing the change from the source systems.

More than a user finder

Once the tool existed, use cases showed up that had nothing to do with the original brief.

• Environment health checks. A question like "is this environment down?" now checks login viability across a sample of accounts and reports back, rather than someone finding out the hard way.
• Data investigation. Natural-language interrogation of the harvested data, replacing much of the ad hoc SQL people had been writing by hand.
• Combinatorial enrollment queries. The kind that require joins across several tables became conversational, rather than something only a few people could write correctly.
• Login validation. Its own dedicated tool: a check of whether a specific account can authenticate against an environment, and what it can reach once it does.

The chat interface stayed the single entry point for all of it, so nobody had to learn a different tool for each.

Lessons and what's next

Two decisions paid off far beyond what they cost to make. Building the evaluation system early meant it kept paying dividends every time the model or the prompt changed, rather than finding out about a regression from a user. And the atomic schema-swap load kept paying out in ways it was not designed for. Rollback is one more rename rather than a restore, because the schema that was live a moment ago is still sitting there under its timestamp. But the retained schemas turned out to be a record as much as a safety net: a user missing from last night's harvest can still be answered from the night before, and a question about what changed upstream is a comparison between two loads rather than an investigation.

Across the organization, the tool meaningfully reduced the time people spent digging through Confluence, asking around, and waiting on a data team's two-week queue. That saving repeats every quarter.

The bigger piece of unfinished work is what happens when nothing matches at all. Right now, a search that comes up empty is still a dead end. The plan is to let the agent generate a matching QA user on the fly instead: synthesize one that satisfies the requested attributes even if nothing like it exists yet in the harvested data, so a search that used to end in "try again later" ends in a usable account instead.

That plan only works because the empty result is trustworthy. An exact query returning nothing is not a failure to find something, it is a positive statement that no such user exists yet, and that statement is exactly the signal synthesis needs to fire on. A retrieval layer that answered with the closest thing it had would never produce that signal. It would hand back an account that looked close enough, nobody would know a user needed creating, and the test would run against someone who did not satisfy the requirements it was chosen for. The empty result is not the gap in the system. It is the thing that tells the system what to do next.