A validator that rejects any output without a source

1 September 2026Majed Alandajani

In a system that produces a price quote for a client, hallucination is a commercial risk. The quote can name a price that never existed or promise a date nobody can meet. It can even list a service that is not available.

What did not work

Better prompts came first. "Do not invent" and "stick to the sources" reduce the probability without eliminating it, and the probability that remains tends to surface at the worst possible moment.

Human review alone did not hold up either. A staff member approves twenty quotes a day, and somewhere in those twenty, something slips past.

What worked

A verification layer that runs after generation and before sending. Every item in the output must carry a reference to a row that actually exists: a product, a price, an availability rule. Any item arriving without one is rejected and dropped from the output.

Why this is different in kind

Verification asks the database, not the model. However convincing the generated text may be, an item with no row behind it never leaves the system.

The price you pay

Shorter outputs. Sometimes the itinerary comes out with four items instead of six, because two of them found no source. That is the correct behaviour, even though at first glance it looks like a deficiency.

The trap

The validator has to be boringly strict. My first version accepted a fuzzy match on the name, and an item got through whose name sat close to a real product without being that product. Matching now runs on the identifier, never the name; the name only appears in what the customer sees.