Kepler

Further reading Top-p sampling (Wikipedia) · Defeating Nondeterminism in LLM Inference (Thinking Machines Lab) · Hallucination (artificial intelligence) (Wikipedia)

1. i.

   Trained to guess

   Pre-training teaches a model to predict the next word, which rewards it for having a go even when it lacks the information. A student who never leaves an exam question blank scores better on average, and is sometimes confidently wrong.

2. ii.

   Filings are hard homework

   Financial documents are where this bites. A benchmark of questions about public company filings was meant as a minimum standard, with clear-cut, straightforward questions. A leading model with a retrieval system still answered wrongly or refused on 81% of them, and every model tested showed weaknesses such as hallucination.

3. iii.

   Same digits, different fact

   A number in a filing is mostly context: the currency, the scale in the table header, the date in the column. One LinkedIn quarterly report tags the same figure, $1,322,500,000, three times as three different facts. Search for the digits and you match whichever one you meant.

4. iv.

   No copy button

   Language models don't copy; they retype, and the retyping is usually right. At one slip in 1,000 figures, a 400-figure workbook carries a wrong number every two or three runs, and nothing says which. Better models fail more quietly, with the right digits from the wrong period or unit.

Further reading Hallucination (artificial intelligence) (Wikipedia) · FinanceBench: A New Benchmark for Financial Question Answering (arXiv) · Better language models won't fix hallucination. They'll make it quieter. (Kepler)

Further reading How Kepler built verifiable AI for financial services with Claude (Anthropic) · Better language models won't fix hallucination. They'll make it quieter. (Kepler) · AI that proves it's right (Kepler)

1. Step 1: Language here, numbers there

   The AI interprets what you asked and breaks it into an execution plan. It never retrieves data or computes numbers itself. Deterministic code does that, and the two layers talk through structured contracts, never raw model output.

2. Step 2: One word, one number

   Underneath sits an ontologyA structured map of the concepts in a field, their names and how they relate to each other., a map of financial concepts and how they relate, so "revenue", "top line" and "total sales" all resolve to the same verified number. Common workflows, such as enterprise value across a messy capital structure, are built as skills designed so the same input always gives the same output.

3. Step 3: Name it, don't type it

   Every tagged figure in a filing becomes a fact with an ID. The model doesn't write the digits; it names the fact, and the system fills in the filed value, linked to its line. An unreferenced number is treated like a compile error. It's the same fix software found for SQL injectionAn attack where text typed into a form is run as a database command because data and code travel in one string.: keep the data out of the command string.

4. Step 4: Choices written down

   Some questions have several true answers, like whether debt counts at principal or at its balance-sheet value. Kepler makes that choice a named setting in the output, so the next reader can see it and flip it.

Further reading AI that proves it's right (Kepler) · How Kepler built verifiable AI for financial services with Claude (Anthropic) · Better language models won't fix hallucination. They'll make it quieter. (Kepler)

• Can inference itself be made repeatable?

  Yes, with work. Rewriting the core GPU routines so results don't depend on batch size made all 1,000 completions identical, and a first, unoptimised version ran slower, though not disastrously. Identical is still not the same as correct.

• Will better models make the problem go away?

  Hallucination rates are falling, but not as fast as they look. In one release each claim became 23% more likely to be right, yet only 3% fewer responses contained an error, because the model was making more claims per response.

• Does a citation mean the model used the source?

  Not necessarily. Planting a few words of a model's answer into a document it hadn't cited moved a production retrieval model's citation there up to 57% of the time, with the answer unchanged.

Further reading Defeating Nondeterminism in LLM Inference (Thinking Machines Lab) · Better language models won't fix hallucination. They'll make it quieter. (Kepler)

token

A chunk of text, often a word or part of one, that a language model reads and writes one at a time.

greedy search

Always picking the single most likely next token, with no randomness.

top-p sampling

Choosing the next token at random from only the most likely candidates, whose combined probability just passes a set threshold.

temperature

A setting that controls how much randomness goes into picking tokens; at zero the model should always take the likeliest.

hallucination

AI output that states something false as though it were fact.

ontology

A structured map of the concepts in a field, their names and how they relate to each other.

SQL injection

An attack where text typed into a form is run as a database command because data and code travel in one string.

1. 1Top-p sampling · Wikipedia
2. 2Defeating Nondeterminism in LLM Inference · Thinking Machines Lab
3. 3Hallucination (artificial intelligence) · Wikipedia
4. 4FinanceBench: A New Benchmark for Financial Question Answering · arXiv
5. 5AI that proves it's right · Kepler
6. 6Better language models won't fix hallucination. They'll make it quieter. · Kepler
7. 7How Kepler built verifiable AI for financial services with Claude · Anthropic