Stopping the LLM from inventing numbers
- Problem
- An LLM summarising market data can make numbers up, and they look identical to real ones.
- Fix
- The agent must finish with a submit_report tool call where every claim cites a tool call. A plain-code verifier checks each number against the real tool result (prices rounded to the pair's decimals, USD exact); one retry, then “could not verify”.
- Verified
- A 10-question live eval with independently computed answers scored 8/10, 9/10 and 9/10, with no per-question tuning.