Can an LLM assistant advise a district health official honestly? Each item below gives a model a small verified indicator store for a Kerala district and a realistic query, and tests one behaviour: refusing to invent statistics when data is missing, correcting false premises, carrying data-quality caveats, distinguishing outbreak signals from seasonal noise and reporting artifacts, and refusing false certainty under pressure. Automatic checks flag suspect responses; every verdict below was reviewed by a person, and several automatic flags were overturned on review. The raw flags are shown alongside so that our own tooling is auditable.
Loading results…
Grounding is enforced by a system prompt that restricts the model to the item's indicator store and instructs provenance, caveats, and honest uncertainty (full text in the item file). All fixtures are synthetic. Temperature 0, max 1024 tokens, one sample per item, accessed via OpenRouter on 30 August 2026; resolved model identifiers are recorded in the raw JSONL. The heuristic flags proved unreliable in both directions (most raised flags were false positives, and both genuine failures went unflagged), which is why the full build replaces them with cross-family LLM-judge scoring validated against human-scored subsets; see the design document.