Chatbots in government services promise triage at scale: answer routine questions, route cases, spare staff for complex work. Where they are scoped as navigators — which office to call, which form number, how to book — they perform acceptably, and several national digital services have run such assistants for years. Where they are scoped as advisers — telling users what they qualify for, what a deadline is, whether a right has expired — the failure modes are specific and documented. The U.S. Government Accountability Office's technology-oversight work, federal AI risk-management guidance based on the National Institute of Standards and Technology's AI Risk Management Framework, and the binding operational guidance the Office of Management and Budget issued under executive orders on federal AI use all converge on the same control list; the failures below are the list running in reverse.
This article publishes information about government technology, not advice on any individual claim or benefit.
The failure modes, in the order they bite
First, confident wrong answers on consequential facts. A generative assistant trained loosely on agency content will state an incorrect income threshold or processing time with full fluency, because fluency is what the model produces and verification is what the deployment must add. Second, hallucinated process steps: forms that do not exist, offices that moved, appeal paths that are not real — errors of navigation that strand exactly the users with the least slack to recover. Third, escalation starvation: systems designed to maximize containment — the share of sessions the bot resolves without a human — quietly raise the wall between the user and the person who could actually help. Fourth, unequal performance: speech interfaces and English-dominant training degrade service quality for limited-English-proficiency users, whom public agencies are legally obligated to serve, a tension civil-rights reviews of automated systems have repeatedly flagged. Fifth, drift without monitoring: content changes — a new rule, a moved deadline — and the assistant keeps answering from the old corpus until someone audits it.
What the control list requires
The OMB guidance for federal AI use, alongside the NIST framework, translates into four deployable tests for any public chatbot. Inventory and risk-classify: uses that affect benefits, rights, or safety are high-impact and carry heavier duties. Ground the answers: responses must trace to a versioned, authoritative corpus — the statute, the regulation, the published procedure — with retrieval that cites its source, so a wrong answer is findable. Monitor in production: sample transcripts against ground truth on a schedule, track wrong-answer rates by question type and language, and log escalation outcomes rather than containment rates alone. And keep a real exit: a human channel whose availability does not depend on the bot failing first, with the bot required to offer it whenever the query touches rights, deadlines, or appeals.
The procurement tells
You can read a deployment's risk posture from its contract before it launches. Danger signs: success measured in containment rate alone; no requirement to cite sources or version the knowledge corpus; no independent accuracy sampling; no language-access testing tied to legal obligations; and vendor liability disclaimers for wrong answers in a service the government itself is legally answerable for. The healthy inversions are equally legible: published accuracy audits, a stated list of questions the assistant is not allowed to answer, and an escalation metric the agency reports on the same dashboard as its cost savings.
FAQ
Are government chatbots allowed to advise on benefits?
High-impact uses touching rights and benefits carry the heaviest duties under OMB guidance: grounded answers, monitoring, human escalation — and many agencies scope bots to navigation instead of advice.
What is containment rate and why is it a bad target?
The share of sessions the bot resolves without a human; optimizing it alone rewards keeping users away from staff who could fix consequential errors.
How should a public chatbot be monitored?
Scheduled sampling of transcripts against authoritative sources, wrong-answer rates by question type and language, and audit trails from each answer to the cited document version.
For more context, read FedRAMP: How a Cloud Service Gets Federal Authorization.
For more context, read open meetings law.
For more context, read government payment delivery.