AI That Says "I Don't Know"
What happens when AI flags its own uncertainty.
The Architecture Series · 7 minute read · By Ed | Founder & CEO, DealiOS
The most dangerous AI system isn't the one that's wrong. It's the one that's wrong and confident.
A 2025 Carnegie Mellon study found that large language models remain stubbornly overconfident even after producing incorrect answers. Humans, when shown they performed worse than expected, adjust their confidence downward. The AI doesn't. It keeps insisting it did well. The researchers compared it to a friend who swears they're great at pool but never makes a shot.[1]
In enterprise contexts, this confidence inversion has consequences. A 2024 Deloitte survey found that 38% of business executives reported making incorrect decisions based on hallucinated AI outputs.[2] Not "received bad data." Made actual decisions. Allocated budgets. Changed strategies. Signed contracts. Based on information the AI presented with full confidence and zero accuracy.
The architectural challenge isn't making the AI right more often. It's building systems where uncertainty is surfaced, not suppressed. Where "I don't know" is a useful output, not a failure state.
The Confidence Inversion
The less the AI knows, the more confident it sounds. This is the opposite of human expertise, and it's the most dangerous property of current language models.
A first-year legal associate says "this is definitely enforceable." A 30-year partner says "this is likely enforceable under the following conditions, depending on jurisdiction and the specific facts." The associate's confidence comes from ignorance of complexity. The partner's caution comes from decades of watching "definitely" turn into "it depends."
The AI talks like the first-year associate. That's not a feature. That's a design flaw.
The less the AI knows, the more confident it sounds. This is the opposite of human expertise, and it's the most dangerous property of current language models.
The AI has the confidence of a guy who's been to one wine tasting. Everything is "oaky" and "full-bodied." An actual sommelier would say "this has interesting characteristics but I'd want to taste it with food before making a judgment." Uncertainty isn't weakness. It's the thing that separates knowledge from performance.
Research on LLM calibration consistently shows that models exhibit minimal variation in confidence between correct and incorrect answers.[3] When the model is right, it's confident. When the model is wrong, it's equally confident. The confidence signal carries almost no information about accuracy. For a user trying to decide whether to trust the output, that's worse than useless. It's actively misleading.
In legal contexts specifically, a 2024 study found that LLMs hallucinated in 69% to 88% of legal queries.[4] Not fringe questions. Standard legal research tasks. The model confidently cited cases that didn't exist, quoted statutes that said something different, and applied precedents from the wrong jurisdiction. All presented with the same linguistic certainty as the answers it got right.
Three Uncertainty Mechanisms
Epistemic humility in AI isn't a prompt instruction. You can tell the model to "be honest about uncertainty" and it will write the words "I'm not certain about this" while still generating fabricated content with perfect grammatical confidence in the next paragraph. Instructions don't change architecture. Mechanisms do.
Instructions don't change architecture. Mechanisms do.
DealiOS implements three:
Mechanism 1: [TO BE CONFIRMED] placeholders. When the AI lacks specific information required for a contract (governing law, liability cap, insurance requirement, data processing terms), it doesn't guess. It inserts a literal [TO BE CONFIRMED] marker and flags the gap in the document's metadata.
This marker isn't a suggestion. It's a visual alarm. The DOCX export renders every [TO BE CONFIRMED] in amber highlight, impossible to miss during review. The published page renders them prominently, so a client receiving the document sees exactly where the gaps are. The mechanism isn't a hard block that prevents sending. It's something better: it makes the gaps impossible to ignore. A human can still send a contract with unresolved markers if the situation calls for it (a preliminary draft for discussion, a framework agreement where terms will be negotiated). But they can't do it accidentally. They can't miss the amber flags and think the document is complete.
This is the architectural equivalent of a compiler warning versus a compiler error. An error stops you. A warning makes sure you see the problem and forces a conscious decision. [TO BE CONFIRMED] forces every person who touches that document to confront the gap. The uncertainty is visible at every layer, in the editor, in the export, in the published page, until a human resolves it or explicitly decides to proceed anyway.
Mechanism 2: Conservative estimation rules. When the ROI framework computes projections, it applies a systematic conservatism bias. If industry benchmarks suggest 40% improvement, the system models at 25-30%. If comparable deployments show 6-month payback, the system projects 8-10 months.
This isn't sandbagging. It's calibration. The system defaults to understating rather than risking overstatement. The buyer who exceeds the projection becomes an advocate. The buyer who falls short of an aggressive projection becomes a detractor. Conservative estimation is an architectural decision that prioritizes long-term trust over short-term impressiveness.
Mechanism 3: Source labeling. When the AI references data, it must distinguish between buyer-specific data ("Your team's current close rate of 23%") and industry benchmarks ("Industry benchmarks suggest an average close rate of 28%"). These are not interchangeable claims, and the system doesn't let them look interchangeable.
"Your data shows X" is a statement of fact. "Industry benchmarks suggest X" is a reference point. When buyer-specific data is unavailable, the AI uses industry averages but must label them as such. The label forces the reader to evaluate the claim on the right terms. An unlabeled "you'll see 28% improvement" is a promise. A labeled "industry benchmarks suggest 28% improvement in comparable deployments" is information. One creates liability. The other creates informed decision-making.
The Retrieve-Before-Generate Rule
The primary defense against confident fabrication is a simple architectural rule: check the knowledge base before generating.
RAG (retrieval-augmented generation) has become a standard pattern, but most implementations treat it as a quality improvement. In DealiOS, it's a honesty requirement. The system retrieves relevant documents from the workspace knowledge base before generating any content. If the knowledge base contains the answer, the system cites it. If it doesn't, the system tells the user.
The alternative is what most AI systems do: generate first, hope it's right. That's the pattern that produces confident hallucinations. The AI doesn't know whether it's retrieving a real fact or generating a plausible-sounding one. It treats both with equal confidence because it can't distinguish between memory and invention.
The "check first, flag gaps" pattern doesn't eliminate errors. But it changes the error mode. Instead of confidently wrong output that the user can't distinguish from correct output, you get explicitly uncertain output that the user knows to verify. The error is visible. Visible errors are manageable. Invisible errors are the ones that cost 38% of executives their judgment.
The Deloitte Number That Should Scare You
Let me sit with that Deloitte finding for a moment: 47% of enterprise AI users made at least one major business decision based on hallucinated content.[2]
Not a minor decision. A major one. Nearly half of enterprise AI users. And these are the ones who admitted it. The actual number is certainly higher because hallucination detection requires knowing the right answer, and if you knew the right answer, you wouldn't need the AI.
This is the confidence inversion at organizational scale. The AI was confident. The executive trusted the confidence. The decision was wrong. And nobody knew until the consequences arrived.
The architectural response isn't better models (though better models help). It's systems that make uncertainty visible at every layer:
The document layer shows [TO BE CONFIRMED] markers where the AI couldn't verify information. The number layer shows conservative estimates with labeled methodology. The source layer distinguishes buyer data from industry benchmarks from AI-generated analysis. The export layer renders uncertainty markers in amber highlight so they're impossible to miss in the final document.
Each layer is a checkpoint. Each checkpoint is an opportunity for a human to catch what the AI missed. Not because humans are always better. Because humans in a well-designed system are better than either humans or AI alone.
What Uncertainty Costs and What It Buys
There's a real cost to building systems that admit uncertainty. Every [TO BE CONFIRMED] marker is a document that takes longer to finalize. Every conservative estimate is a projection that looks less impressive in a side-by-side comparison with a competitor who inflates their numbers. Every source label is a sentence that adds length and complexity.
Sales teams push back on this. "The competitor's proposal says 45% improvement. Ours says 25-30%. We look weak." That's true in the moment. In the moment, confidence wins.
But enterprise sales isn't a moment. It's a 6-18 month process that extends into a multi-year relationship. The competitor's 45% becomes a broken promise at the first quarterly review. Your 25-30% becomes a exceeded expectation that earns a case study, a reference, and an expansion deal.
The compound math is unambiguous. A system that consistently delivers on conservative promises builds trust that compounds across every renewal, every expansion, every reference call. A system that consistently overpromises builds a churn rate that compounds just as reliably.
Epistemic humility isn't a technical constraint. It's a competitive advantage with a delayed payoff. The companies that survive long enough to collect that payoff are the ones that built the uncertainty architecture from day one.
The Null Type for AI
In programming, null exists because sometimes you don't have a value. Before null types were formalized, systems would substitute a default value (zero, empty string, "unknown") and the downstream code would silently process garbage. Bugs were invisible because the system never acknowledged the gap.
Modern type systems force explicit null handling. If a value might be null, the compiler makes you deal with it. You write the code path for "what happens when we don't have this data." The gap becomes visible in the code, visible in the review, visible in the test suite.
[TO BE CONFIRMED] is the null type for AI-generated business documents. It forces explicit handling at every layer. The document renderer shows it. The export pipeline highlights it in amber. The published page displays it prominently to the client. The gap becomes visible because the architecture refuses to let it be invisible.
Every system that generates consequential output needs a null type. A way to say "I don't know" that propagates through the system with the same urgency as a correct answer. Not as an error. Not as a failure. As a data type. As real as any number, any clause, any recommendation.
The most dangerous AI system isn't the one that's wrong. It's the one that has no way to tell you it might be.
The takeaway: Uncertainty should be a first-class data type in every AI system: surfaced, propagated, and made impossible to miss at every layer, because invisible errors are the ones that cost executives their judgment.
References
1. Cash, T. & Oppenheimer, D. "AI Chatbots Remain Overconfident Even When They're Wrong." Memory & Cognition, 2025. Carnegie Mellon University. Study comparing LLMs and humans across trivia, predictions, and image identification tasks. LLMs failed to adjust confidence after poor performance; humans recalibrated. 2. Deloitte Global Survey, 2024-2025. Enterprise AI Decision-Making Risks. 38% of business executives reported making incorrect decisions based on hallucinated AI outputs. Separately, 47% of enterprise AI users made at least one major business decision based on potentially inaccurate AI-generated content. 3. Omar, M. et al. "Benchmarking the Confidence of Large Language Models in Answering Clinical Questions." JMIR Medical Informatics, 2025. Found LLMs show minimal variation in confidence between correct and incorrect answers across clinical question-answering tasks. 4. LLMs hallucinated in 69% to 88% of legal queries per a 2024 study cited by Carnegie Mellon's Department of Social and Decision Sciences (Cash & Oppenheimer, 2025) and corroborated by Stanford HAI legal AI evaluations.
This paper is part of The Architecture Series by Ed, Founder & CEO of DealiOS. For the weekly series and more technical deep-dives, visit dealios.app/library.
See how these principles come together. Explore the platform →