AI and Accountability: The Difference Between Plausible and Proven

Artificial intelligence is rapidly becoming embedded in professional workflows, helping teams move faster, process more information, and make decisions with greater efficiency. But as adoption accelerates, so does an important question: how much trust should we place in AI-generated outputs?

The answer often comes down to understanding how those outputs are produced.

In a UK tax tribunal case, nine legal cases were submitted to support an argument in court. The citations looked legitimate, the formatting was correct and the legal reasoning sounded plausible.

But none of the cases actually existed.

The filings were generated by a probabilistic AI system that produced fictional legal precedents convincing enough to pass an initial review. The citations were the result of AI hallucinations, where a system generates information that appears credible but cannot be traced back to verified sources. As a result, opposing counsel and the court then spent time and resources attempting to locate decisions that had never been real in the first place.

This is not an isolated case. As AI moves into professional workflows, it is becoming increasingly common. 

Why Confident Outputs Can Still Be Wrong A useful way to think about probabilistic AI is autocomplete at massive scale. When your phone finishes a sentence, it predicts the most likely next word based on patterns it has seen before. Large AI models operate similarly, just with far more data and sophistication behind them. They are extraordinarily good at producing outputs that sound coherent, informed, and complete.

But sounding correct is not the same as being verified.

Probabilistic systems are designed to predict what is likely based on patterns in training data. They do not independently confirm whether a citation exists, whether a record is authentic, or whether a conclusion can later be reproduced and defended.

Most of the time, these systems are directionally useful. They accelerate drafting, summarization, and early-stage research remarkably well. The problem emerges when probability is mistaken for verification.

When Plausible Becomes Operational Risk In low-stakes environments, a confident but imperfect answer may not matter very much. In diligence, compliance, legal review, or investment decision-making, it is critical.

The fallout from the fabricated court citations extended far beyond a simple mistake. Time was wasted. Additional review became necessary. Credibility was damaged. What was intended to accelerate work ultimately created more operational friction downstream. This is the governance challenge organizations are increasingly running into with AI adoption.

The issue is not whether AI is useful, as it clearly is. The issue is understanding what kind of system is being relied upon, how outputs are produced, and whether those outputs can later be validated when questions arise.

Why Governance Matters Most professionals only see the final output of an AI system. They review the summary. They assess the finding. They make a decision. What often remains invisible is the process behind the result.

• Was the output generated from statistical probability?

• Was it tied back to verified records?

• Can the same reasoning be reproduced later?

• Is there structured validation or human oversight inside the workflow?

Those questions become significantly more important in environments where findings may later need to be revisited by regulators, investors, boards, or compliance teams. Without governance, automation can accelerate workflows while simultaneously increasing risk later on.

The Shift Happening Now More organizations are moving beyond AI systems built for fast outputs toward architectures designed to produce verified, reproducible results that can be traced to source and defended under scrutiny.

At Intelligo, that philosophy shapes how we think about diligence technology. Intelligo’s platform is built on proprietary deterministic AI for all data collection, entity resolution, and verification, the work where accuracy is non-negotiable. Whereas probabilistic models are used selectively for summarization and data extraction. They operate downstream of verified facts, never as the source of them.

The tribunal case serves as a reminder that AI hallucinations are not simply technical quirks, they are a byproduct of systems designed to predict what sounds right rather than verify what is true.

In high-stakes environments, confidence cannot depend on whether something merely sounds correct, it must be grounded in evidence that can be traced and validated.

Source: Lee, M. 2025, September 08. The rise and rise of fake cases. Counsel Magazine. Link.