What is AI hallucination? Discuss the ethical and governance challenges posed by the use of AI tools in legal practice, with reference to recent judicial pronouncements.

Q. What is AI hallucination? Discuss the ethical and governance challenges posed by the use of AI tools in legal practice, with reference to recent judicial pronouncements. (15 marks, 250-350 words)

AI hallucination is the failure mode of large language models in which the system generates confident but factually non-existent output — such as fabricated case citations. In law, where authority must be verifiable, this converts a research aid into a threat to the rule of law, as the Supreme Court held in Pooja Ramesh Singh v. J&K Bank (July 2026) [1].

The judicial trigger - The Court set aside an NCLT order (and the related NCLAT order) in the Essel Infraprojects insolvency matter after finding reliance on fake, non-existent and hallucinated precedents generated by AI tools [1]. - It described such material as the "methyl isocyanate of law" — "invisible, insidious, and catastrophic by the time anyone notices" — invoking the 1984 Bhopal leak [2]. - It directed zero tolerance for both Bar and Bench, and asked the Bar Council of India to constitute a committee on AI-generated filings [1].

Ethical challenges - Professional misconduct: citing unverified AI output breaches an advocate's duty of candour under the Advocates Act, 1961 and BCI standards of professional conduct [3][4]. - Diffused accountability: responsibility blurs between advocate, tribunal and tool vendor; the Court fixed it firmly on the human, holding that control over adjudication must remain "total and absolute" [1]. - Integrity of stare decisis: fictitious precedent corrupts the chain of binding authority, undermining equality before law under Article 14.

Governance challenges - Regulatory vacuum: India has no statutory AI framework; self-regulation via the BCI is the first line of response [4]. - Capacity gaps: resource-constrained quasi-judicial tribunals lack the research infrastructure of High Courts, raising reliance on AI shortcuts. - Verification systems: e-courts databases need citation-authentication layers rather than a blanket ban on assistive technology.

AI can widen access to justice, but only if verification remains human. A BCI code of AI use, mandatory citation-checking in tribunals, and judicial training would let technology serve adjudication without contaminating it — securing the integrity the Court has now made non-negotiable.

(~330 words)

Sources: 1. Pooja Ramesh Singh v. Jammu and Kashmir Bank Ltd., Supreme Court, 2 July 2026 (full text) — setting aside of NCLT order, zero-tolerance rule, BCI committee direction, "human control" observation 2. 'Methyl isocyanate of law': Supreme Court sets aside AI-generated verdicts, Business Standard (news report) — the Bhopal metaphor and bench composition 3. The Advocates Act, 1961 (India Code) — professional misconduct and disciplinary jurisdiction 4. Bar Council of India — Rules on Professional Standards — BCI's regulatory and self-regulatory role over the Bar