‘Healthcare AI: hospitals lack the capacity to measure real-world patient outcomes’
FlashcardsQuick reviseKey facts. One by one.Start →In this note
- At a Glance
- Why in the News
- Background & Evolution
- Core Static Facts
- Multi-Dimensional Analysis
- Recent Developments (last 12–18 months)
- Prelims Hooks
- Why a Licence Before Launch Cannot Catch This Kind of Harm
- India Has a Body to Measure Value, but Not the Data It Needs
- The Case for Using AI Now and Measuring Later, and Where It Fails
- What Each Actor Should Do After an AI Tool Goes Live
- Anchors for Answers
- Mains Relevance
- Related Topics to Study Next
- NCERT Concepts to Revise
- Common Errors / Trap Areas
1. At a Glance
- The core problem: a healthcare AI system can be technically accurate and still fail to improve patient outcomes such as recovery or mortality. Very few hospitals can measure the gap between accuracy and clinical benefit [1].
- The "accuracy ≠ efficacy" gap: "A system achieving 95% accuracy need not perform better in clinical settings" [1]. Performance in a lab or validation study is not the same as clinical utility.
- The structural cause: hospitals struggle to access and combine patient data after discharge, so they cannot work out the clinical or economic return (ROI) of AI tools [1].
- Why it matters for UPSC: it sits at the meeting point of GS-II (health governance, regulating medical technology) and GS-III (AI, emerging technology), plus GS-IV (ethics of automated decisions). India's own ethical framework for health AI comes from ICMR [2].
2. Why in the News
- On 22 November 2026, The Hindu (Page 10, Tiruchirapalli edition) reported remarks by Emmanuel Bacry, scientific director of France's Health Data Hub, at the Bengaluru Tech Summit [1].
- He said: "There are maybe six or seven hospitals in the whole world today that are able to evaluate medical devices deployed in their system" [1].
- He cited a case where a rules-based AI alert for severe kidney failure in ICUs was linked to a doubling of deaths in 2 of 6 U.S. hospitals where it was deployed in 2022 [1].
3. Background & Evolution
- Early health AI: rules-based expert systems encode clinical "if-then" logic. The kidney-failure alert Bacry cited was of this type, not a machine-learning model [1].
- 2022: the ICU kidney-failure alert was deployed in six U.S. hospitals. Deaths doubled in two of them. Researchers offered hypotheses, but the exact cause remains unknown [1].
- 2023 (India): ICMR issued ethical guidelines for using AI in biomedical research and healthcare, built on 10 patient-centric ethical principles [2].
- 2026: the international debate moves from "is the model accurate?" to "does the deployed model improve outcomes, and can hospitals prove it?" [1].
4. Core Static Facts
| Item | Fact |
|---|---|
| Speaker / institution | Emmanuel Bacry, scientific director, Health Data Hub (France) [1] |
| Forum | Bengaluru Tech Summit [1] |
| Global evaluation capacity | Roughly 6–7 hospitals worldwide can evaluate medical devices deployed in their own system [1] |
| Outcome metrics hospitals cannot track | Post-discharge recovery rates and mortality [1] |
| Case-study technology | Rules-based AI / expert system: an ICU alert for severe kidney failure [1] |
| Case-study result | Deaths doubled in 2 of 6 U.S. hospitals (deployed 2022) [1] |
| Suspected contributing factor | No tested response protocol among hospital staff [1] |
| Indian ethical framework | ICMR guidelines on AI in biomedical research and healthcare, with 10 patient-centric ethical principles [2] |
Key terms
- Accuracy: how often the model's output is correct on a test dataset.
- Clinical utility / efficacy: whether using the model actually improves patient outcomes.
- Real-world evidence (RWE): outcome data collected after deployment, in routine care.
- Post-market surveillance: monitoring a device after approval and deployment.
- Expert system: AI built on hand-coded rules, as opposed to machine learning that learns from data.
- Socio-technical system: the idea that outcomes depend on the tool plus staff workflows and protocols.
5. Multi-Dimensional Analysis
Scientific / Technological
- Strong performance on a validation dataset does not carry over automatically to a ward; "95% accuracy" says nothing about outcome improvement [1].
- Replacing a human-generated alert with a device-generated one "changes the whole thing". The intervention is the workflow, not only the algorithm [1].
- Measuring outcomes needs interoperable, linked patient records that continue after discharge. Most hospitals lack these [1].
Economic
- Hospitals "don't know how to compute" the ROI of AI tools [1]. Purchasing decisions therefore rest on vendor accuracy claims, not proven value.
- There is a risk of spending scarce health budgets on tools with no shown benefit, or with net harm, as the kidney-alert case suggests [1].
Ethical / Governance
- A tool that "functions effectively" was still linked to more deaths [1]. This raises questions of accountability: vendor, hospital or clinician?
- The cause remained unknown even after the event [1], which argues for transparency and mandatory reporting of adverse events.
- ICMR's patient-centric principles [2] create an ethical duty to show benefit, not only safety.
Administrative
- The suspected failure point was the missing tested response protocol among staff [1]. Deploying AI therefore needs SOPs, training and drills.
- Evaluation capacity is concentrated in a handful of institutions worldwide [1]. District and secondary hospitals in India are likely to be even less equipped (this is an inference, not stated in the sources).
Social
- Patients bear the risk of tools that have not been evaluated. Outcomes that are not tracked after discharge [1] hide harms most among people with weak follow-up access, such as rural or low-income patients (an analytical inference).
Legal / Regulatory
- The case for regulating AI across its whole lifecycle, including post-deployment surveillance, rather than through one-time pre-market approval, follows directly from the 2022 case [1].
- The ICMR guidelines are ethical guidance, not a statute [2].
6. Recent Developments (last 12–18 months)
- 22 Nov 2026: Bacry's remarks at the Bengaluru Tech Summit, reported by The Hindu [1].
- No other dated 2025–26 developments were retrieved from whitelisted sources this session. Update this section with PIB, MeitY or ICMR releases before relying on it.
7. Prelims Hooks
- France's national health data platform is the Health Data Hub; its scientific director is Emmanuel Bacry [1].
- Health-AI evaluation capacity was discussed at the Bengaluru Tech Summit (Nov 2026) [1].
- Only about 6–7 hospitals worldwide can evaluate medical devices deployed in their own system [1].
- The ICU severe-kidney-failure alert cited was a rules-based AI (expert system), not machine learning [1].
- The kidney alert was deployed in six U.S. hospitals in 2022; deaths doubled in two [1].
- The precise cause of the excess deaths remains unknown [1].
- Suspected factor: no tested response protocol among staff [1].
- Outcome metrics hospitals struggle to track: post-discharge recovery and mortality [1].
- High model accuracy (e.g., 95%) does not guarantee better clinical performance [1].
- ICMR's AI-in-healthcare ethical guidelines rest on 10 patient-centric principles [2].
- India's AI-in-health ethics guidance was issued by ICMR, not NITI Aayog or MeitY [2].
8. Why a Licence Before Launch Cannot Catch This Kind of Harm
- India's draft rules mostly check the tool on paper, before it is used
- In October 2025, CDSCO (India's drug and medical device regulator) released draft guidelines for medical device software [7].
- They cover SaMD (Software as a Medical Device, meaning software that is itself the medical device) and AI or machine learning (ML) tools [7].
- The draft deals with definitions, risk class and quality management systems (the factory-style checks on how the software is built) [7].
-
A government release says companies seeking approval for AI devices must submit detailed technical documents [8].
-
The kidney alert would have passed this kind of check
- The alert was "functioning effectively". It did what it was designed to do [1].
- The harm showed up only after it was deployed, and only in 2 of the 6 hospitals [1].
-
A file of documents checked before launch cannot show that a tool will hurt patients in one hospital and not in another (inference).
-
Even industry says the rules are unclear on the key question
- The Medical Technology Association of India asked for clearer rules on clinical evaluation protocols. That means how a company must prove the tool actually helps patients [7].
- It also asked for rules on algorithm change management, meaning what happens when an AI model is updated after approval [7].
- A Parliamentary Standing Committee asked CDSCO to finalise separate approval routes for digital health and AI-based medical software [6].
- So the question of who checks outcomes after launch is still open in India.
9. India Has a Body to Measure Value, but Not the Data It Needs
- HTAIn already has the mandate
- HTAIn (Health Technology Assessment in India) works under the Department of Health Research. It judges whether a health technology is worth its cost [5].
- In a renewed agreement with the National Health Authority (NHA), it will produce evidence on digital health tools, including AI-based ones, used under AB-PMJAY, India's public health insurance scheme [5].
-
The agreement covers clinical efficacy (whether the tool improves patients' health), cost-effectiveness and equity (whether all groups benefit fairly) [5].
-
The missing input is outcome data after discharge
- HTAIn can only judge efficacy if someone records what happened to patients after they left hospital.
-
Bacry's point is that hospitals cannot link this data [1]. Without it, HTAIn's efficacy question has no answer.
-
A national study cannot explain local failure
- The same alert doubled deaths in 2 hospitals but not in the other 4 [1].
- A national assessment gives one verdict for one tool. It cannot tell a district hospital whether the tool will work with its own staff and its own protocols (inference).
-
So India needs both national HTA and local monitoring in each hospital, not one in place of the other.
-
HTA is starting to drive money decisions
- A Parliamentary Standing Committee wants HTAIn's findings on efficacy and cost built into NPPA's price-setting rules for costly medicines [6].
- The same logic could later apply to what PM-JAY pays for AI tools. But only if outcome data exists to feed it.
10. The Case for Using AI Now and Measuring Later, and Where It Fails
- The strongest argument for deploying first
- Indian public hospitals are short of specialists. An imperfect alert may still be better than no alert at all.
- If 6–7 hospitals in the world can evaluate their own devices [1], then demanding proof first would stop almost every hospital from ever using AI.
-
Ordinary medical practice also changes over time without a trial for every new workflow.
-
What this argument gets right
- Asking for a full clinical trial before every software tool goes live is too slow and too costly for most hospitals.
-
Accuracy studies are still useful. They screen out tools that are plainly bad.
-
Where it fails
- The kidney alert shows the harm can be large (deaths doubled) and still go unseen. It was a simple rules-based tool, not a hard-to-explain one [1].
- "Better than nothing" assumes the tool cannot make things worse. The 2022 case proves it can [1].
-
If nobody tracks outcomes, nobody finds the harm, and the cause stays unknown, as it did here [1].
-
The fair middle path
- Allow the tool to be used, but only on the condition that its results are watched after launch.
- This is the WHO's approach. Developers must prepare for surveillance after the tool is put to use, and regulators should check that it stays effective and useful over time [3].
11. What Each Actor Should Do After an AI Tool Goes Live
- CDSCO should add outcome reporting after approval to its final SaMD rules
- The draft is about checks before approval [7]. Parliament has asked for a clear route for AI software [6].
-
The WHO's 2021 guidance asks regulators to keep checking that a tool is effective over its whole life, not just at approval [3].
-
Government should require independent audits for AI used at scale
- For large AI models used widely in health, the WHO's 2024 guidance asks governments to make audits and impact assessments compulsory after release. Independent third parties should do them [4].
-
India could apply the same rule to any AI tool deployed across PM-JAY hospitals (inference).
-
Hospitals should test the human response before switching a tool on
- The suspected failure was that staff had no tested response protocol [1].
- So before go-live, a hospital should write down who acts on each alert, and practise it with drills.
-
The hospital should then compare death and recovery rates before and after launch.
-
NHA and HTAIn should use their agreement to study AI tools inside PM-JAY
- The agreement already covers AI-based interventions and equity [5].
-
PM-JAY claims data could become the post-discharge record that single hospitals lack (inference).
-
India can learn from France's national data model
- France pools health data through a national platform, the Health Data Hub [1].
- That is what lets outcomes be followed after patients leave the hospital. India's version would build on ABDM (Ayushman Bharat Digital Mission) health records, within the limits of the DPDP Act on data use.
12. Anchors for Answers
- Data: Only about 6–7 hospitals in the world can evaluate the medical devices used in their own system [1]. A rules-based ICU kidney alert was linked to deaths doubling in 2 of 6 U.S. hospitals (2022) [1].
- Report/Committee: WHO, Ethics and Governance of AI for Health (2021): developers must prepare for surveillance after the tool goes live [3]. Parliamentary Standing Committee on Health, 175th–177th Reports: asks for separate CDSCO routes for AI software [6].
- Law/Case: Medical Devices Rules, 2017, and CDSCO's draft guidelines on medical device software (Oct 2025), which cover SaMD and AI/ML [7].
- Comparison: WHO 2024 guidance on large AI models: compulsory audits by independent third parties after release [4]. France's Health Data Hub: a national platform for health data [1].
- Scheme: NHA–DHR/ICMR agreement: HTAIn to assess AI-based tools under AB-PMJAY for efficacy, cost and equity [5].
13. Mains Relevance
- GS-II: Issues relating to development and management of Health; Government policies and interventions.
- GS-III: Science & Technology: developments and their applications; Awareness in IT, computers, robotics, AI.
- GS-IV: Ethics in public administration; accountability; ethical issues in technology use.
Question stems
- "High algorithmic accuracy does not guarantee better patient outcomes." Examine this statement in the context of deploying AI in Indian public hospitals, and suggest an evaluation framework. (GS-III, 15 marks)
- Discuss the governance and accountability challenges that arise when AI-based clinical decision-support systems cause patient harm. How far do existing Indian ethical guidelines address them? (GS-II/IV, 15 marks)
- Interoperable health data is a precondition for evidence-based adoption of health technology. Critically analyse in the Indian context. (GS-II, 10 marks)
14. Related Topics to Study Next
Items marked † were not retrieved from whitelisted sources this session. Check their facts (years, budgets, agencies) against PIB, PRS or indiacode before use.
- ICMR Ethical Guidelines for AI in Healthcare (2023) [2]: India's main ethics framework for health AI.
- Ayushman Bharat Digital Mission / digital health records †: the linked, longitudinal patient data that outcome measurement depends on.
- Digital Personal Data Protection Act, 2023 †: the legal limits on sharing and linking patient data for evaluation.
- Medical Devices Rules / CDSCO regulation of Software as a Medical Device †: how AI tools are classified and approved.
- IndiaAI Mission †: national AI capacity, compute and datasets, including health uses.
- WHO guidance on ethics and governance of AI for health †: the global norms on lifecycle evaluation.
- Automation bias and human-in-the-loop design: a conceptual link to the "alert replacement" failure in the 2022 case [1].
- Health Technology Assessment (HTA) in India †: the institutional route to measuring clinical and economic value.
15. NCERT Concepts to Revise
- Artificial intelligence (Economy): The article is about AI tools in hospitals; this explains what such systems are and do.
- Indian Council of Medical Research (Polity): ICMR issued India's ethical guidelines for AI in healthcare and biomedical research.
- Health Technology Assessment (Economy): HTA is the formal route to judging clinical effectiveness and cost value of health technologies.
- Ayushman Bharat Digital Mission (Economy): Linked, interoperable digital health records are what post-discharge outcome tracking depends on.
- Digital Personal Data Protection Act 2023 (Polity): It sets legal limits on sharing and linking patient data needed for evaluating AI tools.
- Central Drugs Standard Control Organisation (Polity): India's regulator that approves and oversees medical devices, including AI-based software used in clinics.
- Accountability (Polity): Asks who answers when an AI tool is linked to patient deaths: vendor, hospital or clinician.
- Transparency (Polity): The cause of the doubled deaths stayed unknown, showing the need for open reporting of adverse events.
- Governance (Polity): Safe AI use needs rules, protocols and oversight across the tool's whole lifecycle, not just approval.
- Excess mortality (Economy): The kidney-alert case showed deaths above normal expectation, a measure of real harm from a tool.
- Automation (Economy): Replacing human-generated alerts with machine-generated ones changed workflows and links to over-reliance on automated decisions.
- IndiaAI Mission (Economy): India's national AI mission covers compute and datasets, including health uses of AI.
16. Common Errors / Trap Areas
- Expert system vs ML: the kidney-alert case concerned rules-based AI, not deep learning [1]. Do not assume all health-AI failures come from "black-box" models.
- "The system malfunctioned": wrong. It was "functioning effectively"; the harm was linked to how it was used in practice, most likely the missing response protocol [1].
- Numbers: deaths doubled in 2 of 6 hospitals, not all six, and the year was 2022 [1]. The country was the U.S.; the speaker's institution is French [1].
- Wrong body: India's AI-in-healthcare ethics guidelines are from ICMR [2]. Do not attribute them to NITI Aayog, MeitY or NHA.
- Guidelines vs law: the ICMR guidelines are ethical guidance, not a statute [2].
Sources
- 1The Hindu: "'Healthcare AI: hospitals lack the capacity to measure real-world patient outcomes'", Balasubramanyam C.P., 22 Nov 2026, Page 10, Tiruchirapalli edition:thehindu.com · tier 4
- 2Down To Earth: "ICMR releases guidelines for artificial intelligence use in the health sector" / "Guiding document outlines 10 key patient-centric ethical principles for AI application in health sector":downtoearth.org.in · tier 4
- 3Ethics and governance of artificial intelligence for health: WHO guidance (2021)iris.who.int · tier 2
- 4WHO releases AI ethics and governance guidance for large multi-modal modelswho.int · tier 2
- 5NHA renews MoU with DHR and ICMR to strengthen evidence-based healthcare decision-makingpib.gov.in · tier 1
- 6Press release on the 175th, 176th and 177th Report of Parliamentary Standing Committee on Health & Family Welfarepib.gov.in · tier 1
- 7CDSCO issues draft guidelines to regulate medical device softwarebusiness-standard.com · tier 4
- 8Update on Secure AI in Health Initiativepib.gov.in · tier 1