How AI training models could compromise privacy

1. At a Glance

• Core issue: AI models can memorise personal data during training. Credible studies show that querying a model with "anonymised" data can pull out those memorised details, which can then be used to identify individuals and build their profiles [1].
• Mechanism: Personal data can become part of the model weights, which creates a risk of regurgitation. The OECD flags memorisation as a particular problem for chatbots built on open-source models [8].
• Legal gap: The Right to Privacy was upheld by the Supreme Court in 2017, but India's current legal instruments look inadequate for AI-specific risks [1].
• UPSC relevance: The topic spans fundamental rights (GS-II), technology governance and data protection (GS-III), and the ethics of surveillance and consent (GS-IV).

2. Why in the News

• An op-ed in The Hindu (Chennai edition, page 8, 3 November 2026) by Tulika Avni Sinha, Senior Research Fellow (Data Governance, Privacy and AI) at the World Privacy Forum, argues for filling specific gaps in privacy law instead of imposing broad AI regulation [1].
• The DPDP Rules, 2025 were notified in November 2025 and are being phased in. Some consent and safety provisions take effect only in 2027 [1][2].
• In 2026 the OECD published a report on enhancing user control and privacy protections in the context of AI, which addresses memorisation and regurgitation [8].

3. Background & Evolution

• IT Act, 2000: Before the DPDP Act, this was India's main law on the use and management of personal data [1].
• 2011 Rules under the IT Act: Together with the Act, these broadly regulated the export of "sensitive personal data" [1].
• 2017: The Supreme Court upheld the right to privacy as a constitutional right [1].
• 2023: Parliament passed the Digital Personal Data Protection (DPDP) Act [1].
• 2023–24: OECD work found that generative AI creates privacy risks at both the development (training) and deployment stages. It also held that the data minimisation principle in the OECD Privacy Guidelines applies to AI training [10]. A 2024 OECD paper flagged scraping of personal data to train generative AI as a significant privacy challenge [9].
• January 2025: MeitY released the Draft DPDP Rules, 2025 for public consultation, with comments due by 18 February 2025 [4].
• 13/14 November 2025: The DPDP Rules, 2025 were notified with an 18-month phased compliance window [2][3]. PIB releases give different dates: 14 November in the main release and 13 November in a later note.
• 3 December 2025: PIB announced a simplified compliance framework for start-ups and certain Data Fiduciaries [5].

4. Core Static Facts

5. Multi-Dimensional Analysis

Legal / Constitutional

• The right to privacy has constitutional status (SC, 2017), but the statutory tools for AI-specific harms such as memorisation look inadequate [1].
• Key provisions of the DPDP framework only take effect in 2027, so there is a protection gap during the transition [1][2].
• The 90-day erasure right is hard to enforce for data already built into model weights. An erasure request may not remove what a trained model has memorised [2][8].

Scientific / Technological

• Personal data can be absorbed into model weights during training and later regurgitated in outputs [8].
• "Anonymisation" is not a reliable safeguard, because targeted queries can extract memorised details and re-identify people [1].
• Models built on open-source weights carry a higher risk, because anyone can query or fine-tune them freely [8].

Ethical / Governance

• Training data is collected by scraping without the user's knowledge or consent, which raises consent and fairness problems [9].
• UNESCO calls for privacy-protective data-access frameworks for training and validating AI models [11].
• Data minimisation applies to the whole AI lifecycle, not just the deployment stage [10].

Economic

• AI can greatly boost productivity, and indiscriminate regulation risks holding back innovation and growth [1].
• The article's middle path is to fill specific privacy gaps (the "lower-hanging fruit") without falling behind in the "AI race" [1].
• The simplified compliance regime for start-ups tries to balance protection against ease of doing business [5].

Social

• Profiling from extracted personal data can enable surveillance, discrimination and fraud against individuals [1].
• The OECD flags that leaks of sensitive data, such as medical or financial records, are especially harmful [8].

Administrative

• Phased roll-out means the Data Protection Board is operational while substantive duties are still pending [6][7].
• The Board will need technical capacity to audit AI models. The independent audits and impact assessments required of SDFs are the main lever [3].

6. Recent Developments (last 12–18 months)

• 13/14 November 2025: DPDP Rules, 2025 notified, with an 18-month phased timeline [2].
• 17 November 2025: PIB backgrounder on the DPDP Rules set out the duties of Significant Data Fiduciaries [3].
• 3 December 2025: Simplified compliance framework for start-ups and certain Data Fiduciaries announced [5].
• 2025–26: The PRS Annual Policy Review (April 2025 – March 2026) recorded that the Board-related Rules took effect immediately while other provisions follow staggered timelines [6].
• 2026: PIB's Digital India release highlighted data protection and the setting up of the Data Protection Board [7].
• October 2026: OECD report on user control and privacy protections in AI, covering memorisation and regurgitation risks [8].
• 3 November 2026: The Hindu op-ed on how AI training compromises privacy [1].

7. Prelims Hooks

• The DPDP Act was passed in 2023, and its Rules were notified in November 2025 [1][2].
• The DPDP Rules give an 18-month phased compliance window [2].
• Requests for access, correction, update or erasure must be handled within 90 days [2].
• The Rules on setting up the Data Protection Board took effect immediately [6].
• Significant Data Fiduciaries must carry out independent audits and impact assessments [3].
• The Draft DPDP Rules were released by MeitY, with public comments due by 18 February 2025 [4].
• Before the DPDP Act, the IT Act, 2000 and its 2011 Rules governed sensitive personal data [1].
• The Supreme Court upheld the right to privacy as a constitutional right in 2017 [1].
• Regurgitation means an AI model outputs personal data it memorised in its weights during training [8].
• The OECD says the data minimisation principle of its Privacy Guidelines applies to AI training [10].
• UNESCO's Recommendation on the Ethics of AI calls for privacy-protective data-access frameworks for training AI models [11].
• A simplified compliance framework for start-ups under the DPDP regime was announced on 3 December 2025 [5].

8. Why the Right to Erase Cannot Reach a Trained Model

• Erasure works on stored files, but a model stores nothing in a file
• Under the DPDP Rules, a Data Fiduciary (the company or body that decides why and how your data is used) must act on an erasure request within 90 days [2].
• This works when your data sits in a database row. The company deletes the row and the job is done.
• In an AI model, your data is not kept as a row. It is spread across millions of numbers called model weights (the settings the model learns during training) [8].

• So the company can delete the original training file and still leave a model that remembers and repeats your details. This repeating is called regurgitation [8].

• The technical fix is not ready yet

• Machine unlearning means making a trained model "forget" specific data without training it again from scratch.
• The OECD's 2026 report treats unlearning as a promising area. It asks regulators and industry to study it together, which means it is not yet a proven tool a regulator can demand [8].

• The result: a Data Principal (the person the data is about) can win an erasure request on paper and still not be protected.

• The gap is widest for open-source models

• Once open-source model weights are released, anyone can download, copy and fine-tune them [8].
• No single Data Fiduciary controls every copy, so nobody can be ordered to erase data from all of them.

9. The Scraping Loophole: Public Data Falls Outside the Law

• The DPDP law does not cover data that is already public
• Personal data that a person has made public, or that someone else made public under a legal duty, is kept outside the law [12].

• Social media posts, public profiles and published records all fall in this category.

• This is exactly the data that AI companies scrape

• The OECD calls web scraping of personal data to train generative AI a significant privacy challenge [9].

• Because public data is outside the law, a company scraping it does not need consent and has no duty to notify anyone.

• Small public pieces can add up to a private profile

• One public post seems harmless. But a model trained on millions of posts can join scattered details together.
• The article shows that querying a model with "anonymised" data can pull out memorised details and identify the person [1].

• So the law protects data piece by piece, while the harm comes from combining the pieces.

• This clashes with data minimisation

• Data minimisation means collecting only the data you actually need for a stated purpose.
• The OECD says this principle applies at the training stage too [10]. Scraping the whole web is the opposite: collect everything first, decide later.

10. Fix Privacy Law or Pass a New AI Law? The Real Debate

• The article's position, and the government's
• The op-ed argues for filling specific privacy gaps, not imposing broad AI regulation, so India does not fall behind in the "AI race" [1].

• The India AI Governance Guidelines take the same line. The drafting committee, set up by MeitY in July 2025, relies on existing laws like the DPDP Act instead of a new AI statute [13].

• The strongest case against this approach

• The "existing law" India relies on has the very holes this note describes. Public data is excluded [12]. Erasure cannot reach model weights [8]. Key consent provisions only start in 2027 [1].
• So "use the DPDP Act" can mean "use a law that does not reach the problem".

• A law written for databases cannot easily handle a technology where data is absorbed into weights.

• What the critics get right — and what they miss

• They are right that relying on existing law alone, without changing it, leaves people unprotected.
• But a full new AI law takes years to draft and pass. Targeted amendments can move faster.
• The guidelines themselves recommend targeted amendments on data protection, liability and copyright [13]. The honest debate is not "new law vs no law". It is whether those amendments arrive before most AI training in India has already happened.

11. When the Government Is the One Holding the Data

• The State can exempt its own agencies
• The central government can, by notification, exempt its agencies from the law on grounds such as sovereignty, security and public order [12].

• PRS noted that government agencies are not required to delete data once its purpose is over [12].

• Why this matters more in the age of AI

• PRS warned that these exemptions could let an agency build broad profiles of citizens [12].
• Large government datasets are valuable for training AI models. A model trained on them can memorise personal details just like any other model [8].

• If the training body is exempt, the erasure and consent rights of the Data Principal do not apply at all.

• Data collected for one use can be reused

• The State may process personal data without consent to give benefits, services, licences or permits [12].

• PRS argued that allowing reuse of this data for other purposes weakens purpose limitation (the rule that data is used only for the reason it was collected) [12].

• The constitutional test

• The 2017 Puttaswamy judgment requires any intrusion on privacy to meet the tests of legality, necessity and proportionality [1].
• A blanket exemption that also covers AI training would be hard to defend under the proportionality test (the State must use the least intrusive means for its goal).

12. Who Should Do What Next

• MeitY: bring AI training on scraped public data under the DPDP Act
• Today the public-data exclusion lets scraping happen without consent [12].

• The India AI Governance Guidelines already recommend targeted amendments on data protection [13]. Narrowing this exclusion for AI training is the most direct one.

• Data Protection Board: issue guidance on memorisation and erasure

• The OECD asks regulators and industry to work together on memorisation in model weights and on machine unlearning [8].

• The Board could tell companies what counts as a valid erasure for a trained model. For example, blocking the data in outputs until the model is retrained.

• Make Significant Data Fiduciaries test their models for leaks

• SDFs must already do independent audits and Data Protection Impact Assessments (a formal check of privacy risks before processing) [3].

• MeitY could require these assessments to include tests for regurgitation: try to make the model reveal personal data, and record the result.

• Train models without exposing raw data

• A MeitY draft on the Data Empowerment and Protection Architecture (DEPA) proposes privacy-preserving training [14].
• It uses secure data environments (the model is trained inside a locked space and the raw data never leaves it) and differential privacy (adding careful statistical noise so no single person can be picked out) [14].

• This tackles memorisation at the source, instead of trying to erase it later.

• Work with other countries' privacy regulators

• The G7 roundtable of data protection authorities issued a joint statement on generative AI in June 2023 [9].
• Models are trained across borders, so the Data Protection Board needs to coordinate with foreign regulators, not act alone.

13. Anchors for Answers

• Data: 90 days for a Data Fiduciary to handle erasure requests — a deadline that cannot be applied to data already in model weights [2][8]
• Report/Committee: OECD, Enhancing user control and privacy protections in the context of AI (2026) on memorisation and machine unlearning [8]; India AI Governance Guidelines drafting committee (set up July 2025) recommending targeted amendments to existing law [13]
• Law/Case: K.S. Puttaswamy (2017) — legality, necessity, proportionality test [1]; DPDP Act, 2023 — excludes personal data made public by the person, and lets the government exempt its agencies [12]
• Comparison: G7 Roundtable of Data Protection and Privacy Authorities — joint statement on generative AI, June 2023 [9]; OECD — data minimisation applies to AI training [10]
• Scheme: DEPA draft — privacy-preserving AI training through secure data environments and differential privacy [14]

14. Mains Relevance

• GS-II: Fundamental rights (Article 21 and privacy); government policies and interventions; statutory bodies (Data Protection Board).
• GS-III: Awareness in IT, computers and AI; challenges to internal security through communication networks and cyber security.
• GS-IV: Ethics in technology; consent, autonomy and surveillance; corporate responsibility.
• Possible question stems: 1. "Anonymisation is no longer a sufficient safeguard in the age of large AI models." Discuss with reference to India's DPDP framework. (15 marks) 2. Examine whether the Digital Personal Data Protection Act, 2023 and its Rules adequately protect citizens against privacy risks arising from AI training. Suggest targeted reforms. (15 marks) 3. How can India balance AI-led innovation with the constitutional right to privacy? (10 marks)

15. Related Topics to Study Next

• Right to Privacy judgment (2017): the constitutional foundation for all data-protection law [1].
• DPDP Act, 2023 in detail: Data Fiduciary and Data Principal duties, penalties, exemptions [2].
• IT Act, 2000 and its 2011 Rules: the earlier regime and what the DPDP Act replaced [1].
• IndiaAI Mission and AI governance guidelines: India's approach of promoting innovation over heavy regulation.
• EU AI Act and GDPR: a comparative risk-based regulatory model.
• OECD AI Principles and Privacy Guidelines: international benchmarks on data minimisation [10].
• UNESCO Recommendation on the Ethics of AI: the global ethics framework [11].
• Deepfakes and synthetic media regulation: another set of AI harms to personal identity.

16. NCERT Concepts to Revise

• Right to privacy (Polity): The constitutional basis of the article's argument that AI memorisation and re-identification threaten individuals' privacy.
• K.S. Puttaswamy case (Polity): The 2017 ruling set the legality, necessity and proportionality test for judging intrusions such as data scraping.
• Digital Personal Data Protection Act 2023 (Polity): India's main data law, whose adequacy against AI-specific risks like memorisation is the article's central question.
• DPDP Rules 2025 (Polity): The phased Rules, with 90-day rights and 2027 deadlines, explain the protection gap the article describes.
• Data fiduciary (Polity): AI developers handling personal data are fiduciaries; significant ones face audits and impact assessments.
• Data principal (Polity): Individuals whose data is memorised hold rights to correction and erasure, which are hard to enforce on trained models.

17. Common Errors / Trap Areas

• Date confusion: The Act is from 2023, but the Rules came in 2025. PIB releases give both 13 and 14 November 2025 as the notification date, so "November 2025" is the safe answer [2].
• "Fully in force" trap: The DPDP regime is phased. The Board provisions are immediate, but some consent and safety provisions start only in 2027 [1][6].
• Wrong ministry: The nodal ministry is MeitY, not the Ministry of Home Affairs or DoT [4].
• Anonymised does not mean safe: "Anonymised" data can still lead to re-identification through AI model queries [1].
• Memorisation vs. scraping: These are two different risks. Scraping happens at data collection (the input stage), while memorisation and regurgitation concern what the trained model stores and outputs [8][9].