UPSC Prelims 2026 Q42: Large Language Models — How LLMs Work
Which of the following statements with regard to Large Language Models (LLMs) used in machine learning is/are correct?
1. LLMs assign probabilities to the next possible words and then pick the one with the highest probability.
2. LLMs process data through mathematical optimization to minimise prediction errors.
3. LLMs produce unbiased outputs.
- 1 only
- 1 and 2 only
- 2 and 3 only
- 1, 2 and 3
Answer: (b) 1 and 2 only
Statement-wise explanation
- 1: correct — An LLM breaks text into tokens (words or word-pieces). At each step it computes a probability for every token in its vocabulary and then picks the next one. Always taking the most probable token is called greedy decoding. Many chatbots instead sample from the top candidates (using settings like temperature, top-k or top-p) so the replies vary, which means this statement is a simplification. Its core idea, that LLMs assign probabilities to possible next words and choose from them, is correct, and UPSC treats it as correct.
- 2: correct — Training means adjusting billions of parameters (weights) so the model's predicted next-token probabilities match real text. The prediction error is measured by a loss function, usually cross-entropy. Gradient-descent optimisation then reduces that loss step by step. This is mathematical optimisation to minimise prediction error, exactly as the statement says.
- 3: incorrect — LLMs learn from huge amounts of human-written text, so they absorb the stereotypes and imbalances in that text. UNESCO's 2024 study 'Bias Against Women and Girls in Large Language Models' found clear gender bias in GPT-2, GPT-3.5 and Llama 2. For example, women were linked to domestic roles far more often than men. This is why India's IndiaAI Mission funds bias-mitigation projects under its 'Safe and Trusted AI' pillar. No current LLM can be called unbiased.
How to eliminate
Start with statement 3. 'Unbiased' is an absolute claim, and any tool trained on human-written data carries human bias. AI bias also features in news and government AI-safety programmes. Rejecting statement 3 removes the two options that include it, leaving '1 only' and '1 and 2'. Statement 2 is the basic definition of machine-learning training: reduce the error between prediction and reality. If you know that, you reach '1 and 2 only'. Some well-read candidates may doubt statement 1, because real chatbots often sample rather than always taking the top word. No option says '2 only', though, so the options themselves show that UPSC accepts statement 1. Fact to remember: LLMs predict the next token from probabilities, learn by minimising a loss, and inherit bias from their data.
Know more
How a Large Language Model works
An LLM is a deep neural network, usually built on the transformer architecture. Google researchers introduced the transformer in the 2017 paper "Attention Is All You Need". Its self-attention mechanism lets the model weigh how relevant every earlier word is when it predicts the next one.
| Stage | What happens | Key terms |
|---|---|---|
| Pre-training | Model reads trillions of tokens and learns to predict the next token | Self-supervised learning, cross-entropy loss, gradient descent |
| Fine-tuning / alignment | Model is tuned on curated examples and human preferences | Instruction tuning, RLHF |
| Inference (generation) | Model outputs a probability for each possible next token, picks one, repeats | Greedy decoding, temperature, top-k/top-p sampling |
Why outputs can go wrong
- Bias: training data reflects society's stereotypes on gender, caste, region and language, and models can amplify them.
- Hallucination: the model produces fluent text that sounds likely but is false, because it optimises for probable wording, not for truth.
- Context window: the model can attend only to a limited number of tokens at a time (its short-term memory).
Indian angle
- The IndiaAI Mission (approved March 2024, outlay ₹10,371.92 crore) subsidises GPU compute for training Indian foundation models.
- BharatGen is a government-funded multilingual LLM initiative covering the 22 scheduled languages.
- The Mission's Safe and Trusted AI pillar funds work on bias mitigation, machine unlearning, explainability and deepfake detection.
Mixture of Experts (MoE) is a design in which only some of the model's parameters are used for each token. This cuts the cost of training and running large models, which is one reason Indian firms use it.
In the news
Indian-built LLMs were in the news through 2025–26. BharatGen worked toward models in all 22 scheduled languages, and the IndiaAI Mission subsidised compute for foundation models. Debate on AI bias, deepfakes and labelling AI-generated content ran alongside, which made the basics of how LLMs work a likely Prelims topic.