Automation bias

Indian Economy glossary

Also called: Human-in-the-loop design · Topic: Schools of Economic Thought and Economic Laws · NCERT: Beyond NCERT

Meaning

Automation bias is the habit of trusting a machine's alert or decision too much. People follow it even when their own judgement or other evidence says it may be wrong. They also tend to miss problems the machine does not flag. It is a behavioural bias: a mental shortcut that saves effort but causes predictable mistakes. Human-in-the-loop design is the fix. It builds a system so that a trained person checks the machine's output, can override it, and knows exactly what to do when an alert appears.

Example

In 2022, six U.S. hospitals put a rules-based AI alert for severe kidney failure into their ICUs (intensive care units). Deaths doubled in two of them, even though the tool was working as designed. The suspected reason was that staff had no tested plan for how to respond to the alert. The lesson for Indian hospitals is to keep doctors in charge of each alert, and to write and practise the response before switching any AI tool on.

Don't confuse with

  • Anchoring: here a person leans too much on the first number they see, such as a high printed MRP. In automation bias, the person leans too much on the source of the information, which is a machine.
  • Status quo bias: here a person sticks with the current or default option. In automation bias, the person sticks with whatever the machine recommends, even when that is a change.

Read more