Glossary
What is AI bias
In short
AI bias is a systematic error that makes a model treat some groups of people or some cases worse, or less fairly, than others.
Security and law · AI bias
MMOOZZGG.com editorsChecked: 1 min read
For a child
Imagine you learn to recognize dogs only from pictures of white poodles. Then you see a black dachshund and say it is not a dog. You are not mean: you were just shown too few kinds of dogs. A computer helper can be the same. If it learned from one-sided examples, it makes more mistakes with those it rarely saw. That is why its answers need checking.
In plain words
A neural network learns from examples that people collected. If these examples have many cases of one kind and few of another, the model remembers the imbalance and carries it into its decisions. For example, a resume screening system trained on past hires may give lower scores to people the company rarely hired before, even when their experience fits. Nobody programmed this on purpose: the imbalance came from the data. Sometimes people add it too, when they choose what counts as a good result.
For business
A skew in a model’s decisions hurts a business three times: unfair decisions about customers and candidates, complaints and inspections, and lost trust. The risk shows up most often in hiring, credit scoring, customer assessment and moderation. Under 152-FZ, Russia’s personal data law, a decision with legal effect for a person cannot rest only on automated processing without their written consent or a direct basis in federal law. So before launch, check whether the model errs equally across groups, keep a person in disputed cases and record which features it decides on. Build this check into the pilot instead of leaving it for later.
For a specialist
AI bias is a systematic deviation in a model’s results that leads to unequal errors or outcomes for groups. NIST SP 1270 (Schwartz and co-authors, March 2022) names three categories: systemic, statistical and computational, and human. It stresses that zero bias risk does not exist. The sources are sampling bias, a historical skew in the labels, proxy features linked to protected attributes and the choice of the target metric. A classic example is the Gender Shades study (Buolamwini and Gebru, 2018). It found that three commercial systems erred in determining gender up to 34.7% of the time for darker-skinned women, against a maximum of 0.8% for lighter-skinned men. Fairness metrics include demographic parity, equalized odds and calibration by group. Methods to reduce bias are data audit and rebalancing, constraints during training, threshold tuning after training, sliced evaluation and a human in the loop.
Example
An installment-payment service noticed that its model refused residents of some districts more often. A check showed that the address in the training data stood in for income. The team removed the feature, and an employee now reviews the disputed refusals.
Related terms
Questions and answers
Sources
- Buolamwini, Gebru, Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification, PMLR 81, 2018proceedings.mlr.press
- Schwartz et al., Towards a Standard for Identifying and Managing Bias in Artificial Intelligence, NIST SP 1270, March 2022nvlpubs.nist.gov
- Federal Law No. 152-FZ of 27 July 2006 "On Personal Data", Article 16, ConsultantPlusconsultant.ru
Not sure whether your company needs this?
AI solutions map