MMOOZZGG.com

Glossary

What is AI bias

In short

AI bias is a systematic error that makes a model treat some groups of people or some cases worse, or less fairly, than others.

Security and law · AI bias

MMOOZZGG.com editorsChecked: 1 min read

For a child

Imagine you learn to recognize dogs only from pictures of white poodles. Then you see a black dachshund and say it is not a dog. You are not mean: you were just shown too few kinds of dogs. A computer helper can be the same. If it learned from one-sided examples, it makes more mistakes with those it rarely saw. That is why its answers need checking.

In plain words

A neural network learns from examples that people collected. If these examples have many cases of one kind and few of another, the model remembers the imbalance and carries it into its decisions. For example, a resume screening system trained on past hires may give lower scores to people the company rarely hired before, even when their experience fits. Nobody programmed this on purpose: the imbalance came from the data. Sometimes people add it too, when they choose what counts as a good result.

For business

A skew in a model’s decisions hurts a business three times: unfair decisions about customers and candidates, complaints and inspections, and lost trust. The risk shows up most often in hiring, credit scoring, customer assessment and moderation. Under 152-FZ, Russia’s personal data law, a decision with legal effect for a person cannot rest only on automated processing without their written consent or a direct basis in federal law. So before launch, check whether the model errs equally across groups, keep a person in disputed cases and record which features it decides on. Build this check into the pilot instead of leaving it for later.

For a specialist

AI bias is a systematic deviation in a model’s results that leads to unequal errors or outcomes for groups. NIST SP 1270 (Schwartz and co-authors, March 2022) names three categories: systemic, statistical and computational, and human. It stresses that zero bias risk does not exist. The sources are sampling bias, a historical skew in the labels, proxy features linked to protected attributes and the choice of the target metric. A classic example is the Gender Shades study (Buolamwini and Gebru, 2018). It found that three commercial systems erred in determining gender up to 34.7% of the time for darker-skinned women, against a maximum of 0.8% for lighter-skinned men. Fairness metrics include demographic parity, equalized odds and calibration by group. Methods to reduce bias are data audit and rebalancing, constraints during training, threshold tuning after training, sliced evaluation and a human in the loop.

Example

An installment-payment service noticed that its model refused residents of some districts more often. A check showed that the address in the training data stood in for income. The team removed the feature, and an employee now reviews the disputed refusals.

Related terms

Questions and answers

What causes AI bias? Most often the cause is the data: the model repeats the imbalances of the examples it learned from. Poorly chosen features and a poor training goal add to it.
How do you test a model for bias? Compare the share of errors and decisions of the model across groups of customers or candidates on one test set. If the gap is clear, change the data, the features or the thresholds, and leave disputed cases to a person.

Sources

Not sure whether your company needs this?

AI solutions map
Contact us