MMOOZZGG.com

AI against corruption

Government algorithm errors: lessons from the Netherlands

6 min read5 sources

Government algorithm errors become a disaster not when a model is inaccurate, but when its flag is taken as a verdict. In the Netherlands, a tax authority model counted nationality as a risk factor from 2013, tens of thousands of parents were wrongly accused of fraud, and the scandal led to the government’s resignation. Below is a review of what went wrong and the rules that protect innocent people.

On this page · 6 sections

In short

Government algorithm errors are costly: in the Netherlands, a tax authority model used nationality as a risk factor, and tens of thousands of parents were wrongly accused of fraud. Errors become fewer not from a more accurate model but from rules: a flag is a reason to check, a person makes the decision, banned features are excluded, and a person has the right to know the basis.

The Netherlands: what went wrong

From 2013, the Dutch tax authority checked child benefit applications with a risk scoring model. One of the factors was nationality: applicants without Dutch citizenship got a higher risk score.

The model was self-learning. According to Amnesty International, it adjusted to its accumulated experience and so locked in the errors of its own design.

The reviewer who checked flagged applications by hand did not know why the system gave a high risk. Benefits were suspended, and for years parents were not told what they had done wrong.

The results for families: debts, lost jobs, evictions. Tens of thousands of parents, mostly from low-income families, were wrongly accused of fraud, and in January 2021 the scandal led to the government’s resignation.

Four errors came together here: a banned feature in the model, a flag instead of a basis, no explanation to the person, and no outside check of the model. Each has a section below.

A flag is not an accusation

The Open Contracting Partnership described 73 red flags for all stages of procurement, from planning to contract performance. The same document says plainly: a flag means a risk, not proof of a violation.

A flag often has an honest explanation. A tender with one bidder happens in a narrow market, a contract price rises when the scope of work changes, and an urgent purchase happens after an accident.

So words matter. Reports and the system interface say “the system flagged”, “risk indicator”, “signal for review”, and never “offender” or “theft detected”.

A flag does not go into a personal file and does not become a basis for sanctions. It opens a review, and the reviewer closes it: with a document, an explanation, or by passing the materials to those with the power to decide.

How indicators add up to a score, and why a score is not an accusation either, is covered on the page about risk scoring.

A person decides, the machine suggests

The European Commission’s Arachne+ system works with 78 risk indicators and data on company owners, sanctions lists, and public officials. Yet it makes no automatic decisions: it prepares material for a person.

In Russia, the same principle is written into law. Article 16 of 152-FZ, Russia’s personal data law, bans decisions that create legal consequences for a person or otherwise affect their rights based only on automated processing of their personal data.

In practice, this means three things. The reviewer sees not only the score but the features it came from. The reviewer can remove a flag with an explanation. And the reviewer has time to look into it, not a target for the number of closed signals.

The danger lies elsewhere: a person decides on paper but in fact signs what the machine proposed. In the Netherlands, there was a manual check, but the reviewer did not see the basis and could not argue with it.

A repeat sample check helps against this. A second reviewer re-examines some of the closed and some of the confirmed signals, and they discuss the differences together.

Banned features and a discrimination check

The Dutch model showed that one feature can spoil the whole system. Nationality entered the model as a risk factor, and the system flagged people more often by origin than by their actions.

After its review, Amnesty International called for a ban on nationality and ethnic origin data in risk scoring for fraud checks.

Before launch, make a list of banned features: citizenship, ethnicity, sex, age, place of birth, health. The model does not get them either directly or through substitutes, such as the language of the application or the district where a person lives.

A discrimination check is simple in principle: compare the share of flagged cases across different groups of people or organizations. If small suppliers or one region are flagged several times more often with no difference in confirmed violations, the model is fixed before it goes live.

In procurement, the risk is lower than in benefits: the features concern procedures and companies, not families. But here too you check that the system does not flag new suppliers just for being new.

A person’s right to know the basis

In the Netherlands, for years parents got no answer to the question of what they had done wrong. That is what turned a model error into a disaster for families.

Article 16 of 152-FZ requires the operator to explain to a person how a decision is made based on automated processing and its possible legal consequences, to let the person object, and to review the objection within 30 days.

For a control system, this becomes a simple requirement: every flag leads to a document. The reviewer and the employee see which feature fired, on which data, and which documents settle the question.

A model that cannot show the basis for its flag is not fit for checks that affect people’s rights. Amnesty International calls for dropping “black boxes” and self-learning algorithms where decisions have a major effect on human rights.

Requirements for data and models in the public sector, including Federal Law 243-FZ, are collected on the page about requirements and security.

How to check the model itself

You check a model the same way as an employee: by results. A study by Ash, Galletta, and Giommoni on budget data of Brazilian municipalities showed that, with the same number of audits, a model finds almost twice as many municipalities with corruption as random selection.

This is the right question for any system: how much more accurate is it than random selection or the current selection practice? The answer comes from a comparison on past audits whose outcome is already known.

1. Before launch: run the model on closed cases and count how many confirmed violations it flagged and how many honest purchases it hit.

2. After launch: every quarter, count the share of signals that reviewers closed as false. If this share grows, the data or the practice has changed.

3. Once a year: an outside check of the model and an assessment of its effect on people’s rights. Amnesty International names a mandatory human rights impact assessment as a condition before such systems are used.

4. Always: a log of model changes. Who changed a feature or a threshold, when, and why must be as visible as changes to the rules.

The fears of employees and citizens behind these rules are covered on the page about why AI in control meets resistance.

Sources and fact-checking

Checked by the editors on 16 October 2026.

  1. Amnesty International, 25 October 2021 (amnesty.org)
  2. OCP, Red flags in public procurement, 2024, 2024 (open-contracting.org)
  3. European Commission, Anti-Fraud Knowledge Centre (checked 16 October 2026), 16 October 2026 (antifraud-knowledge-centre.ec.europa.eu)
  4. Ash, Galletta, Giommoni, American Economic Journal: Economic Policy, May 2025, 2025 (aeaweb.org)
  5. Federal Law No. 152-FZ of 27 July 2006, On Personal Data, Article 16 (ConsultantPlus, latest version), 27 July 2006 (consultant.ru)

Found a mistake? Tell us

Publications

For government bodies

We build such systems for government bodies and state companies, from data analysis to a pilot on one scheme. The data stays on the organization's own servers.

Contact us