MMOOZZGG.com

Glossary

What is explainable AI?

In short

Explainable AI (XAI) is a set of methods that show why a model made a specific decision and which data influenced it most.

Security and law · explainable AI

MMOOZZGG.com editorsChecked: 1 min read

For a child

When a teacher gives you a grade, you can ask why you lost points, and the teacher shows the mistake. Then you know what to fix next time. Smart programs also give their own “grades”: they decide who gets a discount or which email matters more. Explainable AI is like a teacher who does not just give a mark but shows the reason for it.

In plain words

Many neural networks work as a black box: data goes in, an answer comes out, and even the developer cannot see how it formed. Explainability adds a short reason to the answer. For example, a system rejects an application for an installment plan. Next to the refusal, you see that a short work history and past late payments mattered most, while the home district played almost no role. Such a reason does not reveal the whole model, but it shows which features the model relied on in this case. With it, a person can check whether the decision makes sense and challenge it.

For business

Explanations of model decisions matter wherever AI affects people and money: hiring, credit, discounts, supplier selection, hints for a doctor. Under 152-FZ, Russia’s personal data law, a company that makes a decision about a person only by automated means must explain the procedure and its consequences to that person. Without explainability this is hard, and a dispute with a customer or an audit is easy to lose. Explanations also help inside the company: they show that the model relies on a random feature before it hurts revenue. The cost depends on the type of model. For simple models an explanation costs almost nothing, while large neural networks need separate tools and engineering time. Write the requirement for explanations into the specification at the pilot stage.

For a specialist

Explainability covers methods that make a model’s behavior clear to a person. There are models that are clear by design (linear regression, decision trees, rules) and post-hoc explanations for black boxes. Local methods explain one prediction. LIME (Ribeiro et al., 2016) trains a simple interpretable model on the neighborhood of a specific example. SHAP (Lundberg, Lee, 2017) splits the contribution among features by Shapley values and unifies six earlier approaches. For neural networks, saliency maps and gradient attributions are used, while global methods describe the whole model through feature importance and partial dependence plots. Mechanistic interpretability looks for circuits inside the network that drive its behavior. Article 86 of EU Regulation 2024/1689 gives a person the right to a clear explanation of the role of a high-risk system in a decision that affected them. The limit: a post-hoc explanation is approximate and can differ from the model’s real mechanism.

Example

A bank introduced a model that approves credit limits. A customer complained about a refusal. From the explanation of the decision, an employee saw that an old late payment, long since closed, played the main role. The records were updated, and the decision was reviewed.

Related terms

Questions and answers

Why do you need explainable AI if the model is already accurate? High average accuracy does not tell you why the model decided this way in your case. An explanation shows what the model relied on and helps you catch an error or a bias before the customer does.
Can you explain the decisions of a large language model? Partly. You can ask the model to explain its answer or show sources, but such an explanation does not always reflect its real computation.

So important decisions are checked against data and documents, not against the model’s words.

Sources

Where companies use it

Not sure whether your company needs this?

AI solutions map
Contact us