Glossary
What is explainable AI?
In short
Explainable AI (XAI) is a set of methods that show why a model made a specific decision and which data influenced it most.
Security and law · explainable AI
MMOOZZGG.com editorsChecked: 1 min read
For a child
When a teacher gives you a grade, you can ask why you lost points, and the teacher shows the mistake. Then you know what to fix next time. Smart programs also give their own “grades”: they decide who gets a discount or which email matters more. Explainable AI is like a teacher who does not just give a mark but shows the reason for it.
In plain words
Many neural networks work as a black box: data goes in, an answer comes out, and even the developer cannot see how it formed. Explainability adds a short reason to the answer. For example, a system rejects an application for an installment plan. Next to the refusal, you see that a short work history and past late payments mattered most, while the home district played almost no role. Such a reason does not reveal the whole model, but it shows which features the model relied on in this case. With it, a person can check whether the decision makes sense and challenge it.
For business
Explanations of model decisions matter wherever AI affects people and money: hiring, credit, discounts, supplier selection, hints for a doctor. Under 152-FZ, Russia’s personal data law, a company that makes a decision about a person only by automated means must explain the procedure and its consequences to that person. Without explainability this is hard, and a dispute with a customer or an audit is easy to lose. Explanations also help inside the company: they show that the model relies on a random feature before it hurts revenue. The cost depends on the type of model. For simple models an explanation costs almost nothing, while large neural networks need separate tools and engineering time. Write the requirement for explanations into the specification at the pilot stage.
For a specialist
Explainability covers methods that make a model’s behavior clear to a person. There are models that are clear by design (linear regression, decision trees, rules) and post-hoc explanations for black boxes. Local methods explain one prediction. LIME (Ribeiro et al., 2016) trains a simple interpretable model on the neighborhood of a specific example. SHAP (Lundberg, Lee, 2017) splits the contribution among features by Shapley values and unifies six earlier approaches. For neural networks, saliency maps and gradient attributions are used, while global methods describe the whole model through feature importance and partial dependence plots. Mechanistic interpretability looks for circuits inside the network that drive its behavior. Article 86 of EU Regulation 2024/1689 gives a person the right to a clear explanation of the role of a high-risk system in a decision that affected them. The limit: a post-hoc explanation is approximate and can differ from the model’s real mechanism.
Example
A bank introduced a model that approves credit limits. A customer complained about a refusal. From the explanation of the decision, an employee saw that an old late payment, long since closed, played the main role. The records were updated, and the decision was reviewed.
Related terms
Questions and answers
Can you explain the decisions of a large language model?
Sources
- Ribeiro, Singh, Guestrin, «Why Should I Trust You?»: Explaining the Predictions of Any Classifier, 2016arxiv.org
- Lundberg, Lee, A Unified Approach to Interpreting Model Predictions, 2017arxiv.org
- Regulation (EU) 2024/1689, article 86, AI Act Service Desk of the European Commissionai-act-service-desk.ec.europa.eu
- Federal Law No. 152-FZ of 27 July 2006 "On Personal Data", article 16, ConsultantPlusconsultant.ru
Where companies use it
Not sure whether your company needs this?
AI solutions map