SHAP Falls Short on Agentic AI Fraud, Researchers Argue

Fraud detection teams have leaned on SHAP values for years to justify why a model flagged a transaction, but that approach breaks down once the thing being scored is an autonomous agent rather than a single prediction. A new analysis published on Towards Data Science argues that agentic AI systems — models that plan, call tools, and act over multiple steps — expose an explainability gap that current attribution methods cannot close. The problem matters because banks and payment processors are already piloting agent-based systems for real-time fraud review.

SHAP, short for SHapley Additive exPlanations, assigns each input feature a contribution to a model's output by borrowing a concept from cooperative game theory. It works cleanly when a model maps a fixed set of features to one score: this transaction's amount, merchant category, and location each get a number showing how much they pushed the decision. Agentic systems do not work that way. An agent may query a customer database, call a risk API, revise its own plan, and then block an account, so the final action depends on a chain of intermediate choices rather than a single feature vector.

That chain is where the explainability problem bites. If an agent declines a legitimate purchase after misreading a tool's output three steps earlier, a SHAP summary of the final scoring call will point at the wrong culprit — or at nothing useful at all. Regulators in the EU and U.S. have started asking lenders to document how automated adverse decisions are reached, and a feature-attribution chart that ignores tool calls and memory state will not satisfy that request. The gap is structural, not a matter of tuning hyperparameters.

The practical stakes are rising fast. Fraud rings have begun using their own automated agents to probe bank defenses at scale, generating thousands of adaptive attempts that shift tactics as soon as a rule catches them. Defenders who deploy agents to keep pace inherit the same opacity: when the system freezes a card, the customer service team needs a reason it can put in writing. Attribution methods built for tabular credit models were not designed for a decision tree of API calls, retries, and self-corrections.

What replaces SHAP is still unsettled. Candidate approaches include tracing the full action trajectory and logging each tool response, building causal graphs over an agent's steps, and constraining agents so their decisions stay auditable by design. None has emerged as a standard, and none has been tested at the volume a major card network handles daily. The open question is whether explainability for agentic fraud tools gets solved before regulators force the issue — or whether institutions will be asked to justify decisions their own systems cannot reconstruct.