Agent washing is real: how to evaluate financial crime AI vendors

Ask your next AI vendor one question and watch what happens.

‍

"Walk me through how you engineer context for a single alert. What data does the agent see for this specific decision, and what do you deliberately keep out?"

‍

If the answer is some version of "we send all the relevant data to the model," you are not looking at an agent. You are looking at a chatbot with a confident voice. I have watched senior teams sign real budgets on the strength of a demo that would fall apart on that one question, and I want to save you from being the next one.

‍

There is a name for what is happening here. We borrowed it from "AI washing," the older habit of overstating how much AI is really in a product. The newer, sharper version is agent washing: claiming autonomous AI agents while shipping automation and chatbots underneath. It is widespread in the financial crime market in 2026, and it works for exactly as long as the buyer has no way to test the claim. This post is about that test.

‍

Why agent washing is everywhere right now

The word "agentic" started selling before the technology caught up. Buyers rewarded the term, so vendors reached for it, and "agent" got stapled onto copilots, onto rules engines with a chat window, onto retrieval systems that fetch a document and call it reasoning. None of that is dishonest by accident. It is dishonest on purpose because the gap between a Level 2 copilot and a real Level 3 agent is enormous, and the demo is designed so you never see it.

‍

If you want the full framework behind that gap, I laid out the four levels of agentic AI in a separate piece. The one-line version: the test is whether the system performs the work or merely informs it. Agent washing lives in the space between those two verbs.

‍

The one question that gives it away

Context engineering is the single most important differentiator between a production system and a demo, and almost no one who is faking it can talk about it convincingly.

‍

Here is why it matters. Every model has a context window, and quality degrades as you fill it. A system that dumps all available data into the model produces worse decisions than one that hands each task the minimum sufficient context. A real sanctions check sees the entity, the watchlist matches, and the relationship graph. It also does not see the full transaction history and device data that belong to a fraud case. Scoping that correctly, per task, is hard engineering built on knowing what a good analyst actually looks at. It is not a slider in the settings menu.

‍

So when a vendor cannot explain, in plain language, how they scope context for one specific decision, that is not a small gap. That is the whole thing.

‍

The red flags

Beyond the context question, here are the signals I look for. Any one of them should slow you down.

‍

They cannot explain context scoping per task. Covered above. A credible answer names the exact fields used for the decision and why the rest are excluded. "All the relevant data" is a Level 2 answer wearing a Level 3 badge.

‍

No evaluation datasets from real human decisions. Production systems validate accuracy against years of actual analyst decisions with known outcomes. Ask how many examples, what the source is, and what accuracy threshold a task has to clear before it ships. Synthetic test data is a much weaker answer, and "we tested on some samples" is not an answer at all.

‍

The "agent" is a chatbot over a data lake. No structured workflow. No tool calls. No memory across steps. Just a conversational window for asking questions about your data. That is Level 2 no matter how it is marketed.

‍

No configurable autonomy ladder. Mature platforms give you at least four autonomy levels, configurable per queue and per risk tier, from on-demand assistance to auto-close within guardrails. If it is all-or-nothing, the human always decides, or the AI always decides, the design is not ready for production.

‍

Audit trails without reasoning. Showing what the AI decided, without the reasoning chain, the evidence it cited, and a confidence score, is not defensible to an examiner. A real audit trail carries all three.

‍

The system cannot generate code to query your data. The reliable pattern is an agent that writes deterministic SQL or Python to pull specific fields, rather than letting the model interpret raw data directly. The model reasons, the code executes. Ask whether they do this. If the LLM is reading raw data and "figuring it out," your accuracy will wobble in ways you cannot predict.

‍

What a real answer sounds like

The fastest way to feel the difference is to put the Level 2 and Level 3 answers side by side. Same question, two very different vendors.

‍

How are investigations broken into steps? The copilot says, "The AI summarizes the alert." The agent says, "we run five to eight specialized sub-agents per alert type, in parallel, each with its own scoped context and tools."

‍

How do you validate accuracy? The copilot says, "We test on sample data." The agent says, "we benchmark against years of human-reviewed decisions, measured against our best analysts specifically, not average performance."

‍

Can I configure automation levels? The copilot says, "The human always decides." The agent says, "four levels, configurable per queue, process, and jurisdiction, independent of each other."

‍

You do not need to be technical to hear the difference. One set of answers is about the demo. The other is about production.

‍

The advantage you cannot fake

There is a reason context engineering separates the real from the washed, and it is not clever prompting. It is data. A vendor that has watched analysts make millions of real decisions knows what data actually drives each call, and can build that knowledge into how its AI agents scope every task. A vendor that launched an "agent" last quarter is guessing. This is the structural edge that compounds; the platforms with deep histories of analyst behavior get more accurate in a way newcomers cannot shortcut, and it is worth asking every vendor exactly how much real decision data sits behind their product.

‍

Build the scorecard before the calls, not after

The mistake I see most is teams walking into vendor calls without a rubric, getting dazzled, and reverse-engineering the criteria to fit the demo they liked. Flip it. Write down the six red flags and the questions above, score each vendor the same way, and require a live demonstration for every claim rather than a slide. The full Chartis framework has seven evaluation dimensions and a complete vendor scorecard template, which is exactly the artifact you want in the room.

‍

The question to leave every call with

At the end of each vendor conversation, ask yourself one thing for yourself: on the workflow you care about most, did they show you the system doing the work, or did they show you a window that helps your analyst do it faster? If you cannot tell, assume it is the second one, because the vendors who can do the first will never leave that in doubt.

‍

The best way to calibrate your ear is to watch a real agent answer these questions live. Book a demo and put every question in this post on the table. And if you want the full evaluation framework first, download the Chartis report we co-produced, The Agentic AI Maturity Spectrum in Financial Crime, or see how Chartis scores the field in its 2026 RiskTech Quadrant vendor spotlight.

‍

Frequently asked questions

What is agent washing? Agent washing is when a vendor markets autonomous AI agents but actually delivers automation or a chatbot underneath. It is the agentic-AI version of "AI washing," and it is common in financial crime technology because buyers reward the word "agentic" faster than they verify it.

‍

What is the one question that exposes it? Ask the vendor to explain how they engineer context for one specific decision, what data the agent sees and what it excludes, and why. Vendors running a real agent can answer in detail. Vendors running a chatbot fall back on "we send all the relevant data to the model."

‍

Why does context engineering matter so much? Model quality degrades as the context window fills, so feeding an agent everything produces worse decisions than giving it the minimum sufficient context per task. Scoping that well requires knowing what a good analyst actually uses, which is hard to fake.

‍

What should I ask about evaluation data? Ask how many examples are in the evaluation set, whether it comes from real human-reviewed decisions or synthetic data, and what accuracy threshold a task must hit before it goes live. Real decision data is the gold standard.

‍