How to safely auto-close alerts with AI

Auto-closure is the feature teams ask about first and deploy last. In almost every conversation about AI in compliance, someone raises it within ten minutes: if the agent can review an alert, pull the evidence, and reach the same conclusion an analyst would, why does a person still need to click close on something obviously harmless?

‍

And yet six months later those same teams are running AI across their queues with auto-closure switched off. It is one of the most requested capabilities in the category and one of the least deployed. The gap is not really about trust in the abstract. It comes down to two things: teams are measuring the wrong number, and they are managing the model differently from the people doing the same work.

‍

Watch the brief

Tyler Allen, Unit21's CEO, on the two metrics that decide whether auto-closure is safe, and why an AI agent should be audited the same way a human analyst is.

‍

Measure the false positives, not the accuracy

Ask a team how they would evaluate auto-closure and most say accuracy. Some say true positive rate. Both are reasonable instincts and both will mislead you here.

‍

Consider what auto-closure is actually for. It is not finding crime. It is clearing junk out of a queue so your analysts can spend their hours on the alerts that deserve them. So the question worth answering is narrower than accuracy: of the alerts the agent closed, how many should have been escalated?

‍

Two numbers answer that.

‍

• Precision: how accurately the agent closes false positives without missing an escalation. This is the safety number.
• Recall: what proportion of your total false positives the agent is willing to attempt in the first place. This is the volume number.

‍

They pull against each other, which is what makes this feel like a harder decision than it is.

‍

Start narrow at full precision, then widen

The way through is to hold precision fixed and move recall slowly. Start by demanding full precision on the narrowest possible band, the alerts so obviously junk that no analyst would pause over them. In practice that might be three out of every ten false positives in a queue. The agent closes those, you verify nothing escalatable went with them, and then you widen the range.

‍

That is a slower rollout than most roadmaps assume. It is also the version that holds up under examination, because at every point you can show exactly what the agent was permitted to close and on what evidence. This is the same logic that underpins the whole trust ladder for AI in compliance, where auto-close sits at L5 and is reached deliberately rather than switched on.

‍

Manage the agent the way you manage an analyst

The second reason is less obvious.

‍

Every compliance team already runs quality control on human work. You sample decisions, audit a percentage of closed cases, find the mistakes, correct them, and feed that back. Nobody asks whether human review is safe enough to permit. The sampling process is what makes it safe.

‍

Agents rarely get the same treatment. Teams either hold them to a standard of perfection they never asked of people, or they switch them on and stop looking at them entirely. What works is the process you already have: random sampling of auto-closed alerts, the same audit you run on your analysts, findings routed back into how the agent is configured.

‍

Something worth noticing once you start sampling. Teams tend to assume the agent has to clear a perfect bar, when the honest comparison is the analyst who was reviewing those alerts last quarter. Self-driving ran into the same argument. The question was never whether the car is flawless, it was whether it is safer than the person who would otherwise be at the wheel. When teams run that comparison against their own historical closures, the agent often comes out ahead of the human baseline, and that result surprises people more than it should.

‍

It depends on the queue

Auto-closure is not one setting, and treating it as one is where a lot of rollouts go wrong.

‍

Sanctions screening is full of obvious false positives. A transliteration match, a date of birth two decades off, a nationality that does not line up. Those close with high confidence and the volume is significant.

‍

A level two case review is a different animal. Deciding whether to file a SAR carries nuance that does not reduce to a pattern, and so the share of alerts you can safely auto-close is smaller. That is not a shortcoming of the agent. It reflects what the work actually involves, and it is the same reason SAR narrative automation keeps a human reviewing before anything is filed.

‍

Expect different rates across queues and configure them separately. A single global setting will be too timid for sanctions or too aggressive for case review, and probably both at once. If the underlying false positive volume is the real problem, reducing false positives in AML transaction monitoring is the better place to start than closing them faster.

‍

FAQs

‍

What metrics should you use to measure AI auto-closure?

Precision and recall on false positives, rather than overall accuracy or true positive rate. Precision is how accurately the agent closes false positives without missing an escalation. Recall is what proportion of your total false positives it attempts to close.

‍

Why is overall accuracy the wrong metric for auto-closure?

Auto-closure is not finding crime, it is clearing junk out of a queue. Overall accuracy and true positive rate measure a different job. The question that matters is narrower: of the alerts the agent closed, how many should have been escalated?

‍

How should a team start with auto-closure?

Hold precision fixed and move recall slowly. Demand full precision on the narrowest band of obviously junk alerts, perhaps three out of every ten false positives in a queue, verify nothing escalatable went with them, then widen the range on evidence.

‍

How do you QA an AI agent that closes alerts?

The same way you QA human analysts. Randomly sample auto-closed alerts, audit them, and route the findings back into how the agent is configured. Sampling is what makes human review safe, and it is what makes agent review safe too.

‍

Should auto-closure rates be the same across every queue?

No. Sanctions screening is full of obvious false positives and supports high auto-closure volume. Level two case reviews involve nuance that does not reduce to a pattern, so a smaller share can be closed safely. Configure the levels separately per queue.

‍

Is an AI agent more accurate than a human analyst at closing false positives?

The honest comparison is not against a perfect standard but against the analyst who was reviewing those alerts before. When teams run that comparison against their own historical closures, the agent often comes out ahead of the human baseline.

‍

Where to start

If auto-closure keeps sliding down your roadmap, pick one queue. Sanctions is the usual place. Decide what precision you need before anything gets switched on, start with the narrowest band of obvious junk, sample what gets closed, and widen when the evidence supports it.

‍

The teams who get there are not braver than everyone else. They built the measurement first. See how Unit21's AI Agent handles auto-closure, queue by queue, with the audit trail to back it.

‍