Running AI agents in production: how Rippling and Zepz prove quality every day
Getting an AI agent live is a milestone. Keeping it accurate month after month is the actual job. At The AI Files, Unit21's virtual summit on AI in fraud and AML, two financial crime leaders who run agents on real alert volume explained how they got there and how they keep checking the work.
Rob Fenner is Director and Global Deputy Head of Financial Crimes Compliance at Rippling. Rippling started as an HR platform and now moves money across global payroll, corporate cards and e-wallet balances, with licensed entities in the US, Canada, the UK, Ireland and Australia. Rob owns the controls for those entities: testing, ongoing monitoring, and teams across sanctions, transaction monitoring, onboarding and EDD.
Zaheer Jassat is VP of Product for Payments, Financial Crime and Customer Care at Zepz, the company behind WorldRemit and Sendwave. Zepz helps immigrants send money home, is regulated across the Americas, Europe and APAC, and serves customers mostly in Africa, the Far East and Latin America.
Both were among the first teams to put Unit21's AI agents into production, and both still keep a human on every alert. Unit21 CEO Tyler Allen moderated.
Two different starting points
Rippling started where the volume was. Level one AML transaction monitoring alerts are simple, and Rob's team had already tuned them to be as quick to review as possible. There were still enough of them to eat a large share of analyst time. Rob wanted that capacity back so his team could spend it on the complex, strategic work of the FIU.
Zepz started where the analysis was hardest. At Zepz, even a basic transaction monitoring alert often means asking the customer for more information. That might be a self-declared answer, like their relationship to the recipient, or evidence, like a source of funds document. Analysts then ran transaction and customer analysis and wrote up their findings. The process averaged more than 60 minutes per alert, with a lot of number crunching and plenty of room for error.
So Zepz pointed its first agent at the hardest part of that simple alert. The agent checks the authenticity of the customer's document, pulls out the relevant information and completes the transaction analysis. Analysts now spend their time deciding the case instead of gathering data and switching between screens. Zaheer said the main gain was quality, and shorter handling time came with it.
One use case at a time
Both teams chose a specific, structured use case over giving everyone open access to a general AI tool. Rob tried giving everyone Claude access at Rippling, but the return on token spend wasn't there when the work wasn't tied to a clear task. His test is simple: know what the agent is for. Is it cutting time per alert? Is it gathering data so the analyst's review is more structured?
Rob also pointed out what analysts get from it. Nobody enjoys copying and pasting between systems. When the agent fills in the information, the analyst's job becomes checking that information and confirming or rejecting the proposed disposition.
Zaheer added a nuance. The agent work at Zepz is rigid and structured, but first-line managers do have AI connected to the company's data lake for exploratory questions. One manager used it to check whether customers responded to requests for information at different rates depending on their language, which could point to communication gaps. Rippling does something similar with its own internal Rippling AI, which the financial crime team uses to pull customer data and compare it with what shows up in transaction monitoring.
The pattern is simple: build an agent for work that is repeatable and high volume, and use general AI tools for one-off exploration. For more on how teams choose a first use case, see where to start with agentic AI in AML.
Testing took longer than building
Zaheer said testing the agent took Zepz far longer than developing it. Both teams tested the same way: they put the agent's work next to their best human work and didn't go live until it matched.
Zepz benchmarked against high-quality cases. The team pulled alerts that level one analysts had already worked and that had scored high in quality review. They compared the agent's data analysis on those alerts with the human analysis, and set a minimum number of cases and a minimum accuracy rate for each regulated jurisdiction. Every test result was documented.
Rippling ran side-by-side reviews for about six months. The agent and an analyst reviewed the same test alerts, and the team compared the outputs. Where the agent missed something, they refined the context and got more specific about what each task should look for. They repeated that until every task in the agent hit the standard they needed. Rob built this into a formal AI governance program for the financial crimes team, covering both pre-launch testing and ongoing monitoring.
The testing also exposed gaps in their own processes. Tyler noted that when Unit21 builds a custom agent, the client's SOP is often out of date, with key steps living only in an experienced analyst's head. An agent follows the SOP as written, so anything missing turns into a miss. Zaheer called building the agent a forcing function: Zepz rewrote its processes to remove ambiguity, and the team now follows them more consistently whether or not an agent is involved.
Proving it every day after go-live
Neither team treats go-live as the finish line. Rippling has no auto-closing: the agent fills in the information and proposes a determination, and a human analyst checks it and decides whether to agree. Between them, Rob and Zaheer described four layers of ongoing checks.
- QC on the agent's work. Zepz updated its first-line QC so reviewers sampling alerts also look at what the agent did in its data analysis. Rippling's QA team does monthly reviews comparing the agent's inputs with the human's, and flags where the analyst had to correct a data point.
- A record of every disagreement. When a Rippling analyst disagrees with the agent's proposed determination, the disagreement and the reason are logged. QA checks whether the human or the agent got it right, and the findings go back to Unit21 to refine the agent's context. Zepz runs the same loop, sending what reviewers find to the team that maintains the agent.
- Drift testing every quarter. Rob's data scientists run a fixed set of similar transaction alerts through the agent each quarter and check that the results still hold. If the agent starts reaching different conclusions on the same kind of alert, they investigate why.
- An independent QA agent. Rob built a quality assurance agent inside Rippling AI that reviews completed alerts and KYC files. Rob says it reviews 200 to 300 percent more work than his team could cover by sampling. He built it on different models than the ones doing the investigation so it isn't grading its own homework. Zaheer's team is exploring the same idea: an AI quality score on every alert as another layer of defense alongside human QC, not a replacement for it.
Zaheer was struck by how similar the two setups had become even though the teams started in such different places.
Do rules survive?
Both said yes, for now. Rules and agents do different jobs. Rob pointed out that at Rippling the rules sit separately: they trigger the alerts, and the agent reviews what was flagged. The agent never decides what gets alerted, so it can't suppress suspicious activity.
Where they differ is what comes next.
Rob wants agents looking for what the rules miss. Rules have to be rigid and narrow. Instead of writing 35 rules to cover every typology, he would like agents scanning transaction data in the background for shifts in behavior, such as a group of entities onboarded in one jurisdiction suddenly sending money to another. Anything unusual would surface as an alert for a human to review. He expects that within 12 to 18 months. In his words, the data is where the value is.
Zaheer sees AI as one more layer of defense. Borrowing from fraud, where Zepz runs analytics, real-time alerting and several controls at each level, he expects AI to add detection layers rather than replace rule sets. Zepz is already running one-off exercises where AI reviews a specific typology across transaction data, account data and call data. He also sees value in using AI to calibrate rules, with the caveat that calibration needs heavy documentation.
Tyler agreed with both and raised a third question: how do teams decide which rules to deploy in the first place? Most of the industry's AI work today is in investigation. Detection is the next frontier.
Advice for teams turning on their first agent
Tyler closed by asking what they would tell a team deploying its first agent in the next few months.
- Know exactly what you're solving for. Rob said not to deploy an agent just because everyone else is. Decide whether the goal is faster data collection, a more structured review or something else, and build for that.
- Get your process clear. Zaheer said an agent follows your process exactly, so the process needs to be unambiguous before you build.
- Fix your data. Zepz had to fix a lot of its data while building the agent. Know your data problems going in.
- Plan for more testing than you expect. Both said to have governance and a testing framework in place before you deploy. That includes regression testing against well-known alerts and a clear bar for accuracy. Our guide to evaluating AI agents for fraud and AML covers what to test.
- Make sure you can explain every decision. Rob's test is whether he could read one of the agent's reviews and understand how it reached its conclusion.
- Keep testing after launch. Models drift. As Rob put it, upfront testing is not one and done, no matter how rigorous it was.
The two teams started in different places, one with the highest-volume alerts and one with the hardest analysis. Both got to production the same way: test against your best people before launch, and keep proving it every day after.
For the groundwork that comes before this stage, watch the session with Kinecta and Crypto.com, and read what 1M live AI alerts tell us about how fincrime teams adopt AI.
Frequently asked questions
How long does it take to test an AI agent before going live?
For Rippling, about six months of side-by-side reviews for transaction monitoring alerts. Zepz said testing took far longer than building the agent. Both compared the agent's work with their best human work and set a clear accuracy bar before launch.
How do teams monitor AI agent quality after go-live?
Rippling and Zepz use four layers: QC that reviews the agent's work, a log of every human disagreement with the agent, quarterly drift testing on a fixed set of alerts, and an independent QA agent built on different models.
Do AI agents replace transaction monitoring rules?
Not for now. At Rippling, rules trigger the alerts and the agent reviews what was flagged, so the agent cannot suppress suspicious activity. Both leaders expect AI to add detection layers alongside rules rather than replace them.
Should an AI agent auto-close AML alerts?
Rippling does not auto-close. The agent gathers the information and proposes a determination, and a human analyst reviews it and decides whether to agree.
This session was part of The AI Files, Unit21's virtual summit on how fraud and AML teams are putting AI to work.
Gal Perelman is the Product Marketing Lead at Unit21, where she spearheads go-to-market strategies for AI-driven risk and compliance solutions. With over a decade of experience in the fintech and fraud sectors, she has led high-impact launches for products like Watchlist Screening and AI Rule Recommendations.
Previously, Gal held marketing leadership roles at Design Pickle, Sightfull, and Lusha. She holds a Master’s degree from American University and a Bachelor’s from UCLA, and is dedicated to helping banks and fintechs navigate complex regulatory landscapes through innovative technology.
