AI Fraud Detection in Finance: Rules vs Models vs Agents, and What Each One Misses

Every finance team evaluating fraud detection is shown the same ladder: you have rules today, machine learning is the upgrade, agents are the future. It is a tidy story and it is the wrong shape. Rules, models, and agents are not three rungs. They are three different things with three different failure surfaces, and a team that replaces one with another usually trades a blind spot it understood for one it does not.

The useful question is not which is best. It is which of the three your current gap sits in — because the answer determines whether you need a config change, a data project, or a new operating procedure, and those are wildly different amounts of work.

The three approaches, stated precisely

Rules are deterministic conditions written by a person. Block payment if the bank account changed within thirty days of the invoice. Hold anything over the approval limit from a vendor created in the last quarter. The logic is legible, the outcome is reproducible, and someone can be named as its author.

Models are statistical functions fitted to historical data that return a score rather than a verdict. They learn what normal looks like across many variables at once and flag distance from it. Nobody writes the condition; the condition is inferred, and it is usually not expressible in a sentence.

Agents are language-model systems that can take actions — read the invoice, pull the vendor's history, open the email thread the invoice arrived on, query a registry, compare a bank-change request against prior correspondence — and then produce a written assessment. They do the thing an experienced analyst does when a case looks odd: go and look.

These are not competing implementations of the same job. Rules encode a decision you have already made. Models rank cases by how unusual they are. Agents investigate. Confusing the three is how teams end up asking a model to explain itself and an agent to be consistent — neither of which is a thing it does.

What rules miss

Rules fail in a specific and predictable way: they catch the fraud that has already happened somewhere, to someone, and been written down.

The deeper problem is that a rule publishes its own threshold. Set a control at $5,000 and you have told anyone inside the process exactly what size invoice avoids scrutiny. Every experienced fraud examiner has seen the resulting shape — a cluster of amounts sitting just underneath a limit that nobody outside AP was supposed to know. A rule cannot see this pattern, because each individual invoice passes the rule it was built to pass.

The second failure is organisational rather than technical. Rule sets only grow. Each incident adds one, nobody is willing to be the person who deleted the control that would have caught the next one, and after four years the AP team receives more alerts than it can read. At that point the control has inverted: the volume of flags is itself the cover. Fraud detection that produces an unreviewable queue is not detection.

What rules are unmatched at is evidence. When an auditor asks why a payment was stopped, a rule answers completely — the condition, the version in force that day, the values that triggered it, the person who authorised the exception. No other approach comes close, and this is why rules should never be removed from the stack, only reduced in number.

What models miss

Models are pitched as the answer to novel fraud, and they genuinely see combinations no analyst would hold in their head. But three things break them in finance specifically.

The labels do not exist. Supervised fraud detection needs examples of confirmed fraud. Most finance teams have a handful of confirmed cases across several years, many of them discovered by a tip rather than by a control, and the far larger population of fraud that was never detected is by definition absent from the training data. A model trained on what you caught learns to catch what you already caught. Unsupervised anomaly detection avoids the labelling problem and inherits a worse one: it flags everything unusual, and in accounts payable most unusual things are legitimate — a genuine one-off capital purchase, a supplier's first invoice after a merger, a year-end true-up.

Normal moves. A model fitted on last year's payment behaviour degrades quietly through a reorganisation, a new ERP, a change in payment terms, or a seasonal shift. Nothing announces the degradation. Detection rates fall and the dashboard looks calmer, which is indistinguishable from things getting better.

The feedback loop closes on itself. When analysts review flagged cases and their dispositions become training labels, the model learns the analysts' habits — including the ones that were wrong. A vendor that reviewers have historically waved through becomes a vendor the model scores as safe.

And the score is not an explanation. When a payment is held because a model returned 0.83, the finance lead who has to call the supplier has nothing to say. Storing the feature values at scoring time is what makes the decision reconstructable six months later, and it is the step most implementations skip because nothing appears to break when they do.

What agents miss

Agents close a gap the other two cannot touch. The Lithuanian scheme that took over $100 million from Google and Facebook worked because the invoices referenced real supplier relationships and real contract numbers. No threshold was crossed. Nothing was statistically unusual. The only way to catch it was to go and check whether the entity sending the invoice was the entity in the contract — which is investigation, not scoring.

That capability comes with three costs that are structurally different from anything in the first two approaches.

Non-determinism. Run the same case twice and you can get two assessments. For an investigation that assembles evidence, this is tolerable. For a decision that stops a payment, it is not — and it is the reason an agent should produce a case file rather than a verdict.

The documents are attacker-controlled input. An agent that reads invoice PDFs and supplier emails is consuming text written by whoever wants the payment made. Instructions hidden in that text — in white-on-white body copy, in a PDF comment field, in the quoted history of an email chain — are read with the same weight as the analyst's own instructions. Prompt injection is the top entry in the OWASP guidance for LLM applications for exactly this reason, and invoice fraud is the case where the incentive is most direct. Any agent in this position needs its instructions separated from document content, its tools scoped to read-only, and no ability to release a payment.

Confident narrative. An agent asked whether an invoice is fraudulent will produce a well-argued answer either way. The failure is not that it is wrong; it is that wrong output is indistinguishable in tone from right output, so reviewers calibrate to the prose rather than the evidence. Requiring the agent to cite the record it drew each claim from — and rejecting assessments that cite nothing — restores the distinction.

The layering that actually works

The three approaches map cleanly onto a question that has nothing to do with AI: how reversible is the action?

Rules own the irreversible decisions. A small set — five to fifteen, not two hundred — covering the things that must never happen without a human: a bank detail change paired with a payment inside the same window, a first payment to a vendor created this month above a threshold, a duplicate of an already-settled invoice. Deterministic, versioned, and few enough that every alert is read.

Models own the ordering. Not the verdict — the queue. Given a day's payment run and a review capacity of thirty invoices, a model's job is to decide which thirty. This uses exactly the capability models have, and it is robust to the failure they suffer from: a mis-ranked queue costs review attention, not a blocked supplier. Drift shows up as a queue that stops surfacing anything interesting, which a reviewer notices.

Agents own the case file. For each item the queue surfaces, the agent assembles what an analyst would have spent forty minutes gathering: the vendor's history, the change log on the bank details, the email the invoice arrived on, the matching contract, the discrepancies it found and where each came from. It hands that to a person. It does not approve, and it does not block.

This is the calibration question we run at every layer of an operation: which decisions get delegated to the agent, which get surfaced for approval, and which stay entirely human. Fraud is the clearest case for the middle answer. The gathering is delegable and the deciding is not, because the cost of a wrong block lands on a supplier relationship and the cost of a wrong release lands on the balance sheet — and neither is a cost an agent can be held to.

The same logic governs AI invoice processing generally: the boundary is not between simple and complex work, it is between reversible and irreversible.

Finding your own gap

Three questions, in order.

Are your alerts being read? Pull last month's flags and count how many received a documented disposition. If the answer is well under all of them, your gap is rule sprawl, and adding a model to a queue nobody reads makes it longer. Fix the rules first — this is a week of work, not a project.

Do you know what you missed? Look at how your last confirmed fraud was found. If it was a tip, a supplier call, or a bank reversal rather than a control, your controls are not covering the shape of fraud you actually face, and the fix is a ranking layer that surfaces the unusual rather than the prohibited.

Are reviewers deciding without the evidence? Time how long it takes an analyst to assemble the context for one flagged invoice. If it is over half an hour, your gap is investigation capacity, and that is the agent layer — the only one of the three that reduces the human minutes per case rather than increasing the number of cases.

Most teams find their gap in the first question and are being sold the third.

What to ask a vendor

The standard question is "how accurate is it," and the answer is unfalsifiable — accuracy against a test set that does not resemble your invoice population tells you nothing. Ask instead what happens when it is wrong.

• When a legitimate payment is held, who unblocks it, how long does that take, and does the supplier hear from a person?
• When something is missed, what record exists that it was seen and scored? Can you reconstruct the inputs six months later?
• Who owns the false-positive volume — your team or theirs? If it is yours, the running cost is headcount you have not budgeted.
• For a model: what triggers retraining, and how would you know it had drifted before detection rates fell?
• For an agent: are its instructions separated from document content, are its tools read-only, and has it been tested against documents containing injected instructions?
• What is the per-case cost at your volume, not at a demo volume?

A vendor who answers these fluently has run the system somewhere real. One who redirects to accuracy has not.

The bottom line

Rules catch what you have already seen and leave perfect evidence. Models catch what is unusual and cannot explain themselves. Agents catch what needs looking into and are inconsistent by construction. Each covers the others' blind spot, which is why the answer is a stack rather than a choice — and why the sequence matters: fix the rule sprawl, then rank the queue, then automate the gathering.

The teams that get this wrong are not the ones that picked the wrong technology. They are the ones that let a probabilistic system make an irreversible decision, and then discovered they could not explain to a supplier, an auditor, or a board why the payment stopped. Understanding why AI projects fail in operations usually comes back to that single boundary.

FAQ

Should we replace our rules engine with a machine learning model?

No. Reduce the rule set and keep it. Rules are the only layer that produces a complete, reproducible explanation for a stopped payment, which is what an auditor and a supplier both require. The realistic move is to cut a sprawling rule set down to the small number of conditions covering irreversible actions, then add a model above it to rank everything the rules let through. A model replacing rules leaves you unable to answer why a specific payment was held.

What can an AI agent do in fraud detection that a model cannot?

An agent can go and check. A model scores an invoice against patterns in the data it was given; an agent can open the email the invoice arrived on, pull the vendor's contract, compare a bank-detail change to prior correspondence, and check whether the invoicing entity matches the contracted one. That is the class of fraud where nothing is statistically unusual — the amounts, timing, and vendor all look right because the fraud impersonates a real relationship. The trade-off is consistency: the agent should assemble evidence for a human decision, not make the decision.

How do we stop an agent from being manipulated by the documents it reads?

Treat every invoice, attachment, and email as untrusted input, because the person who wants the payment made wrote it. Three controls do most of the work: keep the agent's instructions structurally separate from document content rather than concatenated with it, scope its tools to read-only so no path exists from a document to a released payment, and include documents carrying injected instructions in your test suite so you find out in evaluation rather than in production. An agent that can only produce a written case file has a hard ceiling on what a successful injection achieves.