Building Trust in AI for Business-Critical Finance: The Evidence That Moves an Agent From Demo to the Close

The short version: Nobody trusts an AI agent because it demoed well. Trust in finance is an accumulated record — provenance you can re-derive, disagreements you classified rather than counted, and a blast radius small enough that being wrong was survivable. The work is building the evidence, and it takes periods, not weeks.

The demo goes well. Someone asks a question in plain English, the agent queries NetSuite, and thirty seconds later there's a number that would have taken an analyst an afternoon. The room is impressed. The controller is impressed.

Then somebody asks the only question that matters: would you let it near the close?

And the answer is no — not because anyone doubts the demo, but because the demo proved capability, and the close requires reliability, and those are different properties measured in different ways. A capability is demonstrated once. Reliability is demonstrated over repetitions, under conditions nobody chose, with the failures visible.

This is where most finance AI programs stall. Not at "can it do this" — that was settled months ago — but at the crossing from interesting to depended upon in functions where an error costs real money. What follows is the practitioner's version of how that crossing actually gets made: what evidence counts, what to instrument before you start, and how to design the workflow so the inevitable wrong answer is a bad afternoon instead of a restated quarter.

Trust is a record, not a decision

Nobody in a finance department decides to trust the new senior accountant. Over a couple of closes they hand them progressively more consequential work, check it, notice the checks keep coming back clean, and find one day that they've stopped checking everything. Trust was the residue of a process.

Agents are no different, with two adjustments.

The first is that the record has to be deliberately produced. A person's work leaves a natural trail — a workpaper, an email thread, a conversation where they explain their reasoning. An agent produces an answer and, unless you built for it, nothing else. The trail is not a byproduct; it's a feature you specify.

The second is asymmetry. Trust accrues slowly and collapses at once. A finance team that has quietly relied on an agent for six months will stop using it entirely after one visibly wrong number in a board deck — and they'll be right to, because they have no way to know whether that was the first one or the first one they caught. That asymmetry is the whole argument for provenance: the difference between an incident you can bound ("here's the query that was wrong, here are the four outputs it touched") and an incident you can't ("we don't know what else it got wrong") is the difference between a fixable problem and the end of the program.

It's worth naming that the vendors know this is the open question. BlackLine's April 2026 agentic-operations launch was framed explicitly around the industry's unsolved "AI governance and trust gap." When the incumbent's marketing leads with the problem rather than the solution, you're looking at a live constraint, not a solved one.

You cannot grade an answer by reading it

The instinct is to build trust by checking outputs. Run the agent, check the number, repeat until the checks get boring. This fails for a reason specific to how these systems go wrong.

An early adopter of NetSuite's AI Connector reported revenue figures off by more than 2× — not garbled, not obviously broken, just wrong, from the ordinary causes: a join that double-counted, a date filter that caught the wrong side of a boundary, missing intercompany eliminations (as documented by PS Global Consulting). Oracle's own FAQ for the feature says it plainly: AI may hallucinate, always validate results against source data.

The important part isn't that it was wrong. It's that the wrongness was invisible in the output. A number that's off by 2× still has the right shape, the right currency, the right period label, and a confident sentence around it. There is no reading of that answer, however careful, that reveals the problem. The only thing that reveals it is looking at how the number was produced.

So the unit of review is not the answer. It's the answer plus the path to it — which is why provenance has to be built in before the trust-building starts, not bolted on after the first incident.

Provenance as a first-class output

Treat every agent output as a workpaper with the number attached, not a number with some logging behind it. A usable provenance record for NetSuite work carries:

• The question as the agent understood it — its own restatement. Scope drift shows up here first, and cheaply.
• The exact query, verbatim. The SuiteQL that ran, not a description of it. "Queried the GL for Q2" is not provenance.
• The row count. A summary without a count hides an empty result set, which is the failure that reports itself as good news.
• Period and state — which accounting period, and whether it was closed when the query ran. An analysis over an open period has a shelf life and should say so.
• Scope — the subsidiary set enumerated (not "all"), the currency and the rate basis, the inclusion and exclusion rules applied.
• Identity and timestamp — which user identity the query ran as, and when. Not a shared integration token: an agent that runs as svc_integration inherits every permission anyone ever granted it and produces an audit trail that names nobody.
• Assumptions and unresolved items — what the agent decided on its own, and what it couldn't determine.

Two implementation notes that teams learn the hard way.

Retain it yourself. NetSuite's first-party AI Connector Service does keep an execution log — OAuth 2.0 with PKCE, role-scoped, logged — but the documented production retention is 21 days. Twenty-one days is shorter than the interval at which anyone will ask you to explain a number. Whatever the platform retains, keep your own copy on a schedule that matches your audit cycle, not the vendor's.

Provenance is a design constraint, not a report. If the record is generated after the fact by asking the model to explain what it did, you have a plausible narrative rather than a record — the same failure mode as the answer itself, one level up. The query log, the row counts, and the identity have to be captured by the plumbing, not narrated by the agent.

The vendors doing this well make it structural. Vic.ai logs every decision with a confidence score and the rationale behind it; Rillet's in-ledger agents work propose-and-approve with the audit trail as part of the mechanism rather than a reporting feature. The pattern to copy is that the record exists whether or not anyone asks for it.

What "explainable" has to mean against an ERP

"Explainable AI" imported from the model-governance literature means something almost useless here — feature attributions, saliency, why the model weighted what it weighted. No controller has ever wanted that. Against an ERP, explainability reduces to three tests, all of which a finance person can run without knowing anything about models:

1. Re-derivation. Hand the provenance record to someone else. Can they run the same query against the same period and get the same number? If yes, the answer is explained in the only sense that matters. If the query isn't in the record, or the period has moved under it, it isn't.
2. Drill-through. Does every figure trace to the NetSuite internal IDs of the transactions that compose it? A total that can't be decomposed into records a reviewer can open is an assertion, not a finding. This is also the fastest way to catch the plausible join — a decomposition with too many lines is visible immediately.
3. Assumption disclosure in the output. Period, subsidiary set, currency basis, and inclusion rules stated where the reader sees them, not buried in a prompt file that only the person who wrote it has read.

And one anti-test: a confidence score is not an explanation. It's a number the model generated about itself, useful for routing and worthless as evidence. Confidence thresholds do real work — the AP vendors have made them the core of their autonomy design, with Vic.ai auto-posting above a stated threshold and routing everything below it — but they're a triage mechanism. The evidence is the query and the rows.

The trust ladder is earned, not configured

The rungs — read, propose, act with approval, act unattended — are the standard autonomy ladder, and where to draw the line for a given workflow is covered in agentic ERP operations. The part that gets skipped is that each rung has an exit criterion, and the criterion is a measurement, not a calendar date. Nobody graduates because it's been a month.

Rung	Agent does	You're measuring	Evidence to advance
Read	Answers questions, produces analysis nobody acts on yet	Whether the numbers re-derive	A sample of outputs re-derived independently, including ones that agreed. Materially fewer provenance defects than the prior period
Propose	Drafts the entry, the commentary, the follow-up — a human sends it	Edit rate and what kind of edits	Edits shifting from substance (wrong number, wrong scope) to style. Substantive edits still arriving means you're at the wrong rung
Act, bounded	Executes reversible actions inside a fence; everything else queues	Escalation quality and reversal frequency	A full period with no reversal that a human had to discover. Escalations landing on real cases, not noise
Act, unattended	Runs the loop without anyone watching that morning	Whether anyone would notice a failure	A deliberately injected failure — a broken filter, a stale field — caught by the monitoring before a person caught it

That last row is the one teams skip, and it's the only one that tests the thing you actually care about. Unattended operation isn't a statement about the agent's accuracy; it's a statement that your detection is good enough to survive the agent being wrong while nobody is looking. If you've never tested detection, you haven't earned the rung — you've just stopped watching.

Two directional rules for the climb. Advance one workflow at a time, so a setback is diagnosable. And demotion has to be as mechanical as promotion — if the criterion for a rung stops being met, the workflow drops back, automatically, without a meeting. A ladder you can only climb isn't a control.

Running a shadow period so it produces evidence

Shadow mode — the agent runs alongside the existing process, changing nothing — is the standard advice, and most teams run it in a way that produces a warm feeling instead of a record.

Measure in periods, not weeks. Month-end generates the failure modes worth seeing: cutoff, accruals, intercompany, the close calendar, the reclass someone always does by hand. A shadow run that never crosses a period boundary has tested the easy case. Two full periods is a realistic floor for a recurring deliverable.

Classify disagreements; don't count them. Every difference between the agent's answer and the human's gets one of four labels:

• Agent wrong — the model or the query got it wrong. Fix and note the class.
• Human wrong — the existing process was wrong, which happens more than teams expect and is the single most valuable output a shadow run produces.
• Both defensible — a judgment call, usually an undocumented convention. Write the convention down; that's a permanent asset regardless of what happens to the agent.
• Data wrong — a mis-mapped account, a renamed custom field, a subsidiary missing from a hardcoded list. This is data foundation work, and it's the most common label in the first period.

An accuracy percentage tells you nothing actionable. A distribution across those four labels tells you exactly what to fix and in what order — and the trend across periods is the trust evidence, because a program where data wrong dominates period one and shrinks in period two is visibly working.

Sample the agreements too. Right answer by the wrong path is the failure this catches — the number matched because two errors canceled, or because the query was scoped by accident to the same population. Pull a handful of matching runs each period and re-derive them from the provenance record. A team that only investigates mismatches is building confidence in a metric rather than in a system.

Log the near-misses. Cases where the agent was right but the reviewer had to think about it are the highest-information events in the whole run; they're where the conventions live. They're also the first thing to disappear once the run stops feeling novel, so make it a field, not a habit.

Where the agent's work is checked by another agent rather than a person, the same discipline applies to the internal disagreement rate — see staffing an agent team for how that review is structured.

Blast radius and reversibility

Everything above is about being right. This section is about the other case, and it's the one that determines whether a finance team can extend trust at all — because they aren't asking "will it be wrong," they're asking "what happens when it is."

Rank every action the agent can take by reversibility, and make that ranking, not model quality, the thing that governs unattended authority:

Tier	Examples in NetSuite	Reversal cost	Unattended?
No ledger effect	Add a note, attach evidence to a close task, update a follow-up date, tag a record	Undo it; nothing downstream moved	Yes, once bounded
Staged	Draft transaction, queued JE, proposed match — pending human approval	Delete the draft	Yes — the approval is the control
Posted, open period	JE posted into a period still open	Reversal entry; leaves a visible trail	Only with a named owner and same-day review
Irreversible	Payment released, period closed, external filing, an email to a lender or the board	Not undoable — only explainable	No

Note that the last tier includes things that are technically read-only. A number generated by an agent and pasted into a covenant certificate is irreversible the moment it's sent, regardless of what the agent's NetSuite permissions were. Blast radius is measured at the recipient, not at the API. This is the tier that catches teams who thought "read-only" was the whole safety argument.

Inside the reversible tiers, five bounds do the work:

• Per-run caps. A dollar ceiling and a record-count ceiling. An agent that can touch 12 records can't touch 12,000 because of a filter that stopped matching.
• Scope fences. One subsidiary, one account range, one transaction type. Widen deliberately, one dimension at a time.
• Its own identity. Per-agent, per-user credentials rather than a shared token, so actions are attributable and revocable in one move. This is also the difference between an incident you can scope and one you can't.
• Idempotency. A retried loop must not act twice. This is unglamorous plumbing and it's where most real-world damage originates — not from a wrong decision, but from a right decision executed four times.
• A kill switch a finance person can pull. If stopping the agent requires an engineer and a ticket, the effective response time is the length of a Slack thread. Trust depends on the brake being reachable by the person who notices.

The useful reframe: you're not certifying that the agent won't be wrong. You're building a system in which being wrong is cheap and visible, and then letting the record accumulate.

The incident is part of the record

A finance team that has never seen its agent fail does not trust it. They have an untested assumption, which feels like trust right up until the moment it isn't.

So plan the first visible failure as an event with a protocol, before it happens:

1. Define an incident in advance. A wrong number that reached a deliverable. An action taken outside its fence. A silent zero that shipped as "no exceptions." Vague definitions produce arguments at exactly the wrong time.
2. Scope it from the provenance, immediately. Which query, which period, which outputs downstream. This is the moment the record earns its entire cost. Without it, the honest answer to "what else did it get wrong?" is "we don't know," and that answer ends programs.
3. Correct through the normal process. Whatever a human error of the same size would go through. Agent errors that get a special path teach everyone that the agent lives outside the control environment.
4. Publish the miss. Internally, in the same place the wins get shared. A visible error log is a trust asset — it's the evidence that detection works. A program with no recorded misses isn't a program without errors; it's a program without detection.
5. Change something structural. A new check, a tightened fence, a demotion down a rung. "We'll be more careful" is not a change.

Every mature control environment in finance is built on the same shape: something went wrong, it was caught, the catch was documented, the process changed. There's no reason for agents to be exempt — and, run this way, the first incident increases trust, because it demonstrates that the detection layer is real. That's the counterintuitive part worth internalizing: the goal isn't an agent that has never failed. It's an agent whose failures have all been caught, bounded, and written down.

Kognitos does a version of this in product — exceptions escalate to a human, and the human's resolution becomes a permanent runbook step. The generalizable idea is that each failure should leave the system stronger in a way you can point at.

What your auditor will ask

Practitioner perspective, not audit advice — but the questions are predictable enough to design for, and a team that can answer them has usually built something worth trusting regardless of who's asking.

• Completeness. How do you know the agent saw the whole population? Not the records it returned — the ones it should have. This is a query-scope question, answered from the provenance record.
• Who performed the control, and who reviewed it. Per-user identity, review evidence, and separation between the two. A shared token fails this question on the first sentence.
• Consistency across the period. Not that it worked on the day someone watched. That it worked every day, which requires retained logs covering the whole period — see the 21-day retention note above.
• What happened to the exceptions. Every escalation with a resolution, and no silent retries: an agent that fails and reruns until it gets a clean answer is selecting for clean answers, not correct ones.

If you're heading toward a first audit with agents in the workflow, year-round audit readiness covers the surrounding practice.

A quarter to build the record

1. Pick one recurring deliverable with a known-good comparison. It has to be something a person already produces, so there's an answer to disagree with. The AR aging commentary, the flux narrative, a specific reconciliation.
2. Instrument provenance first. Before any trust question, make sure every run captures query, row count, period and state, scope, identity, timestamp, assumptions. If you can't do this, stop here — everything downstream is unverifiable.
3. Set the escalation thresholds now, in writing. Materiality is the easy one to start from: the variance playbooks treat anything over 25 percent or over $100,000 as critical, which is a workable default until you tune it to your own thresholds. Set them before you have results, so they can't be rationalized afterward.
4. Shadow for two full periods, classifying every disagreement into the four labels and sampling the agreements.
5. Review the log with the people who'd have to trust it. Not a summary — the log. Trust doesn't transfer through a status update; the controller who will sign the deliverable needs to have looked at the disagreements themselves.
6. Advance one rung, on the evidence, with the demotion criterion written down alongside the promotion.
7. Inject a failure before going unattended. Break a filter on purpose. If monitoring doesn't catch it, you've learned the most useful thing of the quarter for the price of an afternoon.

Somewhere in the second period, the character of the conversation changes. People stop asking whether the agent is right and start asking what it found — which is what trust sounds like from the inside, and it arrives as a side effect of the record, never as a decision anyone announced.

The bottom line

Trust in business-critical AI is not a property of the model. It's a property of the system around it: whether the work re-derives, whether the disagreements were classified, whether the failure was cheap, whether the incident was caught and written down.

Build provenance before you build confidence. Climb the ladder on measurements with demotion criteria attached. Rank actions by reversibility and give unattended authority only where reversal is cheap — remembering that the blast radius is measured at the recipient, not at the API. Then run it in shadow long enough for real periods to happen, and let the record do the convincing.

The finance teams that get this right aren't the ones with the most confidence in AI. They're the ones who built enough evidence that confidence stopped being the operative question.

If you want to see what that record looks like in practice — every query visible, every number traceable to the NetSuite records behind it, escalation thresholds set where you set them — request a walkthrough. Bring the deliverable you'd least like to be wrong about; that's the one the evidence has to hold up for.