Audit automation software: how to test it before your firm relies on it

01What does audit automation software mean for an audit firm?

The phrase covers very different products. Search results for it mix external audit platforms with internal audit management, compliance automation for frameworks such as SOC 2, document-matching add-ins and general data analytics. A firm buying for statutory audit work needs to know which kind it is looking at before it compares anything.

For external audit, the useful question is narrow: which part of an engagement does the tool prepare, and what record does it leave for the auditor? Some tools import and test data. Some match documents to ledger lines. Some hold the whole engagement file, from planning to sign-off. Newer tools use AI agents to prepare the work. The more of the file a tool prepares, the more your evaluation should look at review, change control and exit.

02What do the standards expect when a firm buys audit software?

Where a firm applies the IAASB's quality management standard, ISQM 1, buying a tool does not move responsibility to the supplier. ISQM 1 sets a quality objective that appropriate technological resources are obtained or developed, implemented, maintained and used (paragraph 32(f)), and another that resources from service providers are appropriate for use (paragraph 32(h)). Its examples of a service provider's resource include commercial software used to perform audit engagements, and it states that the firm remains responsible for its system of quality management.

ISQM 1 also lists matters a firm may consider for an IT tool (paragraph A100): complete and appropriate data inputs, confidentiality of the data, whether the tool operates as designed, whether its outputs achieve their purpose, the general IT controls behind it, the skills and training users need, and procedures that set out how it works. Paragraph A101 allows a firm to prohibit a tool, or some of its features, until it has been approved for use. That is a ready-made evaluation plan.

ISA 230 sets the bar for the working paper itself. Documentation must let an experienced auditor with no previous connection to the audit understand the procedures performed, the results and evidence, and the significant matters and judgements (paragraph 8). It must record who performed the work and when, and who reviewed it, when and to what extent (paragraph 9). Software that hides those facts behind a completion tick makes the requirement harder to meet, whatever its other strengths.

Guidance on audit technology is moving quickly, so check the status and date of anything you rely on. The table separates standards from guidance and from proposals. Which standards bind your firm, and from when, depends on the rules adopted where you practise. Confirm that with your technical lead.

03How do you check the data and the population a tool tested?

Most automated testing starts with an import. Ask which records were included, which were excluded and how the imported population reconciles to the client's system. Ask to see the mappings and any transformations: changed account codes, merged fields, converted currencies or reformatted dates. A reviewer should be able to see the population actually tested, not assume it is the file someone uploaded earlier.

Where the data comes from the client's own systems, ISA 500 already requires the auditor to evaluate whether the information is reliable enough for the purpose, including evidence about its accuracy and completeness where necessary (paragraph 9). A tool that tests every line of an incomplete ledger has tested an incomplete ledger. Whole-population testing helps only when the population is right.

Then break the input on purpose. Use a fictional file and note what the tool does with each fault below. A good response is visible: the item is flagged, held back or sent to a person. A poor response is silence.

• A missing mandatory field, such as a blank date or amount.
• An exact duplicate transaction.
• An entry dated outside the period under audit.
• A file in the wrong format, or with a broken column header.

04What happens when the tool finds, or misses, an exception?

Plant a discrepancy in the demonstration data and follow it. Does it appear as an exception? Who receives it? Can the client's answer or a replacement document be attached to that item? Does the item stay open until a person with the right role accepts the explanation? Ask the supplier to show what happens when the explanation is not good enough.

A clean dashboard proves nothing until you have planted an exception yourself. The absence of alerts only tells you that the tool's rules found nothing. You still need to know what those rules test and what they cannot see. IESBA describes the opposite habit as automation bias: favouring what technology produced even when other information raises questions.

For tools that use generative or agentic AI, the FRC's March 2026 guidance gives a useful frame. It is UK guidance written for firms' central technical teams, and the FRC says it codifies good practice; it does not change who is accountable. It sorts the risks to audit quality into three categories, and each one turns into a demonstration question.

05Can a reviewer follow the working paper, its changes and its archive?

Open a prepared working paper as the reviewer would. Trace one result to its source, the procedure and the exception history. Check that it shows who prepared it and when, and who reviewed it, when and how far. Those are ISA 230 paragraph 9 facts, not extras. Comments and corrections should still make sense after a new version is created.

Then change something after review. Replace a source document or alter an input the procedure relied on. Ask how the tool flags the affected work, what it records about the change and how it sends the work back for review. A feature name in a sales deck does not settle this. The behaviour on screen does.

Finish at the end of the engagement. ISA 230 requires the final file to be assembled on a timely basis after the auditor's report, and the standard's explanatory guidance puts an appropriate limit at ordinarily not more than 60 days. After assembly, documentation must not be deleted before its retention period ends, which that guidance says is ordinarily at least five years from the date of the auditor's report, or of the group auditor's report if that is later. Any later change needs its reason, and a record of when it was made and reviewed and by whom. Ask how the tool locks a completed file, logs later changes and lets you retrieve an archived file years from now, including after the contract ends.

Ask the same about updates. ISQM 1 suggests considering how often a provider updates a tool and how it handles confidentiality of data (paragraph A107). The FRC treats version control of AI components as a mitigation, because an updated model can behave differently. You want to know how the supplier tells you before a change reaches live engagements.

06How do you score a supplier demonstration fairly?

Score evidence, not impressions. For each criterion, record one of three levels: ‘Evidence seen’, ‘Claimed, not shown’ or ‘Not available’. The scorecard below gives each criterion a weight from 1 to 3 and counts a claim at half its weight. The result is the share of weighted points you have actually seen, plus a list of gaps with the heaviest first.

A total can hide a problem. One weight-3 gap, such as no way to export a completed file, can matter more than several minor points. Decide each heavy gap on its own: ask for more evidence, test it in a pilot, accept it with a recorded reason, or stop. Keep the test file, what you observed and the follow-up owner in your notes.

Use the same scorecard and the same test file for every supplier. Keep the procurement paperwork alongside it: service terms, data processing terms, subprocessors, hosting, retention and exit. Ask for evidence behind any claimed assurance report or certification. A statement about where data is stored is not evidence of how it is protected.

07Where does Vacei fit?

Vacei describes itself as the operating system for accounting and audit firms. Its published audit workflow covers planning, testing by specialist agents, working papers with the evidence attached, and review and sign-off. VEE, the AI inside Vacei, prepares the audit plan for the auditor's approval, specialist agents run the procedures, and the auditor reviews every working paper and signs. Vacei's pages also describe preparer, reviewer and partner sign-off on every file, and a record of every step VEE takes.

Those are Vacei's own descriptions, not results of a test, and this guide does not rate Vacei. Its firm demo is a 30-minute walkthrough of the operating system, shown on sample data.

Vacei also offers outsourced audit support, which is a separate service: fieldwork and working papers prepared in your methodology for your review, with engagement acceptance, professional judgement, the opinion and the signature staying with your firm. Be clear whether a demonstration concerns the software, the service or both, because scope and responsibilities differ.