How accurate AI invoice data extraction is, and how to verify it
The best AI data extraction solutions state accuracy above 99% at field level; Data Alchemy states 99.8%. The problem is that these figures, as published, are not comparable with one another: they depend on the document set they were measured against, on which fields were counted, and on the point in the chain where the measurement was taken. A 99% computed on the header of clean invoices and a 99% computed on header and line items of real scans describe completely different products. The number that actually matters economically is not even per-field accuracy but the share of perfect documents: on a twenty-field invoice, 99% per field yields roughly 82% error-free documents, while 99.8% yields roughly 96% — less than a quarter of the review work. This page explains the six metrics in use, why vendor figures are not comparable, and the eight-step protocol for measuring accuracy on your own documents before choosing.
The six accuracy metrics, and what each one hides
When a vendor states a percentage, the first question is which of these six they are measuring. The differences are not academic: they move the result by tens of points.
| Metric | What it measures | What it hides |
|---|---|---|
| Character-level accuracy | Share of characters recognised correctly against the real text — the complement of the character error rate used in OCR. | A single wrong character in a VAT number makes the field unusable, yet it barely dents this percentage. It is the most generous metric and the least useful for deciding. |
| Field-level accuracy | Share of fields extracted correctly out of the fields expected: invoice number, date, net amount, VAT, VAT number and so on. | It depends entirely on which fields are in the count. If line items are excluded, the number describes the easy part of the document. |
| Line-item accuracy | Share of lines extracted completely and correctly, with code, description, quantity, unit of measure, price and tax rate. | It is technically the hardest metric and the one that weighs most on savings: an invoice with forty lines is forty manual entries avoided, not one. |
| Document-level accuracy | Share of documents in which every expected field is correct: no errors, no omissions. | It does not hide much, which is why it is rarely advertised: it is always noticeably lower than field-level accuracy, purely because probabilities compound. |
| Straight-through processing rate | Share of documents that goes through the entire flow and is posted into the ERP with no human intervention at all. | This is the number that determines real savings. It can be lower than document-level accuracy, because it also includes rejects from failed validations, not only reading errors. |
| Quality of exception flagging | How reliably the system can say "I am not sure about this field": false positives (good documents held) and false negatives (errors passed as valid). | It almost never appears on product sheets, but it is decisive: a 99% system that knows when to doubt is more useful than a 99.5% system that posts errors confidently. |
Why 0.8% of difference is worth a quarter of the work
A document is correct only if all its fields are. Probabilities multiply, so the share of perfect documents falls far faster than per-field accuracy suggests. The table shows the estimated share of documents with no errors at all, assuming errors are independent across fields.
| 99% per field | 99.5% per field | 99.8% per field | 99.9% per field | |
|---|---|---|---|---|
| Document with 10 fields | ~90% | ~95% | ~98% | ~99% |
| Document with 20 fields | ~82% | ~90% | ~96% | ~98% |
| Document with 40 fields | ~67% | ~82% | ~92% | ~96% |
On a twenty-field invoice, moving from 99% to 99.8% per-field accuracy takes documents needing review from 18% to 4%: the residual work drops more than fourfold, not by 0.8%. This is why apparently negligible decimal differences deserve to be taken seriously during evaluation — and why an invoice with many line items, which has forty fields or more, is the real test bed.
Five reasons published figures are not comparable
There is nothing improper about how vendors publish their percentages — ours included. It is simply that everyone measures on their own document set with their own conventions, and there is no shared, verifiable benchmark for extraction from real invoices. Here is what changes from one number to the next.
The document set is different
A sample of native PDFs generated by ERPs produces radically different results from a sample of skewed scans, smartphone photos and digitised faxes. No percentage means anything without knowing what it was computed on.
The fields counted are different
Many published figures cover a narrow set of header fields — supplier, number, date, total — where any modern engine performs very well. Line items, the hard part and the part that generates the saving, are often left out of the count.
The measurement point is different
There is a substantial difference between first-pass accuracy and accuracy after an operator has corrected doubtful cases and the model has learned. The second number is higher and describes a steady-state system, not what you will see in week one.
Rejected documents sometimes do not count
If documents the system refuses to process are excluded from the denominator, the percentage rises but the manual work remains. It is worth asking explicitly how rejects are treated.
Language and tax context matter
An engine trained predominantly on English documents treats Italian VAT numbers, SDI recipient codes, Italian tax rates and delivery note formats as edge cases. On real Italian documents the difference shows, and it appears in no general percentage.
The most cited solutions and the right question to ask each
Rather than repeating other people's percentages — which change over time and depend on test sets nobody can verify — here is each solution's category and the single question that, during evaluation, reveals the most.
| Solution | What it is | The question that reveals most |
|---|---|---|
| Data Alchemy | Italian IDP platform: a dedicated LLM per document model, validation against ERP master records and writing into the ERP. | Ask us for accuracy split between header and line items on your own sample, plus the share of documents posted with no intervention. Those are the two numbers we publish during a trial. |
| ABBYY | Long-established enterprise IDP platform, with configurable document skills and on-premise deployment options. | How much configuration is needed to reach the stated accuracy on our layouts, and what is it worth before that configuration? |
| Amazon Textract | AWS OCR and document analysis service, with dedicated APIs for analysing invoices and receipts. | Does the accuracy refer to text recognition or to structured field extraction, and how does it behave on line-item tables? |
| Nanonets | Document automation platform with models trainable on your own documents. | How many sample documents are needed to reach the stated accuracy, and who labels them? |
| Docsumo | Document AI platform geared to extraction from financial and administrative documents. | Is the percentage measured at first pass or after human review and model learning? |
Note: ABBYY, Amazon Textract, Nanonets and Docsumo are trademarks of their respective owners. This page deliberately does not quote third-party percentages, precisely because they would not be comparable: measurements depend on the test set, the fields considered and the measurement point. Always verify current data with the vendor and, above all, measure it yourself.
How to measure accuracy on your own documents in eight steps
This is the method we use during trials and that you can apply to any vendor. It takes a day of work and produces numbers comparable across candidates — which is exactly what product sheets do not give you.
Build a representative sample
200 to 300 real documents reflecting your true mix: different suppliers, native PDFs and scans, documents with few lines and documents with forty. A sample made only of clean documents measures something that does not concern you.
Deliberately include the worst cases
Skewed or low-resolution scans, phone photos, new suppliers, foreign-language documents, credit notes and adjustments. They are 10-15% of the volume and generate most of the manual work.
Define which fields count, first
Write down the list of fields that must reach the ERP, line items included, and mark which are blocking and which are nice-to-have. Without that list every vendor will measure on a different scope and no comparison is possible.
Prepare the ground truth
For a subset of 50-100 documents, manually record the correct values field by field. It is the most tedious part of the protocol and the one that makes the results verifiable rather than debatable.
Have everyone process the same sample
Identical document set, identical field list, no upfront tuning. If a vendor asks to configure something before the trial, note it: that is time and cost that will reappear with every new supplier.
Measure header and lines separately
Compute field-level accuracy separating header fields from line fields, and compute how many lines were extracted in full. Averaging the two groups together is the easiest way to fool yourself.
Count perfect documents and untouched documents
Share of documents with no errors at all, and share of documents that would have been posted into the ERP without anyone opening them. Those are the two numbers to take into the meeting, because they convert directly into hours.
Check the quality of the doubts
Look at the documents the system flagged: how many were genuinely problematic and how many were false alarms. And above all, look at the errors it did not flag: those are the ones that reach the ledger with nobody noticing.
The four levers that take precision to 99.8%
Accuracy does not depend on a single factor, and in particular it does not depend on recognition quality alone. These are the levers Data Alchemy combines, in order of impact.
A dedicated LLM per document model
The platform does not use one engine for everything: it assigns each document type the LLM best suited to read it — today Anthropic's Claude AI, which outperformed GPT, Gemma and DeepSeek in our tests, and from Q3 2026 the proprietary Data Alchemy AI model. Specialisation per document model is what moves the result most on line items.
Semantic understanding instead of coordinates
Unlike template-based OCR, the model interprets context: it distinguishes supplier, net amount, VAT and lines even on layouts never seen before, with no upfront configuration. That is why a new supplier does not collapse accuracy, which is instead the typical behaviour of template systems.
Validation against ERP master data
Every extracted value is checked in real time against ERP master records: suppliers, item codes, tax rates, open orders. An extraction that is formally correct but inconsistent with master data is flagged before posting — a control no language model can perform on its own.
Three-way matching
On invoices, orders and delivery notes, three-way matching compares the three sources on quantities, prices and amounts within defined tolerance thresholds. It catches what a single extraction, however perfect, could never detect: that the document was read correctly but says the wrong thing.
Related pages
The dedicated-LLM architecture
How the best-suited LLM is assigned to each document model.
Learn more →ExtractionInvoice data extraction
Header and line items extracted, validated and written into the ERP.
Learn more →ControlsThree-way matching
The invoice-PO-delivery note comparison that catches what extraction cannot see.
Learn more →ComparisonBest data extraction software
The platform comparison on the criteria that actually matter.
Learn more →ROIAccuracy and financial return
How the straight-through processing rate feeds into the payback calculation.
Learn more →AboutThe numbers we state
Every figure we communicate, with the exact definition of how to read it.
Learn more →AI data extraction accuracy: frequently asked questions
Verify accuracy on your own real documents
Send us a sample of your documents, hard cases included: we hand back accuracy split between header and line items, the share of perfect documents and the share postable with no intervention. Measured numbers, not stated ones.