The test and the competing models
According to AI4Business, the editorial team built a pipeline for local document management and compared five small language models on a corpus of one hundred real corporate documents, adapted for the test. The models tested were LFM2.5 1.2B, Phi-4-mini-reasoning, Qwen3.5 9B, Gemma 4 E4B and Gemma 4 12B.
The starting point is a practical distinction: attaching a PDF to a chat is not the same as automating a document process. For the first two models, which lack native PDF-reading capabilities, the files first passed through PaddleOCR, which converted text and tables into a structured form usable in subsequent stages.
The required task and how it is measured
The models were asked to read each document, separate commercial documents from technical ones, and return the supplier, invoice number, dates, line items and amounts in JSON. Dates had to follow a fixed convention, line items had to maintain their sequence, and credit notes had to preserve negative signs. The format is not a detail: it is what makes it possible to connect the model to a business management system without interpreting every response.
The corpus comprised 100 files, 230 pages and 346 accounting line items, 80% in Italian and 20% in English, divided into five groups of twenty files. For each document there was a reference with the expected values, so errors could be counted in a verifiable way. A partial result was considered incorrect, because it is incompatible with programmatic use of the data.
The results and limitations of the test
Gemma 4 E4B is the most accurate model: it correctly completes all 100 documents, with a typical processing time of 2.80 seconds. Qwen3.5 9B completes 96 and is slightly slower, at 3.61 seconds. Gemma 4 12B, the largest and slowest (6.00 seconds), stops at 94 documents despite correctly handling 99.7% of fields. The two smaller models are extremely fast but inefficient: Phi-4-mini-reasoning completes 3 out of 100 documents, LFM2.5 1.2B none.
The difficulties of the three best models are concentrated in missing fields, where the required value was null and in several responses a datum from another source was inserted instead. For Gemma 4 12B, five of the six discarded documents reported the corpus reference code present in the footer in the number field. According to AI4Business, this indicates that prompt quality matters as much as model quality: instructions must say what to do and also what not to do.
The test used independent sessions, one request at a time, temperature zero, seed 42, an 8,192-token context and a maximum of 4,096 generated tokens. Formatting issues, misleading data and prompt injection attempts were also introduced: Gemma 4 E4B, Gemma 4 12B and Qwen3.5 9B were not fooled. It remains to be verified how well they hold up on volumes and documents different from those in the sample.
