How we tuned an AI system to match an expert's judgement

The problem

A roof renovation costs millions of forints, and the homeowner usually gets two or three quotes for it. Those quotes are not comparable: different quantities, different technical content, different line-item structure, sometimes handwritten. The homeowner has no expertise to judge which one has real substance behind it and which one is cheap because half of it is missing.

From the contractor’s side the same thing hurts in reverse: they regularly lose to quotes that are only cheaper on paper. Out of this double pain came the idea of a paid, independent review that the homeowner orders before signing with anyone.

What we built

1. Submission and document handling. The client uploads the quotes they received: PDF, photo, scanned paper, handwritten sheet. The system first checks whether they are legible and complete, and asks for missing pieces before analysing anything.
2. Extraction. Line items, quantities, unit prices, tax status, payment terms, warranty, exclusions. Handwritten and photographed quotes go through as well.
3. Scoring. Nine criteria, 1 to 5 points each, 45 at most. The score falls into one of four decision bands. Price is measured against a reference table per trade, not on its own.
4. Human approval. The expert reviews, corrects and approves. The system technically does not allow this step to be skipped.
5. Issue. A PDF opinion with the per-criterion assessment and a list of questions the client should clarify before signing. Automatic client correspondence with deadline tracking on the Hungarian working calendar.

The central design decision: AI prepares, a human approves. This is not caution. Professional responsibility cannot be pushed onto a machine, and legally this is what makes it not a solely automated decision. So the approval gate is not a setting, it is built into the structure of the system.

The hard part: calibration

Writing a scoring system takes a day. Getting it to say what the expert would say is a different task entirely.

The method: the owner scored three real quotes on the same nine criteria in advance, independently of the system. Then we ran the system on the same documents and looked primarily not at whether the scores matched, but at the sign of the differences: if the system consistently errs in the same direction on a criterion, that is a rule error. If it scatters, that is noise, and one can live with it.

What six rounds taught us:

• The most dangerous error is not imprecision but the wrong direction. The system initially saw one quote two decision bands more favourably than the expert. One point off is fine. Two bands means the client may make the wrong decision.
• The error is often in the weighting, not the rule. The same deficiency was deducted on several criteria, so small faults accumulated into disproportionate penalties. The fix was a global accumulation brake.
• Anonymisation is not redaction. The first attempt with black boxes failed: the system saw the documents as incomplete. The solution was substitution with invented but realistic data.
• The system exposed the expert’s own inconsistency. On two criteria the owner first overruled the machine; in a later round he concluded himself that his earlier assessment was stricter than his written rules justified. The machine applied the rules, the human applied intuition, and the gap between them made the rulebook itself more precise.

The final measurement

Quote	Expert	System	Decision band
Contractor “A”	18	27	one band apart
Contractor “B”	33	32	match
Contractor “C”	42	40	match

The decisive result is not in the table. The system’s recommendation agreed with the expert’s on every question: it recommended the same one of the three quotes, and the same one when choosing between any two. For a service like this the stake is not whether 32 or 33 comes out, but whether it reaches the same conclusion.

Measurement, not estimation

The system logs every AI call: case, step, model, input and output tokens. Based on nineteen full runs, at official model prices and the central bank exchange rate, we know exactly what one review costs.

What we measured	Basic package	Premium package
AI calls on average	6.2	9.0
Average machine cost per run	HUF 63	HUF 103
Fixed infrastructure per month	approx. HUF 16,100

This had a business consequence. Since the variable cost is practically zero, the monthly allowance could be generous, and pricing was built on availability and accountability, not on machine time.

What most people skip: legal preparation

This service processes documents written by another business, containing other companies’ data, personal contact details and unit prices that may qualify as trade secrets. That cannot be solved on the fly: a privacy notice with a dedicated chapter on third parties in submitted documents, a liability disclaimer in the issued opinion, a withdrawal statement at ordering, versioned legal texts logged on every issued document, and a gate that stops document issue in production if any real company detail is missing from the legal texts.

What this means for a new partner

The engine is trade-independent: document processing, the scoring machinery, the approval gate, document generation and the legal framework are the same in every field where a consumer gets several quotes for high-value work and cannot judge them. What is unique per partner is the content: the price reference table, the assessment criteria and the typical pitfalls of the trade. That is the partner’s expertise, and it stays theirs.

What has to be built	Effort
Price reference table and internal consistency check	approx. 1 day
Professional rulebook and risk factors	1–2 days
Calibration on three real quotes	1–2 days
Full deployment	3–5 days

The first system took half a year. The second takes a few days. The difference is that the hard part is behind us.

The scores in this case study come from real measurements. The partner’s price reference data and rulebook are trade secrets and are not included.