Notes/LLM · extraction · evaluation
A correct value, attributed to the wrong object
Why LLM information extraction rarely fails on the value and often on the object it belongs to — and how to measure it.
2026-11-10 · 2 min · Mohamed Abi
When you evaluate a system that extracts information from technical documents, the first question is almost always: is the value right? It's a good question, but it isn't the most expensive one.
The error you don't see
Take an end-of-well report. It describes a main well, a deviated branch, and cites a third, neighbouring well as a geological reference. All three have a final depth written somewhere in the document. Ask a classic RAG system "what is the final depth?": it retrieves the passage most similar to the question and answers with a plausible, well-formatted value taken from the text.
It can be perfectly accurate — and belong to the wrong object.
In a reference system, that error is worse than a missing value: it passes format checks, it looks sourced, and it contaminates everything built on the master data.
Separate "who" from "what"
What worked in my work wasn't a better prompt. It was splitting the problem into two passes:
- List the objects the document is about, with their identifiers and role — main subject, branch, or a mere reference to exclude.
- Extract the attributes constrained by that list: a value can only attach to an object already identified.
Two-pass extraction isn't new. What changes everything is combining it with navigation guided by the document's structure — sections, titles, summaries — instead of fixed, context-free chunks, and modelling the domain hierarchy so a neighbouring well can't become the subject.
Measure at two levels
If you only measure whether values are right, you reward the system that always answers something. So I split the evaluation:
- L1 — identification: did the system find the right subject objects, no more, no less?
- L2 — attributes: on correctly matched objects, are the values right?
Missed, mis-identified and invented objects are counted apart. That split revealed something a single score hid: the model best at finding objects was not the best at extracting their attributes. Hence a composite system that gives each step to the best-suited model, trading quality against inference time.
What I keep from it
- A value without its object isn't data: measure attribution.
- An average metric decides for you what matters; split it.
- Write the limits next to the results. Here they exist: some branches remain hard to find, OCR caps some attributes, and nothing shows that a vectorless approach is universally better.
This work's figures stay internal to the company; this note only describes the method.