AI Strategy
Pending forever: data governance for a forgotten file
Published Nov 12, 2026
Data governance consulting: on 11/10, 63 documents sat pending for days, 383 waited on extraction, and 50 of 52 failures were no text. Nothing alarmed.
Pending forever: data governance for a forgotten file
On 11/10, production on our agent platform's document vault showed 63 documents pending for days. 383 were waiting for knowledge-graph extraction. 52 had failed, and 50 of those failed as "no text". Nothing had failed loudly, so nothing was alarmed.
Why do documents get stuck pending in an AI document pipeline, and how does data governance fix it? Documents stay pending when a stage runs out of retries or a timeout kills the function before it can record a failure, when one sweep window starves other stages, or when a scan has no file extension and never reaches OCR. Data governance gives every pending document an owner and ends each item as read or failed with a reason.
Data governance consulting is the work of naming an owner, a definition, an access rule, and a quality check, so a document cannot sit pending without an owner.
The hub for this work is data governance consulting. That page names the owner, the definition, the access rule, and the quality check. This page is what happens when the quality check never fires, and the document says waiting. The sibling that counts rows when a pipeline stays green is data governance best practices: green pipeline, empty metric. The table a team fills for the owner and the rule is the data governance framework template. Which source is allowed to win sits on data strategy consulting.
The three ways a document pipeline forgets a file
Three causes. None of them raised an alarm.
- When a stage ran out of retries, or a function was killed by a timeout before it could record a failure, nothing ever wrote the document again. It said waiting forever.
- The nightly sweep read the 2,000 oldest owed rows across all stages. A backlog in one stage starved the others.
- Scanned PDFs or images stored without a file extension were never sent to OCR.
I kept looking for a loud failure. There was not one. The vault was doing the last thing a stage had managed to write, and for these files that write never came.
What changed in the sweep?
The sweep now files given-up documents as failed, with the stage and the reason. It reads one window per stage. It sniffs file bytes to detect scans. A redrive job, dry-run first, re-reads them.
A separate fix handled six duplicate rows pointing at a file that no longer existed.
What does before and after look like?
Before
- After retries or a timeout. Nothing wrote the document again. The status stayed waiting. Nothing alarmed.
- Sweep. The 2,000 oldest owed rows, across all stages. A backlog in one stage starved the others. Knowledge-graph extraction was holding 383.
- OCR. Scanned PDFs or images with no file extension were never sent. 50 of the 52 failures were no text.
After
- Given up. The sweep files the document as failed, with the stage and the reason.
- Sweep. One window per stage, so a backlog in one stage cannot starve the others.
- OCR. The sweep sniffs file bytes to detect scans, so a missing extension no longer keeps a scan out of OCR.
- Redrive. Dry run first, then the job re-reads them.
The file did not vanish. The pipeline stopped writing it, and the status still said waiting.
What should the pipeline be required to do?
- Level 0: job ran. The sweep finished. Nothing alarmed, because nothing had failed loudly.
- Level 1: every stage owns its rows. The sweep reads one window per stage, so a backlog in knowledge-graph extraction cannot starve the rest.
- Level 2: every document ends read or failed with a reason. A stage that runs out of retries, or a function killed by a timeout, still writes the failure.
- Level 3: redrive with dry run first. The job re-reads the documents, and the dry run comes first.
A document is never allowed to sit in pending without an owner. Every item ends in read or failed with a reason.
When you do not need this
You do not need this when every document already ends the same day as read or failed with a reason, and a stage that stops retrying already alarms.
What should a buyer count?
AI over company documents is only as good as what got ingested. A diligence team or operating partner should ask for the count of pending and failed documents, not a demo of the chat.
A new agent benchmark for long multi-step projects found the leading model fully completed 36.95% of tasks. The benchmark is built to study integration failures. Our version is a pipeline step that quietly stops retrying.
The loop that has to finish the ingest is agentic workflows. If you want the pending count before you trust the chat, start with data governance consulting. Talk to us.
Related reading
Frequently asked questions
Why do documents get stuck pending in an AI document pipeline, and how does data governance fix it?
Documents stay pending when a stage runs out of retries or a timeout kills the function before it can record a failure, when one sweep window starves other stages, or when a scan has no file extension and never reaches OCR. Data governance gives every pending document an owner and ends each item as read or failed with a reason.
What is data governance consulting?
Data governance consulting is the work of naming an owner, a definition, an access rule, and a quality check, so a document cannot sit pending without an owner. On this file the quality check is the end state. Every item ends read, or failed with a reason.
What did the document vault show on 11/10?
Production on our agent platform showed 63 documents pending for days, 383 waiting for knowledge-graph extraction, and 52 failed. 50 of those 52 failed as no text. Nothing had failed loudly, so nothing was alarmed.
Why did scans fail as no text?
Scanned PDFs or images stored without a file extension were never sent to OCR. The sweep now sniffs file bytes to detect scans. A missing extension is not proof the file has no text.
What should a diligence team ask about an AI document pipeline?
Ask for the count of pending and failed documents, not a demo of the chat. AI over company documents is only as good as what got ingested. A document is never allowed to sit in pending without an owner.
Written by Anthony Wentzel