29 of 29
quotes found verbatim and inside the cited section, in the September run
4 of 29
judged enough on their own; "pointer only" is the common honest answer
5
section detectors, best first; one wins per document
0
OCR: a scanned file is parked and flagged, never guessed at
The whole pipeline
Two triggers on one document row: the upload, and a deliberate flip to Queued.
Call
Rows written
Extraction never leaves the isolated runner. Only the two jobs that need a language model run where they can reach it.
1
Ingest: text and structure
The section index is built in the same pass as the text, from the structure the file actually has.
| File | Read with |
|---|
| PDF | pdfplumber, table-aware, so table cells keep their order |
| Word (.docx) | python-docx, keeping paragraph styles and table rows |
| Older Word, RTF, OpenDocument | A text converter |
| Text, Markdown | Read as text |
| Scanned PDF, image, anything else | Parked as Cannot Read, with a note and one alert; a replacement file un-parks it |
Finding the sections: best detector first, one wins
- Declared heading styles in a Word file: authored structure, the most reliable.
- The table of contents, used as the outline when the document has one.
- Markdown headings, in Markdown and plain text only.
- Font size in a PDF, inferred from how the headings are set.
- Numbering ("4.2", "AC-3"), the floor every format reaches. If it matches more than 60% of lines it's noise, and nothing is recorded.
Each section stores its start and end position in the extracted text, its level and its parent. The detector that won is recorded on every section, so a guessed structure never reads as an authored one.
A cited document can't be silently re-read. If evidence already points at a document's sections, re-ingesting it is refused before anything is written, because new offsets would break every existing citation. A changed version is currently a new document.
2
Review: a model reads for the Define objectives
Asked for exact words and an honest judgment, and told that "nothing here" is a good answer.
- Find the assessment. The document's Review Against link, or the only assessment in the base. With none, or several and no link, the review refuses.
- Pick the objectives. Only the assessment's Define objectives: those asking whether something is defined in writing. Perform objectives want artifacts, not policy text, and go to Phase 4.
- Collapse duplicates across frameworks. Objectives joined by an "Equivalent" mapping (110 such pairs) are reviewed once, and the evidence is linked to all of them. Looser mappings are left apart.
- One call per control family, with the family's objectives and the whole document. A document over 120,000 characters is refused, never truncated.
- The rules the model gets: quote verbatim; judge each quote as "Satisfies objective", "Partial, insufficient alone" or "Pointer only, evidence lives elsewhere"; returning nothing is allowed; if the document names a different organization than the client, say so and prefer "Pointer only".
- Locate every quote in the text ourselves. The model is never asked where a quote came from. Its position in the text gives the section, and the section gives the citation.
- Write each match as Evidence, AI-proposed, then set the document to Reviewed. If any family's call failed or came back unreadable, nothing is written, the document stays Queued and an alert fires, so a document marked Reviewed has had every family read.
A policy that says the organization "shall" do something reads a lot like one showing it's defined. That distinction is why "pointer only" is offered as a correct answer and why nothing is counted until a person reads it.
3
Verify: mechanically, then by a person
A model's quote is a claim about the source. Two checks test it before anyone relies on it.
| Check | Passes when | Otherwise |
|---|
| Quote Verified | The quote appears in the extracted text, ignoring only whitespace, quote marks, dash styles and letter case | "NOT FOUND IN SOURCE". The row is kept and flagged, so the rate of invention stays visible |
| Section Verified | The quote's position falls inside the cited section or one of its subsections | "No section cited" or "No section index for this document": both are a review queue, never a pass |
Case is ignored because changing case can't fabricate a claim. Both checks also run as separate jobs that can be re-run after documents or sections change; a check with no source text to compare says so rather than passing.
What counts, and when
| Reader | Counts a document citation when |
|---|
| Coverage check | A person set it Human-verified and it's judged "Satisfies objective" |
| SSP generator | Both of those, and the quote verified verbatim |
| Documentation proposals (Phase 6) | Uses the harvested client values to fill drafts; a value that was never found stays a flagged placeholder |
| Client frequencies (Phase 8) | The same pass reads how often the client says it runs the five activities NIST SP 800-171 has it define; a value is kept only when its quote appears word for word, and a person confirms it |
Verification and sufficiency are separate fields on purpose: a real, correctly quoted pointer is verified and still not enough.
Decision log
The calls that shaped document review, and when they were made.
| Decision | Why | When |
|---|
| Review on a deliberate flip to Queued, not on upload | A person decides when a document is ready and which assessment it's for | 2026-07-20 |
| Sufficiency separate from verification | A true quote can still be only a pointer | 2026-07-20 |
| Build the section index in the same pass as the text | Structure is lost once the text is flattened | 2026-07-20 |
| Derive the citation from the quote's position, never ask the model | A location has to be checkable, like the words | 2026-07-20 |
| Ingest drains until nothing is pending; review drains the whole queue | A missed webhook shouldn't strand a document | 2026-07-28 |
| No-login submission form with a security-data-only notice | The client can send documents without an account | 2026-07-29 |
| Park unreadable files as Cannot Read, alert once | An unreadable scan was failing every later run, even runs whose own documents ingested fine | 2026-09-17 |
| All or nothing per document: a failed or unreadable family call writes nothing and leaves it Queued | A failed family used to be skipped while the document still showed Reviewed | 2026-09-30 |
| Check the organization named in a document | An unrebranded template produced most of one run's strongest matches | 2026-09-17 |
Known gaps
- No OCR. A scanned policy is parked and flagged, but can't be read until someone supplies a text version.
- Another company's template only warns. A document naming a different organization triggers a warning in the log and a hint to the model; the row isn't flagged.
- Versions aren't handled. A new version of a cited document is refused rather than replacing the old one.
- The checks run on request. The quote and section checks, and the orphan sweep, aren't scheduled.
- Verbatim isn't provenance. A quote can match the text exactly and still describe someone else's environment; a person has to judge that.