Evidence Queue: CMMC Readiness

286

evidence recipes: what an assessor wants to see, per objective

195 → 59

Perform objectives in scope, and requests actually sent, in the synthetic Level 2 run

1 of 3

artifacts accepted in that run; the screen caught a wrong product and a stale report

decision by the consultant per request; the rest follows from it

The whole pipeline

Ask, collect, screen, decide. Only the screen uses a model, and it never decides.

Step Written by reconcile Dashed box: the client

This phase covers the Perform objectives: things shown to be done, with an artifact. The Define objectives were answered from documents in Phase 3.

Ask: requests from a recipe library

Each request says why an assessor wants it, what to send, and whose system it comes from.

Recipe field	Holds
Assessor Intent	In plain words, what the assessor is trying to establish
What It Must Show	The specific elements an acceptable artifact contains
Artifact Type, Typical Source System	A screenshot, a configuration export, a report; and where it usually comes from
Responsible Role, Freshness Window	Who usually produces it, and how old it may be
Function (optional)	When set, the recipe names `{{technology}}`, filled with the client's own tool for that function

Recipes with a function are routed to a specific kind of tool; they reach 83 of the 110 Level 2 controls. The rest stay generic.

Generating the requests

Keep only requirements the client claimed Implemented or Partially Implemented in triage (Phase 2e). A claimed "no" never gets a request.
Take their Perform objectives, and the recipes linked to them, one request per recipe.
Fill {{technology}} from the inventory (Phase 2b). If no inventoried tool provides the function, the placeholder stays and the request is flagged "confirm how this control is actually done".
Write the request as the assessor's intent, then "what to provide", then source, owner and freshness, at status To Collect.

Re-running adds only requests that don't exist yet; it never edits or deletes one.

Collect: three ways in

The client proposes; the consultant commits. Nothing a client submits can mark itself verified.

Route	How	The request afterwards
Submission form	A shared form with no login; it links to a request and hides every verification field	Received within seconds: the arrival of a file fires reconcile
Client portal	Upload on the request's page (up to 25 MB); a job files it in the base and removes the upload copy	Received at once; a previous decision and feedback are cleared
Through the consultant	Added and linked by hand	Received within seconds, the same way

Screen: a model reads the artifact

Advisory only: it tells the consultant where to look first.

Pressed as a button, "Run vision check", not fired on upload. It picks up every request's linked images and PDFs not yet decided by a person.
The model gets the request text, the freshness window and today's date, and the artifact.
It returns whether the artifact is the right kind, which required elements are present and missing, whether it's fresh, stale or undated, and a verdict: Likely match, Partial match or Likely mismatch. An unrecognized verdict is recorded as Likely mismatch, toward a closer look.
It writes the verdict and notes on the request, and marks the evidence AI-proposed. It never marks anything MET. A reply it can't read leaves the verdict untouched and fails the run, so the alert fires; a resubmission through the portal clears the old verdict.

What it caught in the synthetic run. Three artifacts came back. One passed. The screen flagged an identity report from Okta sent against a request for Microsoft Entra, and a Defender report dated outside its 30-day freshness window. Both were rejected.

Decide: one call, rolled through

The consultant sets Reviewer Decision in the "Received (verify)" view. Reconcile does the bookkeeping.

The decision fires reconcile. A change of Reviewer Decision to Accepted or Rejected triggers it on the isolated runner, with no button to press. If a reconcile is already running, it retries every 30 seconds, then alerts.
Pass 1, the evidence. Accepted marks the request's linked evidence Human-verified, and "Satisfies objective" if its sufficiency is blank; a file rejected at an earlier decision stays Rejected when a newer file is there to accept. Rejected marks it Rejected. Pending changes nothing.
Pass 2, the request. Verified evidence that satisfies moves the request to Verified; real but unverified evidence to Received; nothing usable back to To Collect. A request marked Not Applicable by a person is left alone.
The client sees rejections. Reviewer Feedback is what the client reads next to a rejected request in the portal, on its next hourly refresh.
It reports how many Perform objectives now have every covering request Verified. Marking them MET is Phase 5, and stays a person's call.

What counts later. For Phase 5 and the SSP, only evidence that is Human-verified and judged "Satisfies objective" counts. An AI-proposed row, or an unset one, counts as unverified.

Decision log

The calls that shaped the evidence queue, and when they were made.

Decision	Why	When
Sufficiency separate from verification	A real artifact can still be only part of the answer	2026-07-20
A recipe library, with tool-specific recipes by function	One ask, worded in each client's own tools	2026-07-21
Generate, screen and reconcile built; screen validated on hand-graded samples	The screen caught an MFA policy with no enrollment and a stale scan as Partial	2026-07-22
Reconcile fires on the decision itself	No second button to forget	2026-07-23
Reviewer Feedback as a client-facing field	The client needs to know what to fix, in plain words	2026-07-23
No-login submission form, verification fields hidden	A client can't mark its own evidence verified	2026-07-29
Portal upload, reviewed like any other submission	The client proposes; the consultant commits	2026-07-31
A busy button retries, then alerts	A press during a run should wait, not vanish	2026-08-20
An accept never revives an earlier rejection; an unreadable screen reply fails loudly	Both could leave a wrong state with nothing to show for it	2026-09-30
A file's arrival fires reconcile; a busy decision run retries, then alerts	Form uploads sat unseen at To Collect, and a decision made mid-run could wait	2026-09-30
Age accepted evidence against its window: Refresh due, then Stale; a stale artifact stops counting, the mark and score stay the consultant's; a new upload reopens review	A screenshot proves a setting on the day it was taken, and a refresh must be reviewed, not accepted under last year's decision	2026-10-01

Known gaps

The screen has blind spots. Spreadsheets and Word files aren't graded, and a request with several files shows only the last verdict.
Freshness ages the request, not the score. A daily sweep turns accepted evidence Refresh due, then Stale, and a stale artifact stops counting; nothing emails the client yet, so the ask reaches them through the portal or the monthly review.
Requests only grow. A requirement later downgraded keeps its requests, and an existing request isn't linked to objectives added after it was created.
Two counts disagree. The coverage check and the SSP credit an accepted request artifact differently, so a Perform objective can read "No Evidence" in one and count in the other.