An AI Control Dashboard Is Not Control Evidence

An enterprise demo ends with a reassuring screen: activity history, approval status and a security integration. The agent has updated a product record and the change appears in the dashboard. Everything visible looks controlled.

But the buyer has not yet established what the screen cannot see. Which action routes were outside its scope? Did the event merely enter a queue, or did the monitor finish evaluating it? Was the approval attached to the precise change that reached the business system? Could the same business effect occur through a connector with a different permission boundary?

This is a hypothetical procurement scene, not a reported customer incident. Its point is a missing acceptance condition: evidence about one successful demonstration is not evidence about the full set of actions the buyer is authorising.

The useful procurement unit is therefore not “an agent with monitoring”. It is a defined set of consequential actions, in a defined configuration, with demonstrable control coverage.

Four claims that should not share one percentage

An agent's consequential actions might include publishing a price, changing a supplier record or sending an external message. Before calculating coverage, distinguish those business effects from model requests and technical events. One action may involve many requests; one request may initiate several actions. Counting requests alone need not tell you how well business effects are controlled.

Four different questions follow:

1. Eligible for monitoring: Does the configured monitoring policy intend to include this route or event? This is a scope claim, not confirmation that monitoring occurred.
2. Actually monitored: Is there reconciled evidence that the event reached the relevant monitor and that the specified evaluation completed within the agreed time? A log receipt alone may not establish this.
3. Independently tested coverage: Has someone separate from the implementation work checked the monitoring claim against a defined test set and configuration? Passing selected tests establishes evidence within that tested boundary, not universal completeness or future reliability.
4. Permission and approval enforcement: Can the business effect occur only within its authorised limits, with any required approval enforced before the effect? Observation after execution is not a preventive approval gate.

Do not average these into a single “control coverage” score. A process could have a detailed history of actions it should never have allowed. Conversely, a strong permission boundary might prevent a prohibited action while leaving insufficient evidence of the attempted action for investigation. Those are different shortcomings with different remedies.

What the METR pilot does—and does not—show

METR's 22 October 2026 post by David Rein and Adam Kaufman reports a high-access pilot conducted from 21 July to 11 August. It examined OpenAI's internal misalignment monitoring system as it existed on and before 21 July, not the subsequently expanded system. The public post describes selected findings from a 61-page internal report; the full report is not public. [1]

METR estimated that roughly 75–94% of agentic internal Responses API traffic from models at least as strong as GPT-5.5 was eligible for monitoring during the historical exercise. The authors expressly report substantial uncertainty in both directions. Their separate estimate of 99.2% eligibility concerns the narrower subset of Responses API traffic used for internal Codex agents. Neither figure is the percentage actually monitored, nor a coverage estimate for all internal agent traffic. The traffic estimate excludes other known sources of internal agentic coding traffic whose volume the authors could not estimate. [1]

The post's definition matters more to an enterprise buyer than the headline range: eligible traffic is intended to be sent to a monitor, but load shedding, monitor errors or implementation bugs can prevent it from completing the monitoring pipeline. That distinction is why intended scope and observed completion need separate evidence. [1]

In an update dated 29 October 2026, METR reports that OpenAI has stated most of the identified vulnerabilities have been patched, with one remaining fix still in progress that OpenAI says does not affect its misuse safeguards stack, and that OpenAI has described significant subsequent changes to the intended coverage of its monitoring system. METR says it has not evaluated those claims. This is not an independently retested account of the current system, and it does not establish current OpenAI production exposure. Nor does this internal research-system assessment establish how a business customer's agents or applications behave. [1]

The transferable lesson is a measurement question, not a supplier verdict: what is the denominator, and what exactly happened to the events counted as covered? No exploit procedure is needed to ask it.

Vendor controls are a starting hypothesis to verify

Akeneo's original 28 October 2026 Autumn Release announcement supplies a concrete enterprise example. It says Ziggy operates within the human team's role-based permissions, records every action as a traceable change and proposes changes for human approval before application. It also describes SIEM connectivity as bringing PIM activity into enterprise security operations. Specific capability availability varies by Akeneo Product Cloud edition. [2]

These are vendor assertions about capabilities, not independently validated enforcement or customer outcomes. The announcement does not establish that every agent event appears in the SIEM, that every consequential action is approval-gated in a buyer's configuration, or that the receiving team can respond before an effect occurs. This article makes no beta-status claim.

The fair response is neither to accept the claims as assurance nor to dismiss native controls. Ask the supplier to translate each relevant claim into a versioned acceptance condition for the edition, connectors, roles and action types being bought. A supplier's native control may be the best enforcement point because it sits close to the system of record. Its location does not remove the need to test the boundary.

Buyer-side evidence adds a useful, narrower perspective. The Bank of England and FCA's 26 October 2026 survey received responses from 196 UK financial firms, of which 105 completed the full questionnaire; detailed findings generally use valid question-level responses from that full survey, not all 196 firms. Among relevant respondents, 94% reported firmwide AI policies, principles or standards. More than 90% reported each of pre-deployment review, human approval and vendor due diligence as AI data-risk controls, while ongoing monitoring, leakage testing and periodic review or recertification were each reported by around half; 54% identified applying model-risk management to vendor models as a challenge. These are self-reported arrangements and challenges, not independently tested control effectiveness, causal findings, audited benefits, representative sector prevalence or evidence of demand for Boesing services. Nor should agent adoption be read as broad authority: among relevant full-survey respondents, 80% reported production decision-support agents—humans decide and execute—but only 2% reported broad-autonomy agents. Respondent and definition differences also limit cross-year comparisons. The buyer implication is to ask what evidence connects a declared control to the particular action boundary being expanded, not to infer that any respondent's controls failed. [4]

Build an action-to-evidence map, not another dashboard

Begin with a short table for the proposed deployment. For each business effect, record the route and connector, executing identity, permission limit, required approval, event expected at the monitor, evidence of completed evaluation and evidence in the destination system. Include explicit exclusions. Treat an unclassified action route as unknown, not silently covered.

Use that map to specify separate measurements:

• Intended scope: which enumerated action routes are included and excluded by policy. A route count is not a traffic-weighted rate.
• Observed monitoring completion: completed evaluations divided by independently reconciled in-scope events expected to be evaluated, over a stated period. Separately report events discovered outside the intended scope; otherwise a narrow denominator can hide a wide gap.
• Evaluation timing: event-to-evaluation and event-to-review time against the agreed response requirement. Report missing, failed, delayed and unresolved events rather than dropping them from the calculation.
• Enforcement results: allowed and prohibited test cases attempted, and the corresponding effects observed at the business-system boundary. State which cases required approval and which approval checks were exercised.

These are proposed buyer measurements, not METR metrics, a recognised certification or a tested commercial method. Request counts and business-action counts should remain labelled separately. Do not infer an enterprise's coverage from a percentage in another organisation's study.

The denominator also needs scrutiny. If the same component produces both the action log and the coverage dashboard, both may omit the same route. Reconcile against a suitable separate source, such as destination-system records or gateway observations, while minimising collected personal and sensitive data. That reconciliation is evidence about observable effects, not a guarantee that every possible action has been discovered.

A bounded acceptance test before the next expansion

The buyer does not need an unlimited adversarial exercise before allowing a low-impact drafting assistant. It does need an explicit test boundary before giving the agent a materially different power. A pragmatic acceptance checklist is:

1. Freeze the decision scope. Name the edition, version, connectors, identities, business effects and excluded routes. Identify the new authority being purchased or expanded. Record unknowns instead of converting them into green status.
2. Define the evidence and tolerance before testing. Specify required preventive controls, monitoring completion, response timing and acceptable uncertainty for each effect. For an action designated as requiring prior approval, the acceptance requirement is that it cannot take effect without valid approval—not merely that an alert is generated afterwards.
3. Reconcile expected with observed events. In an authorised, isolated test environment, exercise agreed representative action routes. Match each expected event to receipt, completed evaluation and destination-system evidence. Record omissions, unresolved matches and observed routes outside the policy.
4. Test the permission boundary, including ordinary rejection. Verify a permitted action succeeds within its limit and a prohibited action does not produce the business effect. Test the configured role restrictions and permission revocation. Check the destination state rather than relying exclusively on the agent's response or dashboard label.
5. Test the approval boundary. For approval-required cases, verify that absent, rejected, expired or revoked approval prevents the effect. Check that approval binds to the specific action and material parameters being executed; changing them must follow the agreed reapproval rule. Use benign test records and agreed procedures, not live customer effects.
6. Exercise agreed degraded-service conditions. With the supplier and SecOps team, test safe simulations of unavailable or delayed monitoring and disrupted approval services. Specify beforehand whether each action must pause, be queued, or continue within an explicitly accepted boundary. An asynchronous monitor must not be presented as preventing an action that happens before its evaluation.
7. Separate the reviewer from the implementation. Have competent internal reviewers outside the implementation team, or an appropriately scoped specialist, inspect test selection, results and exclusions. Preserve configuration, test cases, observed effects and unresolved limitations. Independent review is not necessarily external certification and does not prove untested routes safe.
8. Make acceptance conditional on the tested configuration. Choose accept within scope, restrict autonomy, remediate and retest, or defer. Identify changes—new connector, role, edition, action or approval configuration—that invalidate the evidence and require targeted retesting before expansion.

These are acceptance gates for a proposed action boundary, not sunset criteria for deciding whether an existing AI system still earns its keep. They establish neither general accountability nor full security assurance. A pass means the defined tests met agreed conditions; it is not proof that every future consequential action will be controlled.

What NIST adds—and what it does not

NIST AI RMF 1.0 is a voluntary risk-management framework, not a certification or automatic legal compliance. Its original text offers useful discipline: MEASURE 1.2 addresses regular assessment of metrics and existing controls; MEASURE 1.3 involves internal experts who were not front-line developers and/or independent assessors in regular assessments; MEASURE 2.1 calls for documenting test sets, metrics and TEVV tools; and MEASURE 2.7 addresses evaluation and documentation of security and resilience. [3]

Those references support documenting and examining the basis of a control claim. They do not prescribe this article's checklist, certify a vendor, or establish an organisation's legal obligations. The buyer still has to define the action set, consequences and acceptable evidence for its own decision.

The fair counterargument: assurance can become the bottleneck

There is a real objection: requiring specialist testing of every minor agent change can consume more effort than the workflow justifies. Native logs and existing SecOps processes may already provide adequate visibility for a narrowly constrained task. Duplicated monitoring can add noise, sensitive-data exposure and unclear response ownership.

That argues for proportionate testing, not a universal purchase of another security tool. Native vendor controls may be sufficient if their relevant limits are demonstrated. Internal SecOps may perform reconciliation and independent review when it has the competence and time. Specialist security assurance can be appropriate for consequential or technically complex boundaries, but a report must disclose what was actually tested. An external observer can add detection without supplying an enforcement point.

Narrower autonomy is another credible choice: let the agent prepare a proposed change while an existing business workflow applies it through established permissions and approvals. That can reduce the immediate assurance burden, although the handoff still needs testing. Retaining the current manual process while a specific evidence gap is resolved can also be rational.

The CIO's choice is therefore not “buy controls or take uncontrolled risk”. It is which combination of native enforcement, existing operations, independent review and limited autonomy provides sufficient evidence for this particular expansion.

Buy the boundary you can verify

A control dashboard is useful when it helps explain a defined boundary. It becomes misleading when its visible activity is treated as evidence of everything an agent can do.

Before signing or expanding, ask for the action set, the monitoring denominator, the tested enforcement conditions and the exclusions. If the supplier or implementation team cannot produce them, narrow the authority or resolve the gap. The outcome may be a small acceptance test, a configuration repair or no further investment—not necessarily a new platform.

If you need to frame the decision first, start the free Workflow Fit Diagnostic. It can help clarify the business problem and a possible next step; it does not test security controls, certify coverage or replace a specialist assessment. Your free result may be enough.