Constrain an evidence agent

Objectives

The evidence agent currently runs nightly with an engineer's cloud credentials, is told in its system prompt never to modify anything, and writes its summaries straight into the evidence store.

Rewrite the workflow definition so the agent can plan and collect but never conclude: read-only scoped tools, its own identity, a human gate before anything is recorded, output stored as a draft, and provenance an auditor can read.

incidents:
  a "read" tool issued a tag update during a retry loop and disabled a monitor
  an August summary asserted full MFA coverage; its own filter excluded 14 admin service accounts
  a gap analysis cited a control id that exists in no revision of the standards repo

platform provides:
  service principals with per-tool allow-lists and read-only scopes
  a review queue with named approvers
  a drafts area separate from the evidence store
  standards-repo revisions addressable by id

Step-by-step hints

The guided walkthrough, in full. It is free: no XP cost, and it does not touch the paid hint ladder or the reveal gate.

1. Step 1

   Take away the credential, not just the permission to use it

   A prompt instruction is guidance and a credential is authority. Give the agent the dedicated identity the platform offers rather than a human's, then set the tool mode to read-only and list the tools it may call.

   The agent has its own identityTools are read-only and allow-listed

2. Step 2

   Put a human between collection and recording

   The August incident happened because a filter the agent wrote defined the population and nobody checked it. Require approval before anything is recorded, and change where unreviewed output lands so a draft cannot be mistaken for an artefact.

   A human approves before anything is recordedUnreviewed output is stored as a draft

3. Step 3

   Record how the output was produced

   Turn on the three provenance fields, which model, which tool calls with their parameters, and which reviewer approved, and switch on the mechanical check that resolves every cited identifier against the named repository revision.

   Model, tool calls and reviewer are recorded

Rules summary

• XP on pass

  45 bonus XP the first time every check goes green.

• Paid hints

  2 available. Each costs 25% of the lab XP (about 11 XP).

• Reveal solution

  Scores 0 XP and unlocks only after 3 check runs.

• Attempt cap

  60 check runs per learner. The runner warns you in the last 10.

• Drafts

  Your editor content is saved in this browser, so a reload will not lose work.

• Walkthrough

  Always free and never penalised: use it before spending a hint.