Collect, normalize, timestamp and store control evidence automatically.
55 min 240 XPSkills: Pipelines, Evidence
Five stages, and only one of them is glamorous
An evidence pipeline collects from a source, normalizes the payload into a shape you can query, timestamps it from a trusted clock, hashes it, and writes it somewhere it cannot be altered. Collection is the part everyone builds first and the part that matters least; the integrity stages are what turn an export into evidence.
Normalization is where most of the value hides. Once every source produces the same record shape, subject, resource, action, timestamp, source system, one query answers the same control question across ten systems, and a new source is a new collector rather than a new reporting project.
Collection is the easy stage. The last three are the ones an auditor tests.
Chain of custody, concretely
Chain of custody is the unbroken account of where an artifact came from and everything that touched it since. In practice it is four facts recorded at collection: what produced it, when, under whose identity, and the hash of the bytes as collected.
It breaks the moment a human handles the artifact between collection and storage. Someone exports a CSV, opens it to tidy the columns, then uploads it: the recomputed hash no longer matches, and the honest answer to "is this what the system produced" becomes no.
If a person can edit the artifact between the system and the store, you have a report, not evidence.
One local edit, and the artifact can no longer speak for the system.
Immutability is a storage property
Write-once storage, object lock, versioning, retention in compliance mode, is what stops the useful person with production access from fixing a bad artifact quietly. Retention is set from the audit and legal requirement, not from what storage costs.
Note the difference between deleting and superseding. Evidence is never corrected in place; a corrected collection is a new object with its own timestamp and hash, and the original stays exactly where it was.
The completeness question
The first thing an auditor asks of pipeline output is not "is this accurate" but "is this everything". A collector that silently skips a region, a paginated API that stopped at page one, a source that was offline on the third of the month: each produces a clean-looking artifact describing a subset of reality.
So the pipeline records its own run metadata: what it queried, the window it covered, how many records it expected and how many it got. A run that returned nothing must be distinguishable from a run that never happened.
Zero records and no run look identical in storage unless the pipeline writes down that it ran.
The pipeline is itself a control
It has an owner, a frequency, a failure mode and evidence of its own operation. If the pipeline that produces your access-review evidence fails for six weeks, the control it supports has not operated for six weeks, whatever the last artifact says.
So monitor freshness, not just success: an artifact whose timestamp is older than the collection interval is a failure even when nothing errored.
Tasks
Answers are graded on the server: the answer key never reaches your browser.
Task 1: The helpful cleanup
Scenario sim+80 XP
Name the property the edit destroyed, then the record the pipeline must preserve.
Incident brief
The IAM export lands as a CSV with a trailing blank column and inconsistent date formats. An analyst opens it in a spreadsheet each month, fixes the columns, saves, and uploads the tidy version to the evidence store.
The auditor recomputes the digest of the stored file and compares it to the value the collector logged at export time. They do not match. Asked to re-run the export for March, the team finds the source system only retains 90 days.
The tidy files are accurate. Nobody disputes a single value in them.
Term bank, not all of these are right
chain of custodyintegritycompletenesshashnormalizationretention
1.1Name the property the local edit destroyed. (three words)
1.2Name the pipeline stage where the tidying belongs. (one word)
Task 2: Page one of nine
Scenario sim+80 XP
Name the evidence quality the artifact fails, then the metadata that would have exposed it.
Incident brief
The privileged-access collector has run nightly without error for seven months. Every artifact is hashed, timestamped and stored write-once. The chain of custody is impeccable.
During the audit an engineer notices the artifacts all contain exactly 100 records. The source API paginates at 100 and the collector never followed the `next` link. The real population is 847.
Nothing in the store indicates how many records the collector expected, or that there were more pages.
Term bank, not all of these are right
completenessrun metadataintegrityfreshnesschain of custodysampling
2.1Name the evidence quality the artifacts fail. (one word)
2.2Name what the pipeline must record about its own execution. (two words)
Task 3: Six weeks of nothing
Scenario sim+80 XP
Name the signal that would have caught the outage, then what the control's status truly was.
Incident brief
A credential rotation collector authenticates with a token that expired on 4 March. The job schedules, starts, receives a 401, catches the exception and exits zero because "we did not want noisy alerts".
The evidence dashboard shows the control green throughout, because it reads the most recent stored artifact and one exists: from 3 March.
The gap surfaces on 16 April when someone sorts the store by date.
Six evidence collectors feed the control platform. The dashboard is green because it colours on the `status` column, and status is written by the collector itself.
A collector is only useful if it ran recently and brought data back. Return the `name` of every collector whose most recent run cannot support evidence today.
Table: collectors
name source hours_since_run rows_returned status
okta-users okta 2 412 ok
aws-config aws 30 0 ok
github-prs github 49 900 ok
jira-tickets jira 3 0 error
gcp-iam gcp 1 88 ok
vault-audit vault 6 15 ok
Freshness rule: a collector must have run within the last 24 hours.
Return one column: name.
Overnight, eight evidence collectors reported. The morning summary is green because the orchestrator only counts non-zero exit codes.
Raise the collectors whose evidence should not be trusted this morning. Leave the ones that are fine.
List only the lines that qualify, worst first. Severity is one of: critical | high | medium | low. Add or delete rows as needed.
Collector report
P-1 okta-admins: wrote 0 bytes, exit code 0, previous day's file left in place and re-stamped with today's date.
P-2 aws-config: 4,102 records, checksum recorded, manifest written.
P-3 github-approvals: partial page fetch, stopped at page 3 of 9, exit code 0.
P-4 gcp-iam: completed, took 40 minutes instead of the usual 6.
P-5 jira-tickets: completed, 1,204 records, one field renamed upstream so a column is now null throughout.
P-6 hris-leavers: wrote to the evidence bucket with no run id and no checksum.
P-7 scanner-findings: completed, checksum recorded, manifest written.
P-8 crowdstrike: completed, record count 3% below yesterday.
1
2
3
4
5
6
7
8
9
10
11
12
Checks
The right findings are raised
Benign lines are left out
Severity matches the actual exposure
Ranked worst first
Next on your path · Data & Evidence
Continuous Control Monitoring
Move from point-in-time testing to always-on monitoring with alerting and drift detection.
