Script against Okta, Jira and cloud APIs for evidence: authentication, pagination, throttling and proving the population was complete.
55 min 220 XPSkills: APIs, Automation, Evidence
The evidence loop
Almost every automated control is the same loop: authenticate to a source system, pull a population, normalise it into a common record, evaluate an assertion, and store the result with enough metadata to re-identify the run. The frameworks differ; the loop does not.
Write the loop once and control implementations become configuration: which endpoint, which fields, which assertion. Write it per control and you have forty scripts, each with its own way of failing.
A collector that cannot say when it ran, what it asked for, and how many rows came back is not producing evidence. It is producing a file.
One loop, many sources. The run id and the hash are what make the output evidence.
Credentials are a control, not a config value
The fastest way to get a collector working is a personal API token from whoever is writing it. The token inherits that person's permissions, survives their departure, and appears in the audit trail as them, so every automated action looks like a human action by someone who was on holiday.
Collectors authenticate as a service identity with least privilege, with the secret held in a vault and referenced, never inlined. Rotation is scheduled, and the collector fails loudly when the secret is invalid rather than silently returning an empty population.
Pagination and throttling decide whether your population is real
APIs return pages. A loop that forgets to follow the cursor returns the first hundred rows and no error, and every downstream metric is computed on that slice. Throttling does the same thing with a different mechanism: a run that hits 429 and gives up has collected a partial population that looks exactly like a complete one.
The defences are unglamorous: follow every page, retry with exponential backoff and a budget, record the expected minimum population, and refuse to write an evidence record that fell short of it.
Both failures produce output. Only one of them announces itself.
Make the run auditable
Store what you asked for as well as what you got: the endpoint, the query parameters, the timestamp in UTC, the record count and a hash of the canonical payload. An assessor's real question is not "is this true" but "can you show me this was produced the way you say it was".
Alert on absence too. A collector that stops running is indistinguishable, on a dashboard, from a control with no findings, so treat a missing run as a failure of the control it supports.
Tasks
Answers are graded on the server: the answer key never reaches your browser.
Task 1: One hundred perfect users
Scenario sim+75 XP
Name the API behaviour the script ignored, then what the evidence misstates.
Incident brief
The MFA coverage control reports 100% for three consecutive months, which nobody questions because the identity team ran a push campaign in January.
The collector calls `GET /api/v1/users?limit=100` and evaluates what comes back. The tenant has 1,240 users. The response carries a `link: rel="next"` header the script never reads.
The control owner notes the query "returns exactly what the API gives it".
1.1Name the API behaviour the script ignored. (one word)
1.2Name what the evidence record now misstates. (one word)
Task 2: Authenticated as a leaver
Scenario sim+75 XP
Name the identity the collector should use, then the missing lifecycle activity.
Incident brief
The change-approval collector has run nightly for fourteen months. It authenticates with a token created by an engineer who left in March; her account was disabled, but the token was issued at the org level and still works.
Every automated evidence pull for the audit period appears in the source system's audit log as an action by a terminated employee. The token has never been replaced and has admin scope because that was easiest at the time.
The platform lead calls this "fine, it is a machine using it".
Term bank, not all of these are right
service accountrotationleast privilegepersonal tokenvaultrevocation
2.1Name the identity the collector should authenticate as. (two words)
2.2Name the lifecycle activity that was never scheduled. (one word)
Task 3: Nothing to report
Scenario sim+70 XP
Name the retry strategy that was missing, then the detection that would have caught it.
Incident brief
The cloud configuration collector begins hitting 429 responses after the estate doubles. The script retries immediately, three times, then exits zero with whatever it has.
The control dashboard shows no findings for five weeks. Nobody investigates, because no findings is what a healthy control looks like. In week six an engineer notices the evidence records for the last month each contain about a fifth of the expected resources, and two runs produced no record at all.
The dashboard has a green tile labelled "last checked: recently".
An engineer wired up the MFA evidence collector in an afternoon using her own API token and a single unpaginated request. It reports 100% coverage and the identity lead is delighted with "the automation".
Rewrite the collector specification so the run is trustworthy: a machine identity with a vaulted secret reference, pagination that follows every page, throttling handled with a real backoff strategy, a population floor the run must clear, run metadata and a payload hash on every record, and a schedule a scheduler understands.
Current collector, as deployed
source: okta-api (tenant has 1,240 users; the endpoint returns 100 per page)
auth: personal token belonging to an engineer, pasted into the job config
request: GET /api/v1/users?limit=100 # 'link: rel="next"' header is never read
on 429: retried immediately three times, then exits 0 with whatever it has
stored: the raw payload only: no run id, no hash, no record count
runs: "when someone remembers"
The evidence collectors ran overnight and the operations review is in twenty minutes. You may escalate only the entries that compromise the evidence or the identity the collectors run as.
Everything else goes on the backlog. Raise what qualifies, worst first.
List only the lines that qualify, worst first. Severity is one of: critical | high | medium | low. Add or delete rows as needed.
Overnight collector run log: review notes
N-1 Okta collector stopped after page 1 of 7; the quarter's evidence record is marked complete.
N-2 Run log timestamps are written in local time rather than UTC.
N-3 Collector authenticates with a departed engineer's personal API token, which still works and has admin scope.
N-4 Job duration rose from 25 to 40 minutes after the estate grew.
N-5 429 responses retried immediately, no backoff; three runs abandoned mid-collection with no alert.
N-6 Job name still references the pre-rebrand project code.
N-7 Evidence records store the payload but no run id, so a record cannot be traced to its run.
N-8 Output bucket lifecycle keeps data for 8 years; policy requires 7.
1
2
3
4
5
6
7
8
9
10
11
12
Checks
The right findings are raised
Benign lines are left out
Severity matches the actual exposure
Ranked worst first
Next on your path · Automation & Code
Testing Compliance Code
Write tests that can actually fail: negative cases, missing-field cases and assertions on the decision rather than on the run.
