Declared state versus observed state, detection latency, suppression windows, and alerts a human can actually act on at 3am.
50 min 210 XPSkills: Drift, Alerting, Monitoring
Brief
Drift is the gap between declared state and observed state. Someone edits a security group in the console, a break-glass role is never revoked, a bucket policy is loosened for a migration and never tightened. The control definition still says the right thing; reality no longer matches it.
Detecting drift needs three ingredients: an authoritative declaration, a periodic observation, and a comparison whose output is a difference rather than a boolean.
Drift is not a failure of the control. It is a failure of everything after it.
Declared versus observed
The declaration can be Terraform state, a control catalog, an approved baseline or an IdP group definition: what matters is that it is version-controlled and someone owns changes to it. The observation is whatever the live API returns right now.
The comparison must emit the diff: which resource, which attribute, declared value, observed value, first seen. An alert that says 'drift detected in production' sends a human on an expedition; an alert carrying the diff sends them to a line.
The output of drift detection is a diff, not a boolean.
Detection latency is a control property
A daily check means a misconfiguration can live for twenty-four hours; a weekly one means seven days. That window is not an implementation detail, it is part of what the control promises, and it belongs in the control definition next to the frequency.
Where drift is unacceptable, move from polling to events: cloud audit logs, IdP webhooks and admin-action streams turn a one-day window into seconds. Use polling as the completeness backstop, because event streams drop messages.
An alert is finished when the person receiving it knows what to do next.
Designing an alert someone will act on
Alerts that fire on every occurrence of a known, accepted condition train people to ignore the channel. Deduplicate by resource and attribute, suppress within an approved exception window, and always name an owner and a runbook.
Suppression is where discipline is lost. A suppression with no expiry is a silently disabled control, so every suppression needs an owner, a reason and an end date that something enforces.
If an alert has no owner, it has no response. If a suppression has no expiry, it has no end.
Tasks
Answers are graded on the server: the answer key never reaches your browser.
Task 1: Drift detected in production
Scenario sim+70 XP
Name what the alert must carry, then the property being compared against.
Incident brief
The nightly job pages an on-call engineer with the message 'drift detected in production (3 resources)'. There is no list of resources and no indication of what changed.
The engineer spends ninety minutes diffing Terraform plans by hand and finds two intentional changes merged that afternoon and one security group opened to the internet.
The team's fix is to move the alert to a lower-priority channel.
1.2Name the side of the comparison held in version control. (two words)
Task 2: Muted since last April
Scenario sim+70 XP
Name the missing attribute of the suppression, then what the suppression became.
Incident brief
During a migration the team suppressed the public-bucket drift alert for one project, with a Slack thread agreeing it was temporary.
Fourteen months later an external researcher reports a public bucket in that project. The suppression rule is still in place; it has no owner field, no reason and no end date.
The monitoring dashboard has shown the control as healthy throughout, because suppressed findings never reached it.
2.2Name what the suppression turned the control into. (two words)
Task 3: Twenty-three hours of open
Scenario sim+70 XP
Name the property that must be tightened, then the detection source that would do it.
Incident brief
An engineer opens a database security group to the internet at 09:10 to debug a customer issue and closes it at 09:40. The nightly drift job runs at 02:00 and sees nothing.
The team argues the control worked: the group was compliant at every scheduled check.
The audit log recorded both changes within seconds of each other.
The current monitor pages on-call with 'drift detected in production (3 resources)' and nothing else. Last night that cost ninety minutes of manual diffing to find one security group open to the internet among two intentional changes.
Specify it properly: compare against version-controlled declared state, emit a diff, state the detection latency the control promises, route to an owner with a runbook, and make suppressions expire.
What exists today, and what the platform can provide
current alert: "drift detected in production (3 resources)"
available:
terraform_state : version-controlled declaration of intended configuration
cloud_api : live configuration, pollable
cloud_audit_log : event stream, seconds behind the change
oncall_rotation : platform-security
runbooks : runbook://drift/security-group
incident: a security group was opened at 09:10 and closed at 09:40;
the 02:00 nightly poll saw nothing.
1
2
3
4
5
6
7
8
9
10
11
12
Checks
Declared state comes from version control
The monitor emits a diff, not a boolean
Detection latency is tightened past a nightly poll
Overnight drift detection produced eight entries and you are on call with one pair of hands. Escalate only what represents a control that is not currently operating.
Everything else waits for the standup. Raise what qualifies, worst first.
List only the lines that qualify, worst first. Severity is one of: critical | high | medium | low. Add or delete rows as needed.
Drift queue: overnight entries
K-1 Production database security group open to 0.0.0.0/0, first seen this morning, still open.
K-2 Suppression on the public-bucket alert has been in place 14 months with no owner, reason or end date.
K-3 Break-glass admin role granted during an incident nine days ago and never revoked.
K-4 The team would prefer the drift job ran hourly rather than nightly.
K-5 Two intentional infrastructure changes merged yesterday afternoon appear in the diff.
K-6 Alerts route to a channel renamed last month that now has no members.
K-7 The alert message contains a spelling error in the runbook link text.
K-8 Cost-allocation automation changed resource tags exactly as declared in code.
1
2
3
4
5
6
7
8
9
10
11
12
Checks
The right findings are raised
Benign lines are left out
Severity matches the actual exposure
Ranked worst first
Next on your path · Data & Evidence
Vendor Risk Automation
Tier vendors by the data they actually touch, turn questionnaires into scored data, and ingest vendor evidence on a schedule instead of chasing it by email.
