Write tests that can actually fail: negative cases, missing-field cases and assertions on the decision rather than on the run.
50 min 210 XPSkills: Testing, Policy as code, CI
Compliance code is code, and untested code lies
Once a control is a rule in a repository, it inherits every property of software, including the property that it can be confidently wrong. A policy that never denies anything and a policy that denies everything both run cleanly and both produce a green pipeline.
The test suite is what converts a rule into a control. Without it you have automated the execution of a claim, not the verification of a condition.
Fast rule tests prove logic. Only a real run proves the control operates.
Every rule needs a case that fails
A test suite that only contains violations proves nothing about specificity: an always-deny rule passes all of it. A suite that only contains clean inputs proves nothing about sensitivity: an always-allow rule passes all of that.
The minimum viable suite for any rule is three cases: a violation it must catch, a compliant input it must leave alone, and an input with the field missing entirely. The third is where most policy engines quietly default to allow.
Assert on the decision, not on the run
The most common false pass in compliance code is an assertion that the rule evaluated without error. That test stays green when the rule returns the wrong answer, when it returns nothing, and when the fixture it evaluated was empty.
Assert on the output: this input produces this decision, with this message. And fail the suite when the fixture directory is empty, because a suite that iterates over zero cases reports success by definition.
Green means "the tests passed". It never means "the policy is correct" unless a failing case exists to prove the tests can fail.
Three green suites, none of which tests the thing anyone cares about.
Fixtures, regressions and drift
Keep a fixture library of real, redacted evidence snapshots: the odd shapes production actually produces, not the tidy examples in the documentation. Every incident becomes a fixture: when a rule misses a violation in the wild, the first commit of the fix is the case that reproduces it.
Then guard against drift in the source data. When an API adds a field or renames a value, a rule can start passing everything without a single line of it changing, so schema assertions on the fixture belong in the same suite as the policy assertions.
Tasks
Answers are graded on the server: the answer key never reaches your browser.
Task 1: Nine cases, zero fixtures
Scenario sim+70 XP
Name what the pipeline produced, then the input the suite never had.
Incident brief
The bucket-exposure policy ships with nine tests and a green badge in the README. The suite globs `fixtures/*.json`, loops over the matches, and asserts each produces no deny.
In CI the fixtures directory is populated by a step that was removed during a refactor four months ago. Zero files match. The loop body never executes. Nine tests report as passing.
During the same period two public buckets went live and neither was caught.
Term bank, not all of these are right
false passfixtureregressionassertioncoverageflake
1.1Name the outcome the pipeline produced. (two words)
1.2Name the artefact the suite never actually had. (one word)
Task 2: The rule that denies everything
Scenario sim+70 XP
Name the missing kind of case, then the technique that proves a suite can fail.
Incident brief
A refactor of the encryption policy inverts a condition. Every one of the fourteen tests still passes, because all fourteen supply a non-compliant resource and assert that a deny is produced.
The rule now denies compliant resources too. The first symptom is a platform team's deployment pipeline blocking every storage change in staging, which is triaged as a tooling outage rather than a policy defect.
The policy author points out the suite has "full coverage of the control".
Term bank, not all of these are right
negative casemutation testingregression testcoveragefixturesmoke test
2.1Name the kind of case the suite lacks. (two words)
2.2Name the technique that proves a suite can fail. (two words)
Task 3: It ran, therefore it passed
Scenario sim+70 XP
Name what the test should assert on, then the class of statement being misused.
Incident brief
The access-review policy test calls the evaluator inside a try block and marks the case successful when no exception is raised. It has never failed in eleven months.
Halfway through the audit period a schema change renames `role` to `role_name`. The rule now reads an undefined field, matches nothing, and returns an empty result set. No exception is raised.
The control reports zero violations for the rest of the period, and the test suite agrees.
Term bank, not all of these are right
decisionassertionexceptionschemacoverageexecution
3.1Name what the test should have asserted on. (one word)
3.2Name the class of statement that was misused. (one word)
Labs
80 bonus XP
Hands-on work. You edit the artefact, the server runs the checks: the expectations and the reference solution never reach your browser.
A policy author asks you to "just add a test" for the public-bucket rule. The existing suite has nine cases, every one of them a violation, and it asserts that evaluation completed without raising.
Write the minimum suite that can distinguish a correct rule from a broken one: cases that name their input and their expected decision, a violation, a compliant input, an input with the field under test absent, assertions targeting the decision rather than the run, and a suite that refuses to pass on an empty fixture set.
Rule under test and the inputs production actually produces
rule: storage.deny_public_bucket
denies a bucket when acl == "public-read"
denies a bucket whose policy grants read to every principal
observed inputs:
{ "acl": "public-read", "encryption": "aes256" } # violation
{ "acl": "private", "encryption": "aes256" } # compliant
{ "encryption": "aes256" } # acl absent: engine default applies
incident INC-4471: a bucket with no acl field was allowed through.
A team submits a policy test suite for review with a green badge and a note that coverage is 100%. Your job is to decide which cases give the organisation real assurance and which produce a pass that means nothing.
Accept the entries that could genuinely fail if the rule broke. Reject the rest.
Verdicts are accept or reject. Every rejection needs one reason code from: empty-fixture | asserts-execution-not-decision | no-negative-case | default-allow-untested.
policy_tests/: entries under review
G-1 A pair of cases on the same rule: one violation expecting deny, one compliant input expecting allow.
G-2 A suite that globs fixtures/*.json and loops; the directory is empty in CI and the suite reports pass.
G-3 A case supplying an input with the field under test absent, expecting deny.
G-4 A case that wraps the evaluator in try/except and passes when no exception is raised.
G-5 Fourteen cases on the encryption rule, every one of them a violation expecting deny.
G-6 A rule that falls through to allow on unrecognised input shapes, with no case exercising that path.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Checks
Every line in the fixture carries a verdict
Accept and reject land on the right lines
Each rejection cites a valid reason code
Next on your path · Automation & Code
Build an Evidence Pipeline, Step by Step
Build a working GRC evidence pipeline end to end: collectors for identity, cloud, ticketing, CI and HR sources, a normalised evidence record, policy code you write yourself, custody, scheduling and reporting.
