Design vs. operating effectiveness, sampling, populations, and how to write a control that can actually be tested by a machine.
40 min 220 XPSkills: Control design, Testing
Brief
Auditors ask two separate questions about every control, and practitioners collapse them into one constantly. First: if this control worked exactly as written, would it actually stop the risk? That is design effectiveness. Second: did it in fact work, every time it was supposed to, across the whole period? That is operating effectiveness.
A control can pass one and fail the other in either direction. A beautifully operated control that reviews the wrong thing is a design failure with perfect evidence. A well-designed control whose owner went on leave for a quarter is an operating failure with a flawless control description.
Design is about the control statement. Operation is about the population. Confusing them produces remediation plans that fix nothing.
The two gates
Design is tested by inspection and walkthrough: read the control, trace one instance, ask whether the described activity would catch the failure it claims to catch. Operation is tested by sampling: pull items from the period and check each one against defined attributes.
This ordering matters. There is no point sampling twenty-five instances of a control that could not have worked even if performed perfectly. Auditors who sample first and think later produce thick workpapers and weak assurance.
Two gates, in order. Failing the first makes the second irrelevant.
Populations, completeness and sample size
Operating effectiveness testing starts with a population: the complete set of times the control should have operated. If the population is wrong, every conclusion drawn from the sample is wrong too, which is why the first thing a good auditor challenges is not your sample, it is the completeness of the list the sample came from.
Sample sizes follow control frequency rather than population size. Daily controls are typically tested at twenty-five instances, weekly at eight, monthly at three, quarterly at two, annual at one. A ten-million-row population and a four-hundred-row population of the same daily control get the same twenty-five.
The sample is the small part. Completeness of the population is where the assurance actually comes from.
Writing a control a machine can test
A testable control statement names four things: who performs it, what they do, how often, and what record it leaves behind. Drop the frequency and there is no population. Drop the record and there is no evidence. Drop the actor and nobody is accountable when it lapses.
Once those four are explicit, the control is one short step from automation: the frequency becomes a schedule, the record becomes a query, and the attributes you would have checked by hand become assertions in code.
"Access is reviewed regularly" is not a control. It is an aspiration with no population, no evidence and no owner.
Key controls, compensating controls and deviations
Not every control carries the same weight. A key control is one whose failure alone would let a significant risk through; auditors test these first and hardest. Supporting controls add depth but are not relied on alone.
When a key control fails, an organisation can point to a compensating control: a different control that addresses the same risk to a similar degree. It must cover the same risk, the same population and the same period. An alert on something unrelated does not compensate.
Each failed instance in a test is a deviation. One deviation in a small sample can be enough to say the control did not operate effectively; the tester then asks whether something else covered the risk.
A compensating control must cover the same risk, the same people and the same time.
Tasks
Answers are graded on the server: the answer key never reaches your browser.
Task 1: Perfect evidence, wrong question
Answer challenge+50 XP
Two answers: the gate that failed, and the gate that passed.
Incident brief
The quarterly access review runs like clockwork. Every quarter, on schedule, the IT manager exports a list of users, signs it, and files it. Twelve consecutive quarters, twelve signed exports, zero missed dates.
During an incident review you discover the export lists only users created in the quarter. Anyone provisioned before the review period has never appeared on any list, including three contractors who left in 2023 and still hold production access.
The IT manager objects: "The control has never failed. Every review was performed on time and I can show you the signature for each one."
1.1Name the type of effectiveness this control fails. (one word)
1.2Name the type of effectiveness the twelve signed exports genuinely demonstrate. (one word)
Task 2: Twenty-five out of what
Scenario sim+50 XP
Attack the list before you attack the sample.
Incident brief
Change management testing. The engineer exports approved changes from the ticketing tool, hands over 4,120 rows, and the auditor selects 25. All 25 have approvals. The auditor is ready to conclude.
You notice the export filter: status = Closed AND approval_state = Approved. Emergency changes are closed with status Expedited and never carry an approval_state, so none of them are in the export at all. There were 212 last year.
The engineer's view: "Twenty-five is the standard sample size for a daily control, and all twenty-five passed. If they want emergency changes tested they should have asked for a bigger sample."
2.1Name the property of the exported list that the filter destroyed. (one word)
2.2Name the auditor's term for the complete set of control instances the 25 should have come from. (one word)
Task 3: The unfalsifiable control
Scenario sim+60 XP
Name what is missing, then name the test technique it blocks.
Incident brief
Control CTL-118 reads, in full: "Security logs are reviewed regularly by the security team and anomalies are investigated."
Asked for evidence, the team lead sends a Slack screenshot from March where someone says "looks quiet today". Asked how many reviews should have happened in the period, he says "depends on the week". Asked what a reviewer must check, he says "anything unusual".
He then argues the control cannot be failed: "There's no rule we broke. Nothing in the wording says how often, so we're compliant with it by definition."
A control owner submits a test plan: "Sample 25 changes and confirm they were approved." The population is 8,000 changes a quarter, the sample was picked by the owner himself, and the pass criterion is "mostly approved".
Write a test plan with a defined population, a stated sampling method, a sample size, and an unambiguous pass criterion. If the control can be tested against the whole population, say so instead of sampling.
What the change system can tell you
changes(quarter) = 8,142 rows
each row: id, merged_at, approver, author, approved (bool)
the API returns every row; no sampling required
Auditor's standing question: "how did you choose
the items you tested, and could I reproduce it?"
An internal tester has closed six control tests as "effective" and wants your review before the pack goes to the audit committee.
Accept the conclusions the evidence actually supports. Reject the ones where the test proves something other than what was claimed: the wrong period, the design rather than the operation, a sample that cannot represent the population, or a conclusion with no stated pass criterion.
Verdicts are accept or reject. Every rejection needs one reason code from: tests-design-not-operation | sample-not-representative | no-pass-criterion | evidence-outside-period.
Test log (tester's own words)
T-1 Change approval: sampled 25 of 8,142 merges at random; 25 had a recorded approver. Criterion: zero unapproved merges. Effective.
T-2 Access review: confirmed the review procedure document exists and is signed by the CISO. Effective.
T-3 Backup restore: full population of 12 monthly restores; 12 completed within RTO. Criterion: all restores within RTO. Effective.
T-4 MFA enforcement: sampled 10 admins the platform lead selected as representative; all had MFA. Effective.
T-5 Log retention: evidence exported on 14 Jan for a period ending 31 Dec. Effective.
T-6 Vulnerability remediation: reviewed 40 findings; "most were closed promptly". Effective.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
Checks
Every line in the fixture carries a verdict
Accept and reject land on the right lines
Each rejection cites a valid reason code
Next on your path · GRC Fundamentals
The Audit Lifecycle
Scoping, walkthroughs, evidence requests, exceptions and the report. Triage a live PBC list in the scenario sim.
