Write a test plan an auditor can rerun

Objectives

A control owner submits a test plan: "Sample 25 changes and confirm they were approved." The population is 8,000 changes a quarter, the sample was picked by the owner himself, and the pass criterion is "mostly approved".

Write a test plan with a defined population, a stated sampling method, a sample size, and an unambiguous pass criterion. If the control can be tested against the whole population, say so instead of sampling.

changes(quarter) = 8,142 rows
each row: id, merged_at, approver, author, approved (bool)
the API returns every row; no sampling required

Auditor's standing question: "how did you choose
the items you tested, and could I reproduce it?"

Step-by-step hints

The guided walkthrough, in full. It is free: no XP cost, and it does not touch the paid hint ladder or the reveal gate.

1. Step 1

   Re-read the brief and name the trap

   Three separate problems are bundled into one sentence: an arbitrary sample size, a selection made by the person being tested, and a pass criterion made of an adverb.

   The trap is that the plan looks like every test plan you have seen before, so the instinct is to tune it rather than replace it.

2. Step 2

   Read the fixture

   The fixture tells you the API returns every row and that no sampling is required. That single line changes what a correct plan looks like.

   It also gives you the exact quarter count and the fields available per row: enough to define a population precisely.

3. Step 3

   Read the starter

   Four values are blank or zero. The tester is already filled in: check whether it conflicts with anything in the brief before you leave it alone.

4. Step 4

   Define the population

   A population is countable and bounded: which records, from which system, over which window. Ask: could someone else reconstruct exactly the same set from my description?

   Population is defined and countable

5. Step 5

   Choose a method, then size it

   Given that the API returns everything, sampling is a choice to prove less for the same effort. Decide the method first; the size then follows from it rather than being picked out of the air.

   Ask: does my sample size follow logically from my method, or would a reader wonder where the number came from?

   Sampling method is stated and reproducibleSample size matches the method

6. Step 6

   Make the criterion binary

   A pass criterion must be evaluable without judgement. Express it as a count or a threshold: something a nightly job can compute and an auditor cannot argue with.

   Pass criterion is unambiguous

7. Step 7

   Independence

   Whoever runs the test must not be the party whose work is being tested. Re-read who owns this control in the brief, then re-read the tester value in the starter.

   Tester is not the control owner

8. Step 8

   Run and read the failures

   If the size check fails while the method check passes, the two disagree with each other: fix the number to match the method you chose, not the other way round.

Rules summary

• XP on pass

  35 bonus XP the first time every check goes green.

• Paid hints

  2 available. Each costs 25% of the lab XP (about 9 XP).

• Reveal solution

  Scores 0 XP and unlocks only after 3 check runs.

• Attempt cap

  60 check runs per learner. The runner warns you in the last 10.

• Drafts

  Your editor content is saved in this browser, so a reload will not lose work.

• Walkthrough

  Always free and never penalised: use it before spending a hint.