Nobody's sample of 25 was ever statistical
Conventional sample sizes are a customary quantity of work, not a statistical conclusion. That was a reasonable trade when opening 25 files took a day. It is now a choice to test less than you could — and it is increasingly a choice you will be asked to defend.
Ask an auditor why the sample was 25 and you will usually get one of three answers: it is what we did last year, it is what the methodology says, or it is what the external auditor accepts. What you will rarely get is a confidence level and a tolerable deviation rate, which is what would make it a statistical sample.
This is not a criticism of anyone. Attribute sampling is a real discipline and some teams apply it properly. But the honest description of most control testing is that a conventional quantity of work is performed, and the conclusion is drawn with a confidence nobody computed.
That was a defensible trade for a long time. Opening 25 files took a day. Opening 40,000 took a career. The sample was not a statistical choice; it was a throughput constraint wearing statistical language, and everyone involved understood the arrangement.
The constraint is gone and the language stayed
Where the records are digital and reachable, testing every item costs approximately what testing 25 costs, because the work is a query rather than an afternoon. At that point continuing to sample is a decision to test less than you could.
Sometimes that decision is right. The control is manual, the evidence is on paper, the data lives in a system nobody can get an extract from without a project. Fine — say so, and say it in the file. What has stopped being defensible is arriving at 25 by default and describing the result as though the population had been considered.
What actually happens when you stop
The first full-population test a team runs is usually the most uncomfortable week of their year, and it is worth being honest about why.
The work does not reduce. It relocates. A sample of 25 with one exception is a short conversation. A population of 40,000 with 1,900 exceptions is a triage problem. The skill you need shifts from performing procedures to categorising failure — which of these share a root cause, which are data quality rather than control failure, which are genuine deficiencies, and which reveal that the control was never designed for the population it is applied to.
The conclusion gets sharper than people expect. A sample supports an inference. A population supports a statement. Teams accustomed to hedged conclusions sometimes find the precision unwelcome, particularly when the finding is that a control everyone believed was operating has been failing quietly for three quarters. It is generally the third quarter that causes the meeting.
The prior year gets a look it did not ask for. This is the one nobody warns you about. If the control has been failing at 5% all along, the sample of 25 had a decent chance of missing it every year — and somebody will ask why it was clean before. The answer is arithmetic, not negligence, but it needs preparing.
Where to start, specifically
Not with your hardest control. Start where the data is reachable and the exceptions are unambiguous:
- Privileged access. Every account with elevated rights, matched to an approval. No judgment required to spot an exception, and the population is sitting in a directory service.
- Change management. Every production change matched to an approved ticket, with the developer-deployer separation checked. Same property: reachable data, mechanical exception logic.
- Terminations. Every leaver matched to access removal within the policy window. This is the one that finds the accounts nobody closed.
All three are ITGC, all three are miserable by sample, and all three are the areas where an unexpected exception most reliably appears. They are also, not coincidentally, the tests most worth writing as re-runnable definitions, because you will want them monthly once you have seen the first result.
The part that does not automate
Worth saying, because the argument for population testing gets oversold in the other direction.
Of the 138 workflow templates we publish, 16 are third-line assurance work. The other 122 belong to the first and second lines. Assurance is a thin layer on a very large operating surface — which is exactly why auditors cannot test their way to coverage by hand, and also why the leverage is in reading the first line’s systems rather than in auditing faster.
But the walkthrough, where somebody finally tells you how the process actually runs, remains the most valuable hour of most engagements. Population testing tells you what happened. It does not tell you why, and it will not tell you about the manual workaround that has been in place since the system upgrade. Keep the hour.
In practice
Population testing only compounds if the test survives the engagement. A test written as a structured definition — population, source, criteria, exception logic, approver — re-runs next period without being rebuilt; a test written as a spreadsheet gets rebuilt every year by whoever inherits it. CoworkCanvas treats the definition as the durable object and the run as the disposable one.
Evidence