How to Test Operating Effectiveness of Controls

Testing the operating effectiveness of controls means confirming that a control actually ran the way it was supposed to, every time it was supposed to, throughout the period under review. Under PCAOB Auditing Standard AS 2201, that confirmation has two parts: the control performed as designed, and the person performing it had the competence and authority to do so.¹ The work combines inquiry, observation, inspection, and reperformance against a complete population of instances where the control should have operated. Testing is risk-based, not formula-driven: higher-risk controls demand more evidence.

One boundary to set first. Operating effectiveness is not design effectiveness. Design asks whether a control, if executed perfectly, would catch or prevent the risk it targets. Operating effectiveness asks whether it actually did. A wire-transfer approval policy can be well-designed and still fail in operation if supervisors rubber-stamp approvals without reviewing documentation. Both must hold before a control provides real assurance, and design is usually evaluated first through walkthroughs before any operating tests are meaningful.¹

Set Up the Test Before You Touch a Sample

What makes test results reliable is the setup work that happens before selection begins. Skip it and every conclusion downstream is suspect.

Build a Complete Population

Assemble every transaction or event during the period that should have triggered the control. This is the reference set the sample will come from, and completeness is the point. If 50 wire transfers occurred during the quarter but only 45 appear on the list, the missing five may be exactly where the control failed. An incomplete population lets successful items get cherry-picked and hides the failures the test is meant to find.

Anchor to Standard Operating Procedures

The company’s SOPs define what the control performer is supposed to do, what evidence they should generate, and how exceptions get escalated. Without that written benchmark, there is no objective standard against which actual performance can be judged. The SOPs are what “operated as designed” is measured against.

Verify Information Produced by the Entity

This is one of the most common places testing goes wrong. When the evidence for a control’s operation is a report the company itself generated, the reliability of that report has to be established before it can be relied on. PCAOB AS 1105 requires testing the accuracy and completeness of information produced by the entity, or testing the controls (including relevant IT general controls and automated application controls) that ensure it, and confirming the information is precise and detailed enough for the audit’s purposes.² If the population report used to select the sample was itself incomplete, the sample and every conclusion from it are unreliable.

Select the Sample

Once the population is confirmed, pull a representative sample. PCAOB AS 2315 requires that every item in the population have an opportunity to be selected, and identifies random-based selection and haphazard selection as acceptable approaches.³ Random sampling, stratified random sampling, and systematic sampling (such as picking every tenth item) are all valid. The purpose is to avoid any bias that would steer selection toward instances where the control was more likely to have worked.

The Four Testing Procedures

Operating effectiveness testing uses four procedures. Each produces a different strength of evidence, and in practice evaluators combine them.

Inquiry. Interview the personnel who perform the control. Ask what triggers it, what steps they take, what they do with exceptions. Inquiry alone is the weakest evidence, because people often describe what they are supposed to do rather than what they actually do.
Observation. Watch the control performer execute the task in real time. Observing a staff member review and approve a wire transfer confirms the authorization step actually happens.
Inspection. Examine documents and system logs for concrete markers of completion: timestamps, digital signatures, supervisor initials. This produces tangible evidence that the control ran on a specific item at a specific time.
Reperformance. Independently execute the control and see whether you reach the same conclusion the original performer did. If reperforming a bank reconciliation surfaces the same variance the preparer documented, the control worked. A discrepancy indicates failure regardless of what the sign-off says.

Reperformance provides the strongest evidence because it tests the substance of the control rather than confirming that someone went through the motions. For high-risk controls, relying only on inquiry or inspection without reperformance leaves real exposure.¹

Three attributes are what these procedures are ultimately testing for. Did the person performing the control have the competence and organizational authority to do it properly? Did the control run every time it was supposed to, at the frequency set during design? And were the four categories of responsibility, custody of assets, recording of transactions, reconciliation of records, and authorization of activity, kept separate enough that no single person could handle a transaction end-to-end and conceal a problem?¹ Smaller organizations that cannot fully segregate all four functions often rely on more detailed oversight by the audit committee as a compensating control.

How Much Testing Is Enough

PCAOB standards do not prescribe exact sample sizes. AS 2201 takes a risk-based approach: the greater the risk associated with a control, the more evidence needed that it is working.¹ Several factors shape the risk assessment and therefore the extent of testing.

Frequency of operation. A daily control generates far more instances than a quarterly one, so the sample must be larger to be representative. Common professional practice starts at roughly 25 to 30 items for daily controls, around 5 to 8 for weekly controls, 2 to 3 for monthly controls, and a single item for annual controls. These are starting points, not rules.
Materiality of potential misstatement. Controls that prevent or detect errors large enough to be material to the financial statements need more extensive testing than controls over low-dollar processes.
History of errors. Accounts with a track record of misstatements or prior-year control failures warrant deeper testing.
Degree of judgment involved. Controls that require subjective evaluation or complex calculations carry more risk than straightforward matching or approval steps, because judgment introduces variability.
Reliance on other controls. A control that depends on IT general controls or on the output of another control inherits risk from those dependencies. If the underlying controls are weak, the dependent control needs heavier testing.

Automated Controls and Benchmarking

Automated controls behave differently from manual ones because software does not have off days. Once the logic of an automated application control is verified, and the supporting IT general controls over program changes, access, and computer operations are confirmed effective, a benchmarking strategy becomes available. The auditor establishes a baseline by testing the automated control once, then in later years verifies that the control has not changed rather than repeating the full test.¹

Benchmarking works best when the application is stable with few changes between periods, when strong program change controls exist, and when the automated control can be matched to a defined program. The strategy collapses if IT general controls are weak. An automated three-way match in accounts payable is only as reliable as the access controls preventing someone from modifying the matching parameters.

Using Prior-Year Evidence

In subsequent years, knowledge from previous audits can factor into testing decisions. If a control tested well last year and nothing has changed in its design, the people performing it, or the volume and nature of transactions it processes, the risk may be assessed as lower and testing reduced accordingly.¹ AS 2201 also requires varying the nature, timing, and extent of testing from year to year to introduce unpredictability. Testing different controls at different interim periods, changing sample sizes, or shifting the mix of procedures all serve that purpose.

Classifying What the Test Finds

Not every control failure carries the same weight. PCAOB standards define three levels of severity, and the classification determines who must be told and what follows.

Control deficiency. The baseline category. A deficiency exists when the design or operation of a control does not allow personnel to prevent or detect misstatements on a timely basis in the normal course of their duties. This covers both design deficiencies (a needed control is missing or would not meet its objective even if perfectly executed) and operational deficiencies (a properly designed control does not run as intended, or the person performing it lacks the necessary qualifications).⁴
Significant deficiency. A deficiency, or combination of deficiencies, less severe than a material weakness but important enough to warrant attention from those overseeing financial reporting. These must be communicated in writing to management and the audit committee.
Material weakness. A deficiency, or combination of deficiencies, where there is a reasonable possibility that a material misstatement in the annual or interim financial statements will not be prevented or detected on time. “Reasonable possibility” means the likelihood is either reasonably possible or probable.⁴

Several individually minor deficiencies that affect the same account or process can combine into a significant deficiency or material weakness even when no single one would qualify alone. The PCAOB specifically flags ineffective audit committee oversight as an indicator that a material weakness exists.

Testing for Management Override

Every internal control system has a built-in vulnerability: the people who design and oversee the controls can also bypass them. Management sits in a unique position to manipulate accounting records and prepare fraudulent financial statements by overriding controls that otherwise appear to operate effectively.⁵ Because override can happen in unpredictable ways, specific procedures address this risk regardless of what other testing reveals.

AS 2201 identifies several areas where controls should specifically target override risk: controls over significant unusual transactions, particularly those producing late or unusual journal entries; controls over period-end adjustments; controls over related party transactions; and controls around significant management estimates.¹ At smaller companies, where senior management is more directly involved in day-to-day accounting, the risk is higher, and more detailed audit committee oversight is often the compensating mechanism.

Internal controls have inherent limitations because they involve human judgment and compliance. Collusion and management override can circumvent even well-designed systems. The goal of the testing work is not to eliminate that risk but to gather enough evidence, at the right depth for each control’s risk level, to know whether the safeguards actually held during the period.

1
Public Company Accounting Oversight Board. AS 2201 – An Audit of Internal Control Over Financial Reporting That Is Integrated with An Audit of Financial Statements
2
Public Company Accounting Oversight Board. AS 1105 – Audit Evidence
3
Public Company Accounting Oversight Board. AS 2315 – Audit Sampling
4
Public Company Accounting Oversight Board. AS 1305 – Communications About Control Deficiencies in an Audit of Financial Statements
5
Public Company Accounting Oversight Board. Consideration of Fraud in a Financial Statement Audit – AU Section 316.57