A test is circular when its expected answer comes from the method being evaluated. A second risk appears when an examiner learns the intended answer before completing the task. Address both by recording how the reference was established and what information each participant could see. Blinding is a design choice that needs its own audit trail.
Define the decision unit
Specify whether the task concerns a field value, record presence, event sequence, file origin, or another bounded claim. State which outputs count as correct, incorrect, inconclusive, or outside scope. Include a rule for ambiguous inputs before the results are examined. Otherwise, difficult cases can drift out of the denominator while easy examples remain, producing a misleading result.
Construct the reference independently
Keep source-generation records, known actions, native observations, and relevant format documentation. Record which expected values are directly controlled and which are inferred. A scripted action is evidence that a request was issued; confirm the outcome if the test depends on its completion. Preserve failed construction attempts rather than assigning them a convenient label. “Unknown” is a valid reference status when the available records do not settle the answer.
Separate the roles and records
Where feasible, use a dataset custodian to maintain the answer key and an examiner to perform the task. Give each input a neutral identifier that does not reveal its class. Keep the mapping protected until outputs are committed. In a small team, record when one person holds multiple roles and the resulting limits. Do not imply that a nominal blind removes information already known from dataset construction.
Manage context deliberately
The OSAC Technical Series 0002R1 framework, revised February 2019 discusses contextual effects and circumstances where blinding or sequential unmasking may help reduce bias. For this protocol, list the information necessary to perform the task and withhold irrelevant outcome cues. Record each later disclosure, its time, and the reason it was needed. Compare the committed initial finding with any revised interpretation.
Adjudicate without rewriting the trial
Preserve the original answer before investigating a discrepancy. Review source construction, label quality, ambiguous instructions, and tool behavior separately. If the reference is wrong, correct it with a versioned explanation and show which results change. If the task definition changes, call it a revised study. Quietly adjusting the key to match an expected pattern defeats the purpose of an independent reference.
Publish the exposure and limits
Describe who knew the key, what contextual information was visible, and whether repeated inputs or recognizable sources could reveal answers. Report inconclusive cases and exclusions with reasons. State the population of inputs represented by the study. An internally blinded demonstration can identify a defect without supporting an operational error rate across unrelated systems.
- Define a bounded validation question
Set acceptance criteria before observing results.
- Choose meaningful denominators
Distinguish independent cases from repeated measurements.