Where the Error Bar Ends

A factory makes a shaft with a specified diameter of 10.00 millimeters and an acceptable tolerance of plus or minus 0.10.

Inspecting the shaft sounds like the easy part.

Imagine that its true diameter is 10.08 millimeters, inside the tolerance but close to the upper limit. If the inspection system has unbiased, normally distributed error with a standard deviation of 0.04 millimeters, a reading will exceed 10.10 about 31 percent of the time.

The factory may reject an acceptable part. If enough readings cross the limit, someone may conclude that production is drifting and adjust a machine that was working correctly.

Manufacturing has a way to separate those possibilities. A Gage R&R study examines variation introduced by the inspection process. Repeatability concerns the spread of repeated readings under the same conditions. Reproducibility concerns what changes across operators, instruments, days, setups, or other conditions. NIST’s treatment also considers stability, bias, resolution, linearity, and drift. The point is not simply to own a precise gauge. It is to understand the system that produces the reading.

That is already useful. The stranger problem appears when the thing we care about cannot be placed in a gauge.

Four layers hiding in one number

Imagine a university evaluating a degree program.

Students take exams intended to show whether they learned the curriculum. Six months after graduation, the university asks whether they found work in their field. Strong placement numbers may appear in recruiting material. Weak ones may prompt questions about courses, faculty, or whether the program prepares students adequately.

Several different questions are now hiding under the word measurement.

It helps to separate four layers. The target is what we want to know. The instrument is how we observe something about it. The result is the number or classification the instrument produces. The inference is what we conclude from that result.

On the factory floor, the target may be shaft diameter. A gauge is the instrument. The reading is 10.08 millimeters. “This shaft is within tolerance” is the inference.

The links are short, but they still deserve scrutiny. A gauge with too much variation can turn acceptable parts into apparent defects.

At the university, the links separate.

An exam can be scored consistently while doing a mediocre job of capturing whether students can use what they learned. Psychometrics distinguishes consistency from validity. The professional standards for educational and psychological testing describe validity as evidence supporting a particular interpretation of test scores for a proposed use. It is not a permanent property stamped onto a test. A score may support one conclusion and be weak evidence for another.

The graduate survey adds a different set of uncertainties. Suppose 82 percent of respondents say they are working in their field. Did enough graduates respond? Were unemployed graduates harder to reach? Did respondents interpret “in my field” in the same way?

Survey researchers place problems such as coverage, nonresponse, question effects, sampling, and processing within the broader frame of total survey error. A survey can be carefully designed and still contain uncertainty that does not fit into one headline number.

But even a flawless survey would not settle the university’s larger question.

Suppose every graduate is contacted and classified without error. The university now knows the placement rate remarkably well. It still does not know, from that result alone, how much the curriculum caused it. Employment also responds to the labor market, geography, prior experience, hiring cycles, professional networks, and students’ willingness to relocate.

The placement rate may be an excellent answer to “What happened to this year’s graduates?” It is weaker evidence for “How much did this curriculum improve their prospects?”

Nothing has gone wrong with the number. The claim has moved beyond it.

A local kind of precision

Numerical uncertainty often arrives with visible boundaries. A dimension is reported with a tolerance. A poll comes with a margin of error. Once a bound has been calculated, the result can feel contained.

But the bound belongs to a particular part of the evidence chain.

In a probability poll, the familiar margin of sampling error describes uncertainty from observing a sample instead of the full target population, with adjustments depending on the survey design. It does not automatically include nonresponse bias, question wording, coverage problems, or an incorrect model of who will vote. The American Association for Public Opinion Research makes this distinction explicit.

The ± figure can be correct. The mistake begins when its precision is transferred to a larger claim.

A three-point sampling margin does not mean that every assumption between “these people answered this question” and “this is what will happen on Election Day” is also known within three points. The result carries quantified uncertainty. The inference may rest on uncertainties that have no convenient symbol attached to them.

The same pattern appears in educational assessment, product analytics, employee surveys, performance ratings, and machine-learning benchmarks. The part that is easiest to quantify can lend authority to conclusions the calculation never tested.

An error bar is local. It describes uncertainty in a specified result under stated assumptions. It does not expand to cover every proxy, interpretation, or causal story built above it.

The harder case is not a broken instrument. It is an instrument working exactly as intended while the claim made from its result quietly changes the question.

Sources and further reading

• NIST/SEMATECH e-Handbook: Gauge R & R studies.
• NIST/SEMATECH e-Handbook: Variability in a measurement process.
• Standards for Educational and Psychological Testing: Validity.
• AAPOR: Standard Definitions and total survey error.
• AAPOR: Polling Accuracy.
• U.S. Census Bureau: Measures of Nonsampling Error.

Featured image: Photo by Antoni Shkraba on Pexels.