Systematic Error, Controls and the Design of a Measurement
Independent repeats can reduce random uncertainty of a mean, but a shared systematic offset remains. Calibration, null tests, controls and blinding address different risks of bias. An uncertainty budget records the contributions and the rule used to combine them.
What a learner can do afterwards
- Separates random from systematic contributions in a described experiment
- Proposes a null test or a blind analysis for a stated risk of bias
- Builds an error budget and identifies which term dominates
1 · Read
For independent repeats with stable, zero-mean random errors, averaging reduces random uncertainty of the mean. It does not reduce the scatter of individual readings. A fixed systematic offset shifts their common center and remains in the mean. Systematic effects can also depend on conditions, such as a scale factor that changes the size of each reading.
Error is a difference from the true value, which is usually unknown. A reference standard helps estimate that difference, subject to its own uncertainty. Here, absolute error means the magnitude of the difference from the reference. For a positive reference value, absolute relative error in percent is 100 × absolute difference / reference value. This difference is not an uncertainty bound.
A null test uses conditions under which the model predicts zero signal. Random noise can still produce nonzero readings. Compare the estimated signal with its uncertainty. A persistent departure larger than expected from noise calls for investigation, not an automatic verdict of bias. A result compatible with zero does not rule out every systematic effect; the test checks only effects to which it is sensitive.
Calibration compares readings with reference standards and can reveal or correct response bias; correction uncertainty remains. A control group supplies a comparison baseline. Blind analysis hides treatment identities while analysts make choices, then reveals them after the analysis is fixed. This reduces expectation-driven choices. It does not change instrument noise or sample size, and other bias can remain.
An uncertainty budget lists sources, their sizes and their effects on the reported result. Correct known signed offsets first, then budget the uncertainty that remains. Here, entries are upper bounds in common output units: add them for a conservative total, not a standard deviation. Other models use different rules. If each source can be reduced by the same fraction at the same cost, reducing the largest contribution lowers this total most.
For these output bounds, the conservative total is 0.8 + 0.3 + 0.1 = 1.2 g. Calibration contributes most.
Using fertilizer A only on wet plots and B only on dry plots confounds fertilizer with moisture: their effects cannot be separated. Divide the field into moisture zones and randomly assign both fertilizers within each zone. Keep other procedures comparable. Randomization reduces allocation bias but does not guarantee balance in one trial. An untreated group gives a no-fertilizer baseline; it is not required just to compare A with B.
Each moisture zone contains both treatments, so treatment is no longer assigned solely by moisture.
Averaging can reduce random uncertainty of a mean. Check systematic effects with suitable controls, and combine uncertainty contributions using a stated model.
Original lesson, questions and flashcards written for this learning dataset.
Readings and flashcards written for this learning dataset. Linked videos by TED-Ed and James A. Smith, and by Fizzics Organisation.
More to read
Choose the comparison your question needs
Imagine comparing the drying times of two coatings. Every sample of coating A sits on a warm shelf, while every sample of coating B sits on a cool shelf. A difference in drying time could reflect coating, temperature or both. Repeating this arrangement does not separate those effects.
Use both coatings at each shelf condition. Randomly assign comparable samples within each condition and keep preparation and timing procedures alike. Random assignment reduces systematic allocation bias. Chance differences can remain in one experiment.
Choose the comparison group for the question. Uncoated samples give a baseline for asking what a coating changes relative to no coating. Comparing A with B does not always require an uncoated group. Neither comparison repairs an inaccurate timer; that needs a measurement check.
When people participate in research, measurement planning must include their protection. Address informed consent, privacy and the balance of risks and possible benefits before recruitment. Follow the oversight requirements that apply to the study. A design that improves statistical precision does not justify unacceptable harm.
Trace the choices behind a data table
Before interpreting a table, trace how its entries were selected and measured. A large number of entries can make an estimate precise while leaving a consistent selection or measurement bias in place.
For a questionnaire, inspect wording and who did not respond. A question may suggest an answer, and people who respond may differ from those who do not. For observations, consider whether awareness of the study could change behavior. For existing records, examine missing entries, instrument changes and changes in category definitions.
Choose a check that can reveal the specific concern. Compare wording versions, investigate nonresponse or compare measurements made before and after an instrument change. Write down what the check should show and how uncertainty will affect its interpretation.
Analysis choices also need planning. Where feasible, hide treatment identities while deciding how to handle exclusions and ambiguous measurements. Fix the analysis before revealing identities, and report departures from the plan. Blinding reduces opportunities for expectation-driven decisions. It does not establish that selection, instruments or records are free of bias.
Keep reference differences and uncertainty separate
A reference check compares an instrument reading with a documented standard. Their difference estimates error under the tested conditions. The standard also has uncertainty, so the check does not reveal an exact true value.
Suppose a sensor reads 52 units for a reference value of 50 units. The signed difference is +2 units. Its magnitude is 2 units, giving an absolute relative error of 100 × 2 / 50 = 4%. A percentage formed by dividing by a zero reference value is undefined.
A correction changes the reported value; uncertainty describes what remains unknown. Subtracting an estimated +2 offset may improve the result, but uncertainty in the reference and correction remains. Repeated readings can help estimate random variability without removing a common offset.
Check the range and conditions that matter. One reference value can reveal a difference there, but cannot establish the response slope over a range. Several reference values can help distinguish an offset from a scale-factor problem. Record the calibration conditions and carry the remaining uncertainty into the measurement budget.
Check a distance comparison before collecting data
Suppose a small cart launcher has two settings to compare on a straight track. Start each trial with the cart's front at the same marked origin. After launch, mark where its front first comes to rest after moving. Measure the forward distance from the origin to that trial's stopping mark. The endpoint rule stays the same, but its location can vary. Use the same cart, surface and reading procedure for both settings.
Randomize the order of settings rather than always testing one first. Record every planned trial, including short runs. Repeated trials help estimate trial-to-trial variability. They do not remove a common mistake in the distance origin.
Check the ruler against a reference distance and inspect its zero mark. A separate null check of the reading procedure can use a configuration with zero displacement. Interpret small nonzero readings using the measurement uncertainty rather than declaring bias from any departure from zero.
For the final difference between settings, consider how each uncertainty source affects both results. A shared additive offset cancels if it enters both distances equally. A changing offset may not cancel, and a scale-factor error changes the difference. State and test these assumptions. List the remaining output contributions and select a combination rule that matches the uncertainty model.
Explore more
2 · Watch
Take it off screen
Where it sits
Learn first
Optional connections
Where this leads
8 questions wait behind this lesson, each with its answer explained. Every answer feeds the sky: stars light as they are learned, and dim when it is time to come back.