Most statistical errors that reach publication are not sophisticated. They are not failures of advanced modelling or subtle violations of distributional assumptions. They are four recurring mistakes, each of which can be described in a paragraph and avoided without any mathematics beyond arithmetic.
The four are: treating a p value as a measure of how big or how important an effect is; reporting significance without ever stating the size of the difference; counting technical replicates as though they were independent samples; and running an experiment too small to detect the effect being looked for, then interpreting the resulting null result as evidence of no effect.
If you got through your training by running whatever test the previous postdoc ran, this is the material you skipped. It is worth an afternoon, because these errors do not merely weaken a paper. They change conclusions. An experiment analysed with pseudoreplication can produce a confident, publishable, entirely wrong result, and nothing about the figure will look suspicious.
Key takeaways
- A p value answers a question about data given a hypothesis, not about a hypothesis given the data.
- Effect size with a confidence interval carries the information a p value cannot: how large the difference is and how well it is pinned down.
- Technical replicates measure your pipette; biological replicates measure your biology, and only the latter counts toward n.
- Underpowered experiments do not merely fail to find effects; when they do find them, the effects are exaggerated.
- Error bars mean nothing unless the caption says which of the three common quantities they show.
What a P Value Does and Does Not Say
A p value is the probability of observing data at least as extreme as yours, assuming the null hypothesis is true and assuming the model behind the test is correct. Every word in that sentence is doing work, and the assumption is the part that gets dropped in conversation.
Read carefully, the p value is a statement about data given a hypothesis. It is not a statement about the hypothesis given the data. It does not tell you the probability that the null is true, it does not tell you the probability that your result will replicate, and it does not tell you the probability that you have made a mistake. Those are the three things people most often believe it says.
The consequences follow directly. A very small p value does not mean a large effect. With a sufficiently large sample, a difference too small to matter biologically will produce a decisively small p value, because the test is asking whether the difference is distinguishable from zero, not whether it is worth caring about. Conversely, a p value just above the conventional cut-off does not mean the effect is absent. It means this experiment, at this size, did not produce evidence strong enough to reject the null under the chosen convention.
The 0.05 threshold is a convention with no biological basis whatsoever. It was proposed as a rough rule of thumb for deciding which results merited a second look, and it hardened into a publication gate. Treating 0.049 and 0.051 as categorically different outcomes is indefensible on any statistical grounds, and yet whole literatures are shaped by exactly that distinction.
Effect Size as the Quantity That Matters
The effect size is the answer to the question the experiment was actually asking: how much did the treatment change the outcome?
It comes in two useful forms. Raw effect size is expressed in the units of measurement, such as a difference of so many nanograms per millilitre, or a ratio such as a doubling of expression. Standardised effect size expresses the difference in units of the variability in the data, which lets you compare across assays measured in different units. For bench work, the raw form is usually more informative, because you know what the units mean.
Neither number is worth much without a confidence interval. The interval expresses the range of effect sizes reasonably compatible with your data, and it does what a p value cannot: it separates a null result caused by a genuinely small effect from a null result caused by an uninformative experiment. Consider two experiments that both fail to reach significance. In the first, the estimated difference is near zero with a narrow interval, so the data rule out anything large. In the second, the estimated difference is substantial but the interval spans everything from a large decrease to a large increase. These are entirely different findings, and both would be reported as “no significant difference” if only p values were given.
Effect sizes also force a question that significance testing lets you avoid: what size of change would matter? A twofold change in a signalling protein may be biologically decisive. A five percent change in the same protein may be undetectable in any downstream process. Deciding this before the experiment, and stating it, is one of the most useful pieces of discipline available at the bench, and it is a prerequisite for sizing the experiment sensibly.
Technical Versus Biological Replicates
This is the error with the greatest capacity to invert a conclusion, and it is depressingly easy to make.
A technical replicate is a repeated measurement of the same biological material. Three wells loaded from the same lysate, three injections of the same extract, three qPCR reactions from the same cDNA. What varies between them is pipetting, instrument noise, well position, and the rest of the measurement process. Averaging them gives you a better estimate of that one sample’s value.
A biological replicate is an independent instance of the thing you are making claims about. A separate animal, a separate donor, a separate culture passaged and treated independently, a separate colony. What varies between them is the biological variation you are trying to generalise across.
The inferential unit must match the claim. If the claim is that a drug lowers expression in cells of a given type, the unit of replication is the independently treated culture, not the well. Treating three wells from one flask as n equals three inflates the apparent sample size and shrinks the apparent variability, because the wells share every source of biological variation. The result is a standard error far smaller than the truth and a p value far smaller than it should be.
| Layer | What varies | Contributes to | Typical mistake |
|---|---|---|---|
| Technical replicate | Pipetting, instrument, position | Precision of one sample’s value | Counted as independent n |
| Biological replicate | Individual animal, donor, culture | Generalisability of the finding | Too few of them run |
| Independent experiment | Day, batch, reagent lot, operator | Robustness across conditions | Only one performed |
The correct handling is straightforward: average technical replicates to get one value per biological unit, then analyse across the biological units. You lose nothing by doing so, because the averaging is exactly what technical replication is for. The number of technical replicates belongs in the methods; the number of biological replicates belongs in the figure legend as n.
Power and Choosing a Sample Size
Power is the probability that an experiment will detect an effect of a given size, if that effect is real. It depends on four things: the size of the effect you want to be able to detect, the variability of your measurement, the number of independent units, and the significance threshold you intend to use.
Three of those are fixed or chosen. The fourth, sample size, is the one you control, and a power calculation simply rearranges the relationship to tell you how many units you need. The inputs are the honest part. You need a defensible estimate of your assay’s variability, which usually means pilot data or previous experiments with the same protocol, and you need to state the smallest effect worth detecting.
Bench science frequently runs on n equals three, and the reason is historical and practical rather than statistical. Three is the smallest number that lets you compute a standard deviation and draw an error bar. For a very large, very clean effect in a low-variability assay, three may genuinely suffice. For a modest effect in a noisy assay, three has little chance of detecting anything, and running it anyway is a way of guaranteeing an uninformative result.
The consequence of low power that surprises people is not the false negatives. It is what happens to the false positives. In an underpowered experiment, the only way a difference reaches significance is if the random noise happens to push it far from zero. That means every significant result from a low-powered study is, by construction, an overestimate of the true effect, sometimes by a large factor. This is why effects shrink as they are replicated in larger studies, and it is why an exciting result from a tiny experiment should raise caution rather than confidence.
Multiple Comparisons and Inflated Findings
Every test you run at a threshold of 0.05 carries roughly a one in twenty chance of a false positive when nothing is going on. Run twenty independent tests on null data and you should expect one to come out significant. Run a hundred and you should expect five.
This scales badly in modern laboratory work. A transcriptomics experiment tests tens of thousands of genes. A screening plate tests hundreds of compounds. A time course with several treatments and several readouts quietly accumulates dozens of comparisons before anyone counts them. Without correction, the list of hits is dominated by noise.
Two families of correction exist. Controlling the family-wise error rate, of which the Bonferroni adjustment is the simplest example, aims to make the probability of even one false positive across the whole family small. It is conservative, and in a genomewide analysis it is so conservative that real effects are lost. Controlling the false discovery rate instead accepts that some proportion of the reported hits will be false and bounds that proportion. For discovery work that will be followed by validation, the false discovery approach is usually the better fit, because the cost of a false positive is a wasted follow-up rather than a wrong conclusion.
The subtler version of the problem is not a list of tests but a series of choices. Deciding which outliers to exclude after seeing their effect on the result, choosing between normalisation methods based on which gives a cleaner figure, stopping data collection when the p value drops below threshold, or splitting the data by a variable you noticed afterwards, all inflate the false positive rate without producing any countable list of comparisons. The defence is to write the analysis plan before the data exist, and to report every analysis performed, not only the one that worked.
Error Bars and What They Represent
An error bar with no explanation is decoration. The same data can produce three quite different bars, and the difference between them is not cosmetic.
Standard deviation describes the spread of the individual values. It does not shrink as you add samples; it converges on the true variability of the population. Use it when you want to show how variable the underlying thing is.
Standard error of the mean describes the precision of your estimate of the mean. It equals the standard deviation divided by the square root of the sample size, so it shrinks as n grows. This makes it the smallest and therefore the most flattering of the three, which is precisely why it is the most commonly plotted and the most commonly misread. A tight standard error bar drawn over three technical replicates conveys a confidence that the experiment does not support.
A confidence interval describes the range of values compatible with the data at a stated level. It is the most directly interpretable, because it maps onto the question the reader is asking.
The relationship between error bars and significance is not the simple one people assume. Overlapping standard error bars do not imply a non-significant difference, and non-overlapping ones do not guarantee significance; the answer depends on sample sizes and on whether the comparison is paired. Reading significance off a figure by eye is unreliable in both directions.
Choosing a Test to Match the Design
The test follows from the design, not the other way round. Three questions determine most of the choice.
First, are the observations paired? If each unit contributes a measurement under both conditions, such as the same culture before and after treatment, a paired test removes between-unit variability and is substantially more powerful. Analysing paired data as though it were unpaired discards that advantage and can hide a real effect.
Second, how many groups are being compared? Two groups call for a two-sample comparison. Three or more call for an analysis that handles them together, followed by defined post-hoc comparisons. Running all possible pairwise tests between several groups without adjustment is a multiple comparison problem in disguise.
Third, does the data plausibly meet the assumptions? Parametric tests assume approximate normality of the residuals and, in their simplest form, comparable variance between groups. Small samples make normality impossible to assess reliably from the data itself, so the reasonable approach is to think about the measurement. Counts, ratios and concentrations spanning orders of magnitude are typically skewed, and a log transformation often makes them well behaved and simultaneously turns multiplicative effects into additive ones, which is usually what you meant anyway. Where transformation does not help, rank-based alternatives make fewer assumptions at some cost in power, and resampling approaches are increasingly practical.
The structure of the experiment matters as much as the comparison. If samples were processed in batches, or across several days, or by different operators, that structure is a real source of variance and can be included in the model rather than ignored. A design where every treated sample was run on Monday and every control on Tuesday cannot be rescued by any test, because treatment and day are confounded.
Frequently asked questions
Is n equals three ever defensible?
It can be, but only when the effect is large relative to the assay’s variability and the three units are genuinely independent. The way to find out is a power calculation using variability from your own pilot data rather than an assumption inherited from a previous protocol. If three units give you a low chance of detecting the smallest effect you would care about, running the experiment anyway produces a result that cannot distinguish a real absence of effect from an uninformative measurement.
Should I test my data for normality before choosing a test?
Formal normality tests are of limited use at typical bench sample sizes. With few observations they have almost no power to detect real departures, and with many observations they flag trivial ones. A more reliable approach is to think about what you measured: concentrations, counts and fold changes are usually skewed and often become well behaved after a log transformation, while differences between paired measurements are frequently closer to symmetric. Plotting the data is more informative than testing it.
What do I do about an outlier that is clearly wrong?
Distinguish a documented technical failure from an inconvenient value. A well with a visible bubble, a sample that clotted, or a run where the instrument logged an error can be excluded on those grounds, provided the rule is applied consistently and stated in the methods. A value that is merely extreme is data. Removing points because they weaken the result, and keeping equally extreme points that strengthen it, is one of the most effective ways to generate a false positive without noticing.
Why do my results get weaker when I repeat the experiment at larger scale?
This is the expected behaviour of effects first detected in small experiments. When power is low, only results that noise has pushed far from zero can cross the significance threshold, so the published estimate is systematically larger than the truth. Repeating at a larger size samples the effect more accurately and the estimate falls back toward its real value. A shrinking effect on replication is not necessarily evidence that the original finding was fabricated or that the repeat was flawed.
Does correcting for multiple comparisons make it impossible to find anything?
It depends on which correction and which question. Family-wise methods such as Bonferroni are stringent by design and are appropriate when a single false positive would be costly, as in a confirmatory test of a small number of pre-specified hypotheses. False discovery rate methods are better suited to discovery work with thousands of tests, because they bound the proportion of reported hits that are false rather than the chance of any false hit at all, which preserves far more genuine signal for follow-up.
The practical route through all of this is unglamorous but effective. Decide what the unit of replication is before you start. State the smallest effect that would matter. Estimate variability from a pilot and size the experiment to detect that effect. Write down the analysis you will run. Then report the effect size with its interval, say what the error bars are, and give n as the number of independent biological units. Doing those six things puts an experiment ahead of a substantial fraction of the published literature, and none of them require statistics beyond what fits on one page.
