Reading a paper’s conclusion is rarely a passive act. We follow the authors’ chain of inference and decide whether to believe it. A decision can be right or wrong, but its correctness is not identical to the underlying fact or law; it reflects the state of knowledge at a given moment. That is why science need not agonize over every error. In a sense, research proceeds through error. The same data can yield quite different conclusions at different times. Inference is a function of time: it approaches the truth rather than unveiling it. In any decision process, error can be described probabilistically, and much of the popularity of the p-value comes from offering a general decision rule and threshold. That rule brings the familiar pair: false positives and false negatives.
The baseline problem
True and false need a contrast. In most hypothesis tests, the null hypothesis supplies that contrast, usually a distribution or a random condition. Much criticism of hypothesis testing begins here, because in real experiments and observations the baseline often is not distributional or random. Statisticians have developed many ways to balance out the nonrandom part so that randomness can serve as a baseline. A difference test built on that adjustment can then yield a convincing, relatively correct conclusion. In the cautious wording of conclusions, 'relatively correct' is the ideal; 'convincing' is what gets a paper published. Many readers miss this distinction, and that misreading is a kind of anti-scientific reading of the literature.
Balancing out the nonrandom part requires knowing in advance what that part is. Unfortunately, observational research often cannot know this in advance; sometimes the whole point is to discover unknown unknowns. In such cases, hypothesis tests based on p-values or null hypotheses should not be used. Suppose you observe that gene A is associated with disease X. Whether A causes X still needs to be tested by controlling variables. It is quite possible that gene B regulates both A and X, but B was never measured, so the study is incomplete. Would measuring the unknown together with the known through omics make it complete? Not necessarily. As the number of measurements grows, so does the number of hypothesis tests. If the p-value threshold is 0.05, then among 10,000 measured variables, 500 genes will appear different even under random measurement. Someone suggested last year that the threshold be set at 0.005. That does not solve the problem; it only reduces the number of findings that need verification, which has some value. For example, if one of 10,000 genes is real, a 0.05 threshold might produce 501 findings, while 0.005 might produce 51, so fewer need checking. But in real research you may encounter a situation where 0.05 finds 501 and 0.005 finds only 50: a real difference is missed because its effect size, or the difference it produces, is too small for your decision method. You may even see 0.05 find 480 and 0.005 find 48. When the effect you are observing is small, no adjustment of the p-value may be able to find it. The fault is not in the p-value; it lies in studying an effect that is too small with an inappropriate method and hypothesis. This is a type M error, and it is hard to avoid whenever you run many hypothesis tests.
Strong conclusions and noisy measurements
A tempting rule is to trust only strong conclusions: a phenomenon that exists objectively should survive any reasonable statistical method, and its conclusion should not flip when the decision method changes. That ideal now looks too optimistic. Strong conclusions are indeed strong, often obvious, and they are the low-hanging fruit that earlier researchers have mostly picked. If an effect is strong enough, a p-value will find it and a Bayesian method will find it too; the effect-size problem disappears. But most facts and regularities are buried in what we currently call randomness or noise. Our analysis is just barely able to separate a suspected signal from noise, and whether that separation is reliable remains a mystery. Statistics cannot settle it here; technical progress becomes the key. Some studies hope that data mining will solve the problem of discovering phenomena within a field. Data mining helps with obvious but overlooked patterns. For high-noise data, however, reducing measurement noise will do more for conclusions than any search for a clever statistical method that can detect differences. Data superstition shows you pseudo-patterns; advances in measurement reveal valuable regularities. I once wanted to quantify my life completely, but later realized that errors in measurement and sensing can turn quantified data into garbage. Big data is beautiful, but it can also be empty.
Direction is not magnitude
The other problem is direction. A p-value is often a two-sided probability focused on the middle part of a distribution, so when you see a very small p-value you still do not know whether the effect is larger or smaller; you have to look at the effect estimate. Reporting a p-value without an effect size is like saying the weather will change tomorrow but not telling anyone how. In most experimental designs, some change is almost certain. If I knock out a gene to verify its function, there will surely be a difference between the gene change and the function; much of this comes from observational experiments. What matters more is the size of the effect, and judging that requires domain expertise, not a simple p-value. If a science student studies for years and in the end knows only to use a p-value to judge a conclusion, the degree is not worth much. This is a type S error, and confirmatory experiments need to watch for it especially closely.
Exploratory results in disguise
I was reminded of all this by a paper I read toward the end of last year. It measured the concentrations of many pollutants and then correlated them with many health indicators. This is a many-to-many selection of results, common in omics research and, admittedly, a standard move in much environmental health work: report only the results that are statistically significant. I forwarded the paper to Andrew Gelman at Columbia and asked whether he had any advice from a data-analysis perspective. He said he would reply publicly on his blog in about half a year; his schedule was very full. Months later, I saw the response. In summary, Gelman argued that significance testing is unreliable, that Bayesian hierarchical models can reduce the impact of type M errors—an approach I find quite moderate—and that researchers should balance out as many known effects as possible. More importantly, he pointed out that this was exploratory analysis, not confirmatory analysis, so the conclusions should not be trusted too much. The reply was fair, but whether frontline researchers can understand and apply it is another question.
If we quantify the degree to which someone understands the current problems of research as 'research literacy,' training in this area is seriously lacking in China. From students to teachers, career utility often matters far more than understanding the research itself; there is a missing atmosphere. If you read Gelman’s reply, you will find that the comments under the blog post opened up further discussion about reporting research results, open-access journals, and related issues. In domestic science blogs, by contrast, rational discussion is often scarce and simple judgments are common. Building that atmosphere will take a generation or several, including you and me. Open public discussion is one of the research dividends of the internet age. The statistical fixes matter, but the culture around them matters more.