Dos and don’ts of statistics in research

Writing and Reviewing Research Papers

Introduction

Motivation

Visualization Pitfalls

Common Graphical Fallacies

  • Excessive chartjunk
  • Truncated axes
  • Cherry-picked time ranges

Bad vs. Good Example

Excessive Chartjunk

Code
using Plots
using Plots.PlotMeasures
theme(:dao)

# Data
regions = ["North", "South", "East", "West"]
sales = [150, 120, 180, 90]

# Plot 1: Clean Bar Chart
p1 = bar(regions, sales,
    legend = false,
    xlabel = "",
    ylabel = "Sales (in K)",
    title = "Regional Sales",
    color = :dodgerblue,
    framestyle = :box,
    grid = false,
    linewidth = 0.8,
)

# Plot 2: Excessive Chart Junk
p2 = bar(regions, sales,
    color = [:red, :green, :purple, :orange],
    legend = false,
    label = ["North" "South" "East" "West"],
    xlabel = "Regions",
    ylabel = "",
    title = "SALES REGIONS",
    seriestype = :bar,
    linecolor = :black,
    fillalpha = 0.6,
    xrotation = 45,
    bar_width = 0.9,
    linewidth = 2
)

# Show side-by-side (bad plot on the left, as in the other examples)
plot(p2, p1, layout = (1, 2), titlefont=font(10, "Arial"), size=(1100, 400), margin = 5mm, bottom_margin = 10mm)
Precompiling packages...
    608.8 ms  ✓ QuartoNotebookWorkerJSONExt (serial)
  1 dependency successfully precompiled in 1 seconds
Precompiling packages...
   1583.4 ms  ✓ QuartoNotebookWorkerPlotsExt (serial)
  1 dependency successfully precompiled in 2 seconds

Bad vs. Good Example

Truncated Axes

Code
using Plots
methods = ["Method A", "Method B", "Method C"]
accuracy = [91.2, 92.0, 93.1]   # differences of less than 2 percentage points

# Bad: the y-axis starts at 91, so a 2% difference looks like a 5x difference
p1 = bar(methods, accuracy, legend=false, title="Bad Plot", ylabel="Accuracy (%)",
    ylims=(91, 93.5), color=:dodgerblue)
# Better: bar lengths are proportional to the values
p2 = bar(methods, accuracy, legend=false, title="Better Plot", ylabel="Accuracy (%)",
    ylims=(0, 100), color=:dodgerblue)

plot(p1, p2, layout=(1, 2), size=(1000, 400), margin=5mm)

Bad vs. Good Example

Cherry-Picked Time Ranges

Code
using Plots
x = 1:20
y = [2, 33, 14, 15, 26, 7, 8, 9, 6, 11, 10, 13, 15, 15, 14, 17, 21, 19, 20, 22]

p2 = plot(x[11:20], y[11:20], lw=3, title="Bad Plot", xlabel="Time", ylabel="Measurement")
p1 = plot(x, y, lw=3, title="Better Plot", xlabel="Time", ylabel="Measurement")
plot(p2, p1, layout=(1, 2), size=(1000, 400), margin=5mm)

Dos and Don’ts

Do:

  • Use clear, honest plots
  • Show uncertainty (error bars, CIs)
  • Label axes and legends

Don’t:

  • Use deceptive scales or chartjunk
  • Omit uncertainty intervals
  • Cherry-pick data ranges

Dos and Don’ts

“Above all else, show the data.” — Edward Tufte (Tufte 1983)

Statistical Inference Refresher

A Refresher on Statistical Inference

Statistical inference is used to test hypotheses:

  • Null and alternative hypotheses
  • Types of error (Type I and Type II)
  • p-value

Null and Alternative Hypotheses

Null hypothesis: No effect or no difference.

Alternative hypothesis: Effect or difference.

Example:

  • Null hypothesis: The vaccine has no effect.

  • Alternative hypothesis: The vaccine has an effect.

Do:

  • state the null and alternative hypotheses.
  • make the null hypothesis the status quo.
  • make sure that the null and alternative hypotheses are mutually exclusive.

Types of Error

Type I error: Rejecting the null hypothesis when it is true.

Type II error: Failing to reject the null hypothesis when it is false.

Example:

  • Type I error: Jail an innocent person.

  • Type II error: Free a guilty person.

p-value

  • The probability of observing data at least as extreme as those observed, assuming that the null hypothesis is true.

  • It is used to test hypotheses: a small p-value means that data like ours would be unlikely if the null were true.

  • (For historical reasons) It is compared to a threshold, usually 0.05 or 0.01.

p-value: (fair) coin toss example

  • We toss a coin 30 times and observe 20 heads. Is the coin fair? \(H_0: \pi = 1/2\) vs. \(H_1: \pi \neq 1/2\).
Code
T = 30
no_heads = 20
Z = collect(0:T)
π0 = 1/2

f(x) = binomial(T, x) * π0^x * (1-π0)^(T-x)

# At least as extreme as 20 heads, in either direction: X ≥ 20 or X ≤ 10
extreme = [abs(z - T*π0) >= abs(no_heads - T*π0) for z in Z]
p_one = sum(f.(Z[Z .>= no_heads]))
p_two = sum(f.(Z[extreme]))

bar(Z, f.(Z), color=ifelse.(extreme, :red, :lightsteelblue), label="",
    xlabel="Number of heads", ylabel="Probability under the null",
    title="Binomial(30, 1/2): one-sided p = $(round(p_one, digits=4)), two-sided p = $(round(p_two, digits=4))",
    ylims=(0, 0.16), size=(1000, 450), margin=5mm)

How p-Values Behave

  • Under \(H_0\), p-values are uniformly distributed between 0 and 1: p < 0.05 happens 5% of the time by design.
  • Under \(H_1\) they pile up near zero. How much depends on the power of the test.
Code
using Random, Statistics, HypothesisTests

Random.seed!(1)
nsim = 10_000
p_null = [pvalue(EqualVarianceTTest(randn(30), randn(30))) for _ in 1:nsim]
p_alt  = [pvalue(EqualVarianceTTest(randn(30) .+ 0.5, randn(30))) for _ in 1:nsim]

h1 = histogram(p_null, bins = 0:0.05:1, normalize = :probability, color = :lightsteelblue, label = "",
    title = "No effect (null true)", xlabel = "p-value", ylabel = "Proportion", ylims = (0, 0.6))
vline!([0.05], color = :red, linestyle = :dash, linewidth = 3, label = "0.05", legend = :topright)
h2 = histogram(p_alt, bins = 0:0.05:1, normalize = :probability, color = :steelblue, label = "",
    title = "Effect of 0.5 SD, n = 30 per group (power ≈ $(round(mean(p_alt .< 0.05), digits = 2)))",
    xlabel = "p-value", ylims = (0, 0.6))
vline!([0.05], color = :red, linestyle = :dash, linewidth = 3, label = "0.05", legend = :topright)
plot(h1, h2, layout = (1, 2), size = (1100, 420), margin = 5mm)

Misinterpretation of Metrics

Common Pitfalls

  • The p-value is not the probability that the null hypothesis is true, nor the probability that the result is due to chance (Wasserstein and Lazar 2016).

  • A high R² does not mean the model is relevant or that it predicts well out-of-sample (R² never decreases when regressors are added, so it can be inflated by overfitting).

  • (Pearson) correlation only captures linear relationships. Strong non-linear relationships can have a correlation close to zero.

  • The mean is sensitive to outliers. The median is more robust.

  • Confidence intervals are not the same as prediction intervals. A CI is a range of plausible values for a parameter (e.g., the mean), while a PI is a range for a future observation, and is therefore wider.

  • A 95% CI does not contain the parameter with 95% probability. 95% of the intervals constructed this way, over repeated samples, contain the true parameter.

  • Statistical significance does not imply practical significance. A small p-value does not mean the effect is large or important.

p-Hacking

What is p-Hacking?

Selecting analyses based on significance

What is p-Hacking?

Examples

  • Trying many models or variables

  • Selective reporting

  • Data dredging (searching for significant results without a prior hypothesis)

p-Hacking in Action

  • Assume you have a dataset with 20 variables \[(X_1, X_2, \ldots, X_{20}) \] and you want to test if any of them are significantly associated with an outcome variable \(Y\).

  • If none of them is truly associated with \(Y\) and you test each variable separately at a significance level of 0.05, the probability of finding at least one significant result by chance (with independent tests) is:

\[ 1 - (1 - 0.05)^{20} \approx 0.64 \]

p-Hacking in Action

  • Regressing an outcome on each of 20 unrelated (independently simulated) variables, one at a time, and reporting the p-values.
Code
using Random, DataFrames, GLM, Statistics

Random.seed!(12345)

n = 100  # number of observations
p = 20  # number of predictors

# Simulate outcome and independent predictors
y = randn(n)
X = [randn(n) for _ in 1:p]

# Run univariate regressions and store p-values
pvals = Float64[]
coefs = Float64[]

for xi in X
    df = DataFrame(y = y, x = xi)
    model = lm(@formula(y ~ x), df)
    push!(coefs, coef(model)[2])  # slope estimate
    push!(pvals, coeftable(model).cols[4][2])  # p-value of x
end

# Find "significant" ones
significant = findall(pvals .< 0.05)

# Print results
println("Found $(length(significant)) 'significant' predictors out of $p (all should be null).")
for i in significant
    println("x_$i: p = $(round(pvals[i], digits=4)), β̂ = $(round(coefs[i], digits=3))")
end
Found 3 'significant' predictors out of 20 (all should be null).
x_7: p = 0.0255, β̂ = 0.221
x_9: p = 0.0381, β̂ = 0.207
x_13: p = 0.0022, β̂ = -0.246

p-Hacking in Action

Optional stopping

  • “Collect 10 measurements per group, test, and if not significant add 5 more … up to 100.”
  • Each look at the data is another chance for a false positive (Armitage et al. 1969).
Code
using Random, HypothesisTests

Random.seed!(3)
nsim, nmax = 4000, 100
looks = 10:5:nmax
first_hit = zeros(Int, length(looks))
paths = Vector{Vector{Float64}}()

for s in 1:nsim
    a, b = randn(nmax), randn(nmax)                 # no true difference
    ps = [pvalue(EqualVarianceTTest(a[1:k], b[1:k])) for k in looks]
    s <= 6 && push!(paths, ps)
    i = findfirst(<(0.05), ps)
    i === nothing || (first_hit[i] += 1)
end
fpr = cumsum(first_hit) ./ nsim

l1 = plot(collect(looks), paths, label = "", linewidth = 2, yscale = :log10, ylims = (1e-3, 1),
    xlabel = "Observations per group", ylabel = "p-value (log scale)", title = "p-value paths (no effect)")
hline!([0.05], color = :red, linestyle = :dash, linewidth = 3, label = "0.05", legend = :topright)
l2 = plot(collect(looks), fpr, linewidth = 3, marker = :circle, label = "", ylims = (0, 0.35),
    xlabel = "Maximum observations per group", ylabel = "False-positive rate",
    title = "Stop at the first p < 0.05: $(round(100 * fpr[end], digits = 1))%")
hline!([0.05], color = :red, linestyle = :dash, linewidth = 3, label = "Nominal 5%", legend = :topleft)
plot(l1, l2, layout = (1, 2), size = (1100, 420), margin = 5mm)

Dos and Don’ts

Do:

  • pre-register hypotheses (Nosek et al. 2018)
  • fix the sample size (or the stopping rule) in advance
  • report effect sizes and CIs
  • write “we found evidence against the null hypothesis” or “we failed to reject the null hypothesis”

Don’t:

  • interpret p < 0.05 as a “true” result
  • selectively report significant findings (change the model to get a p-value below the threshold)
  • keep collecting data until p < 0.05
  • write that we “proved” the hypothesis
  • confuse improbability with impossibility

Multiverse Analysis

Multiverse Analysis

The core principle of a multiverse analysis is to detect decisions in the research process where reasonable alternatives could have been selected and to examine their impact on the results. (Steegen et al. 2016)

Multiverse Analysis

  • We are interested in testing if \(X\) has an effect on \(Y\) using linear regression.

  • However, model choice is not unique: with or without control variables, transformations of the variables, outlier treatment, etc.

  • The first step is to determine sensible model alternatives: all covariates that may have an effect (theoretically justified or previously shown).

  • Once determined, the estimated effects (with CIs) and their p-values can be reported for all sensible model alternatives (boxplot, histogram, etc.).

Multiverse Analysis

  • Let \(Z_0, Z_1, \ldots, Z_p\) be additional regressors in sensible model alternatives. The practitioner could consider any of the following regression models: \[ \begin{aligned} &Y = \beta X + \varepsilon && \Rightarrow \quad \hat{\beta}_1,\ p_1,\\ &Y = \beta X + \gamma_0 Z_0 + \varepsilon && \Rightarrow \quad \hat{\beta}_2,\ p_2,\\ &Y = \beta X + \gamma_1 Z_1 + \varepsilon && \Rightarrow \quad \hat{\beta}_3,\ p_3,\\ &Y = \beta X + \gamma_0 Z_0 + \gamma_1 Z_1 + \varepsilon && \Rightarrow \quad \hat{\beta}_4,\ p_4,\\ &\qquad \vdots && \qquad \vdots \\ &Y = \beta X + \gamma_0 Z_0 + \gamma_1 Z_1 + \cdots + \gamma_p Z_p + \varepsilon && \Rightarrow \quad \hat{\beta}_{2^{p+1}},\ p_{2^{p+1}}, \end{aligned} \] where \(\beta\) is the effect of \(X\) on \(Y\).

  • Each estimation would potentially yield a different estimated effect and associated \(p\)-value. With \(p+1\) candidate controls there are \(2^{p+1}\) models.

  • Only reasonable controls belong in the multiverse: adding a mediator or a collider changes what \(\beta\) measures.

Multiverse Analysis

Three possible multiverses (stylised)

Code
Random.seed!(1234)

lsize = 10
theme(:dao)

# Stylised p-values from 1000 model specifications in three scenarios
pvals1 = abs.(0.062 .+ 0.008 .* randn(1000))  # reported result is an outlier
pvals2 = abs.(0.020 .+ 0.006 .* randn(1000))  # nearly all specifications agree
pvals3 = abs.(0.050 .+ 0.012 .* randn(1000))  # result depends on the specification

function multiverse_plot(pv, ttl; reported=nothing)
    plt = histogram(pv, label="p-values", legend=(reported === nothing ? false : :topleft), alpha=0.5,
        title=ttl, xlabel="p-value", xlims=(0, 0.1), bins=0:0.004:0.1)
    vline!([0.05], color=:red, label="Significance level", linewidth=4, linestyle=:dash)
    if reported !== nothing
        scatter!([reported], [8], color=:black, label="Reported p-value", markersize=12,
            marker=:star5, markerstrokewidth=0)
    end
    plt
end

p1 = multiverse_plot(pvals1, "Fragile"; reported=0.03)
p2 = multiverse_plot(pvals2, "Robust")
p3 = multiverse_plot(pvals3, "Ambiguous")

plot(p1, p2, p3, layout=(1, 3), legendfontsize=lsize, titlefontsize=12, size=(1100, 450), margin=3mm, bottom_margin=8mm)

Dos and Don’ts

Do:

  • report multiple specifications
  • be transparent about decisions

Don’t:

  • present one path as definitive without justification or discussion
  • hide the multiverse of results (e.g., only report the one with p < 0.05)

Multiple Testing

Problem of Multiplicity

  • Testing many hypotheses increases the chance of false positives.

  • For testing an effect on \(m\) outcomes (or subgroups, or variables), one considers \(m\) hypotheses of the form: \[ H_0: \text{No effect} \quad H_1: \text{Effect} \]

  • As shown before, if all the nulls are true and the tests are independent, the probability of making at least one Type I error is: \[1 - (1 - \alpha )^m \]

Problem of Multiplicity

  • Contrast this with the probability of making a Type I error for a single hypothesis test, which is simply \(\alpha\).

  • One option is to test the joint (global) hypothesis: \[ \begin{aligned} H_0: &\text{ No effect for all } \\ H_1: &\text{ Effect for at least one } \end{aligned} \]

  • Rejecting the joint null tells us that at least one effect is non-zero, but not which one. To make claims about individual effects, we need to correct the individual tests.

Problem of Multiplicity

Correction Methods

  • Family-wise error rate (FWER): Probability of making at least one Type I error in a family of tests.

  • To control the FWER at a desired level \(\alpha\), the Bonferroni correction adjusts the per-test significance level:

\[ \alpha_{\text{adjusted}} = \frac{\alpha}{m} \]

  • Example: You are testing \(m = 20\) hypotheses and want to control FWER at \(\alpha = 0.05\), you only reject individual hypotheses if their p-value is below:

\[\alpha_{\text{adjusted}} = \frac{0.05}{20} = 0.0025\]

  • This ensures that the overall FWER is at most 0.05, whatever the dependence between the tests. With independent tests, the FWER is:
Code
1-(1-0.05/20)^20
0.04883012474683324

Problem of Multiplicity

Correction Methods Example

Code
using Random, Distributions, HypothesisTests, StatsPlots

Random.seed!(42)

# Simulate 20 tests under the null (true hypotheses)
pvals = [pvalue(OneSampleTTest(rand(Normal(0, 1), 30))) for _ in 1:20]

# Apply Bonferroni correction
α = 0.05
bonferroni_threshold = α / length(pvals)
significant = pvals .< bonferroni_threshold

# Visualize (log scale, so that both thresholds are visible)
scatter(1:20, pvals, xlabel="Test index", ylabel="p-value (log scale)", title="p-values from 20 Null Tests",
    color=:steelblue, markersize=8, label="p-values", yscale=:log10)
hline!([0.05], label="Nominal 0.05", color=:blue, linestyle=:dot, linewidth=3)
hline!([bonferroni_threshold], label="Bonferroni threshold", color=:darkred, linestyle=:dash, linewidth=3)

plot!(size=(1000, 400), margin=5mm, ylims=(1e-4, 1), legend=:bottomright)

Problem of Multiplicity

Correction Methods Example

Code
using Random, Distributions, HypothesisTests, StatsPlots
using Plots.PlotMeasures

Random.seed!(2026)

# Repeated null experiments to visualize multiplicity and Bonferroni control.
n_tests = 20
n_draws = 40
α = 0.05
bonferroni_threshold = α / n_tests

anim = @animate for draw in 1:n_draws
  pvals = [pvalue(OneSampleTTest(rand(Normal(0, 1), 30))) for _ in 1:n_tests]
  count_nominal = count(<(α), pvals)
  count_bonf = count(<(bonferroni_threshold), pvals)

  scatter(1:n_tests, pvals,
    xlabel = "Test index",
    ylabel = "p-value (log scale)",
    yscale = :log10,
    ylim = (1e-4, 1),
    legend = :bottomright,
    label = "p-values",
    color = :steelblue,
    markersize = 8,
    title = "Null experiment $draw of $n_draws | nominal rejections: $count_nominal | Bonferroni rejections: $count_bonf"
  )
  hline!([α], label = "Nominal 0.05", color = :royalblue, linestyle = :dot, linewidth = 3)
  hline!([bonferroni_threshold], label = "Bonferroni $(round(bonferroni_threshold, digits=4))", color = :darkred, linestyle = :dash, linewidth = 3)
  plot!(size = (1000, 400), margin = 5mm)
end

# Slow playback so each draw is easy to distinguish for the audience.
gif(anim, "bonferroni_threshold_experiment.gif", fps = 1)

Dos and Don’ts

Do:

  • adjust for multiple testing: Bonferroni or Holm to control the FWER (Holm 1979), Benjamini–Hochberg to control the false discovery rate (FDR) (Benjamini and Hochberg 1995)
  • report how many tests were run, not only the significant ones

Don’t:

  • ignore the problem of multiplicity
  • report unadjusted p-values as confirmatory evidence

Evaluating Predictive Models

Leakage: Selecting Features Before Cross-Validation

  • 50 samples, 2000 features, labels assigned at random. The true accuracy of any classifier is 50%.
  • Wrong: pick the 20 features most correlated with the label using all data, then cross-validate.
  • Right: do the selection inside each training fold (Hastie et al. 2009, sec. 7.10.2).
Code
using Random, Statistics, StatsPlots

Random.seed!(4)
topfeatures(X, y, m) = partialsortperm(vec(abs.(cor(X, y))), 1:m, rev = true)

function cv_accuracy(X, y; k = 5, nfeat = 20, select_inside = true)
    n = size(X, 1)
    folds = shuffle(repeat(1:k, outer = cld(n, k))[1:n])
    sel_all = topfeatures(X, y, nfeat)              # uses the test folds too: leakage
    correct = 0
    for f in 1:k
        tr, te = folds .!= f, folds .== f
        sel = select_inside ? topfeatures(X[tr, :], y[tr], nfeat) : sel_all
        w = hcat(ones(sum(tr)), X[tr, sel]) \ y[tr]  # least-squares linear classifier
        correct += sum(sign.(hcat(ones(sum(te)), X[te, sel]) * w) .== y[te])
    end
    correct / n
end

reps = 100
acc_wrong, acc_right = Float64[], Float64[]
for r in 1:reps
    X = randn(50, 2000)
    y = rand([-1.0, 1.0], 50)
    push!(acc_wrong, cv_accuracy(X, y; select_inside = false))
    push!(acc_right, cv_accuracy(X, y; select_inside = true))
end

boxplot(["Wrong: select, then CV"], acc_wrong, label = "", color = :darkred, fillalpha = 0.6)
boxplot!(["Right: select inside CV"], acc_right, label = "", color = :steelblue, fillalpha = 0.6)
hline!([0.5], color = :black, linestyle = :dash, linewidth = 3, label = "Chance (true accuracy)", legend = :bottomleft)
plot!(ylabel = "Cross-validated accuracy", ylims = (0, 1), size = (1000, 420), margin = 5mm,
    title = "Mean CV accuracy on pure noise: wrong = $(round(mean(acc_wrong), digits = 2)), right = $(round(mean(acc_right), digits = 2))")

Pitfalls in Evaluating Models

Don’t:

  • fit preprocessing (scaling, imputation, feature selection) on the full dataset
  • split time series or repeated measurements at random: the test set then contains near-copies of the training set
  • tune hyper-parameters on the test set, or report the best of many seeds or runs
  • claim one model is better from a single split without uncertainty

Do:

  • keep a test set that is used once; use nested CV for tuning
  • use grouped (by specimen, subject, site) or time-ordered splits
  • report variability across folds or seeds, and compare with simple baselines

Leakage has been found in hundreds of published ML-based studies (Kapoor and Narayanan 2023).

Biases in Data and Analysis

Biases in statistics

  • Selection bias: When the sample is not representative of the population.

  • Survivorship bias: When we only look at the survivors.

  • Confirmation bias: When we look for evidence that confirms our beliefs.

  • Publication bias: When significant results are more likely to be published than non-significant ones.

  • Extrapolation bias: When we extrapolate beyond the range of the data.

  • Correlation vs. causation: When we interpret a correlation as a causal effect (e.g., ignoring confounders).

  • Simpson’s paradox: When a trend appears in different groups but disappears or reverses when the groups are combined.

Summary

Key Takeaways

  • Be transparent with visuals and metrics
  • Avoid p-hacking and data dredging; fix the sample size in advance
  • Report multiple specifications or pre-register
  • Adjust for multiple testing
  • Keep the test set untouched and split the data by unit or time
  • Recognize and mitigate biases
  • Prioritize reproducibility and transparency: share code and data
  • Use notebooks and version control

Conclusion

Conclusion

References

Armitage, P., C. K. McPherson, and B. C. Rowe. 1969. “Repeated Significance Tests on Accumulating Data.” Journal of the Royal Statistical Society. Series A 132 (2): 235–44. https://doi.org/10.2307/2343787.
Benjamini, Yoav, and Yosef Hochberg. 1995. “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society. Series B 57 (1): 289–300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x.
Collaboration, Open Science. 2015. “Estimating the Reproducibility of Psychological Science.” Science 349 (6251): aac4716. https://doi.org/10.1126/science.aac4716.
Gelman, Andrew, and John Carlin. 2014. “Beyond Power Calculations: Assessing Type S (Sign) and Type M (Magnitude) Errors.” Perspectives on Psychological Science 9 (6): 641–51. https://doi.org/10.1177/1745691614551642.
Gelman, Andrew, and Eric Loken. 2014. “The Statistical Crisis in Science.” American Scientist 102 (6): 460–65. https://doi.org/10.1511/2014.111.460.
Greenland, Sander, Stephen J. Senn, Kenneth J. Rothman, et al. 2016. “Statistical Tests, P Values, Confidence Intervals, and Power: A Guide to Misinterpretations.” European Journal of Epidemiology 31 (4): 337–50. https://doi.org/10.1007/s10654-016-0149-3.
Hastie, Trevor, Robert Tibshirani, and Jerome Friedman. 2009. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed. Springer. https://doi.org/10.1007/978-0-387-84858-7.
Holm, Sture. 1979. “A Simple Sequentially Rejective Multiple Test Procedure.” Scandinavian Journal of Statistics 6 (2): 65–70.
Ioannidis, John P. A. 2005. “Why Most Published Research Findings Are False.” PLoS Medicine 2 (8): e124. https://doi.org/10.1371/journal.pmed.0020124.
Kapoor, Sayash, and Arvind Narayanan. 2023. “Leakage and the Reproducibility Crisis in Machine-Learning-Based Science.” Patterns 4 (9): 100804. https://doi.org/10.1016/j.patter.2023.100804.
Nosek, Brian A., Charles R. Ebersole, Alexander C. DeHaven, and David T. Mellor. 2018. “The Preregistration Revolution.” Proceedings of the National Academy of Sciences 115 (11): 2600–2606. https://doi.org/10.1073/pnas.1708274114.
Simonsohn, Uri, Joseph P. Simmons, and Leif D. Nelson. 2020. “Specification Curve Analysis.” Nature Human Behaviour 4 (11): 1208–14. https://doi.org/10.1038/s41562-020-0912-z.
Steegen, Sara, Francis Tuerlinckx, Andrew Gelman, and Wolf Vanpaemel. 2016. “Increasing Transparency Through a Multiverse Analysis.” Perspectives on Psychological Science 11 (5): 702–12. https://doi.org/10.1177/1745691616658637.
Tufte, Edward R. 1983. The Visual Display of Quantitative Information. Graphics Press.
Wasserstein, Ronald L., and Nicole A. Lazar. 2016. “The ASA Statement on p-Values: Context, Process, and Purpose.” The American Statistician 70 (2): 129–33. https://doi.org/10.1080/00031305.2016.1154108.