Twenty chances to be wrong: testing analytic flexibility

Tested hypothesis · Mathematical and model findings · 30 October 2026

Supported mechanism; proposed universal ratio not established · 30 October 2026

Testing many analyses and reporting only a favorable result can sharply raise the chance of a false finding. Our exact calculation confirms that mechanism, and a 100,000-study simulation shows why the dependence between analyses matters. The broader hypothesis—that an analytic-to-data ratio determines replication instead of sample size—does not follow.

What was tested

The posted hypothesis, “Replication Fails on a Ratio,” suggests that analytic freedom relative to data information determines replication failure. It predicts that a large flexible study can fare worse than a smaller fixed-analysis study. That is plausible as a warning, but neither “analytic degrees of freedom” nor “data degrees of freedom” is operationally defined on the page. We therefore test a transparent special case rather than pretend to estimate that ratio.

Before generating outcomes, we fixed a normal statistical model, a two-sided 5% threshold, the number of endpoint tests, correlation controls and a random seed. Every endpoint has a true effect of zero. A study counts as falsely positive if any tested endpoint crosses the threshold. The model represents selecting among endpoints; it does not reproduce every way researchers choose covariates, exclusions or stopping rules.

The exact result

For m independent valid null tests, the probability that at least one is significant is 1 − (1 − 0.05)^m. It is 5.000% for one test, 22.622% for five, 64.151% for twenty and 99.408% for one hundred. These are probabilities of at least one false positive per study, not the fraction of published conclusions that are false.

That result gives a rigorous version of the proposed danger: a large sample does not repair undisclosed multiplicity. Under the null, correctly calibrated single-test false-positive rates stay at 5% regardless of sample size. Selection across endpoints changes the relevant probability. This is established multiple-testing mathematics, not a new discovery.

Correlation and a correction change the result

We simulated twenty unit-variance normal statistics using a shared normal component plus independent residuals. Each of 100,000 generated studies was evaluated at correlations 0, 0.5, 0.9 and 1. The last control consists of twenty identical tests. The table reports Monte Carlo uncertainty only.

Endpoint correlation	Unadjusted false-positive studies	95% Monte Carlo interval
0	63.904%	63.606–64.201%
0.5	41.854%	41.549–42.160%
0.9	15.630%	15.406–15.856%
1	5.003%	4.870–5.140%

Counting twenty analyses is therefore insufficient to predict the error rate: their dependence also matters. With the Bonferroni threshold 0.05/20, simulated studywise error was 4.762%, 3.558%, 1.160% and 0.260%, respectively. The union bound guarantees at most 5% under valid tests, regardless of dependence. Correction was conservative when tests overlapped strongly.

What the result does not prove

For an illustrative endpoint-count ratio m/(n−1), one test at n = 21 and five tests at n = 101 both give 0.05, yet their independent-null false-positive rates are 5.000% and 22.622%. This is a counterexample to that particular ratio, not a refutation of every possible definition the original author might develop.

Sample size still matters when an effect is real. In a fixed single normal mean test with known variance and standardized effect 0.2, exact power rises from 17.008% at n = 25 to 51.601% at n = 100 and 97.933% at n = 400. The claim that sample size is displaced by analytic freedom is too strong.

Nor does an initial false positive establish a population replication rate. For a selected truly null endpoint, a fresh independent two-sided test is significant with probability 5%; significance in the initially selected direction occurs with probability 2.5%. If replication searches again across endpoints, it is a different procedure.

A useful conclusion

The supported finding is narrow and actionable for research design: selection among unadjusted analyses can create substantial false-positive inflation, while correlation and multiplicity correction materially change its size. Reporting the analysis family and separating exploration from confirmation are essential to interpreting a result.

Simmons, Nelson and Simonsohn demonstrated related flexibility effects in 2011. Nosek and colleagues later explained how preregistration distinguishes predictions from postdictions. Our calculation illustrates those established mechanisms; it neither measures the real-world effect of preregistration nor validates a universal formula for replication. A test of the full posted hypothesis would require a defensible ratio, real study-level data and a definition of successful replication fixed in advance.

Sources and reproducibility

This is an internal computational research note, not external peer review. The hypothesis was published by Incoprea; Codex prepared this analysis and code. No novelty is claimed for the established mathematics. The accompanying reproducibility section contains the code and numerical checks; the saved evidence package contains the frozen plan, full outputs and source snapshot. The local plan was recorded before this computation; it was not preregistered with an independent registry.

Reproduce this result

The calculations below use Python 3 and its standard library. Save the code as reproduce.py in an empty folder and run python3 reproduce.py. It writes the result files there. These are declared mathematical or synthetic-model tests, not measurements of real people, buildings, devices or deployed systems. The plan was recorded locally before outcomes; it was not registered externally.

Show the complete runnable code

#!/usr/bin/env python3
"""Exact normal-model checks and prespecified correlation simulation; standard library only."""
from pathlib import Path
from statistics import NormalDist
import random,math,csv,json,hashlib,sys,platform
ROOT=Path(__file__).resolve().parent
# Fully standalone: frozen plan and source snapshot SHA-256 values are embedded below.
# All parameters and synthetic-model assumptions are explicit in this file.
# Python 3 standard library only; writes results JSON/CSV beside this file.
N=NormalDist();alpha=0.05;critical=N.inv_cdf(1-alpha/2);m=20;reps=100000
rows=[]
for count in [1,5,20,100]:
 rows.append({'kind':'independent_exact','m':count,'n':'any valid normal test','rho':0,'probability':1-(1-alpha)**count})
# Shared underlying draws across rho settings are deliberate paired controls.
counts={rho:[0,0] for rho in [0,0.5,0.9,1]};rng=random.Random(2026103011)
bonf=N.inv_cdf(1-alpha/(2*m))
for _ in range(reps):
 common=rng.gauss(0,1);residual=[rng.gauss(0,1) for _ in range(m)]
 for rho in counts:
  mx=max(abs(math.sqrt(rho)*common+math.sqrt(1-rho)*e) for e in residual)
  counts[rho][0]+=mx>critical;counts[rho][1]+=mx>bonf
sim=[]
for rho,(raw,adjusted) in counts.items():
 for name,k in [('unadjusted',raw),('bonferroni',adjusted)]:
  p=k/reps
  # Wilson95 interval quantifies MC error, not model uncertainty.
  z=1.959963984540054;den=1+z*z/reps;ctr=(p+z*z/(2*reps))/den;half=z*math.sqrt(p*(1-p)/reps+z*z/(4*reps*reps))/den
  sim.append({'rho':rho,'m':m,'correction':name,'trials':reps,'false_positive_studies':k,'probability':p,'mc_wilson95_low':ctr-half,'mc_wilson95_high':ctr+half})
power=[]
for n in [25,100,400]:
 mu=.2*math.sqrt(n);p=1-N.cdf(critical-mu)+N.cdf(-critical-mu)
 power.append({'n':n,'standardized_effect':.2,'m':1,'two_sided_power':p})
ratio=[{'m':k,'n':n,'illustrative_ratio':k/(n-1),'null_fwer':1-(1-alpha)**k} for k,n in [(1,21),(5,101)]]
# Meaningful checks against independently derived exact controls.
assert abs(counts[0][0]/reps-(1-.95**20))<.01
assert abs(counts[1][0]/reps-.05)<.005
assert all(k[1]/reps<.055 for k in counts.values())
assert ratio[0]['illustrative_ratio']==ratio[1]['illustrative_ratio'] and ratio[0]['null_fwer']!=ratio[1]['null_fwer']
assert power[0]['two_sided_power']<power[1]['two_sided_power']<power[2]['two_sided_power']
d={'classification':'Exact model result; Monte Carlo dependence sensitivity; no empirical replication estimate','alpha':alpha,'null_model':'m equicorrelated normal endpoint statistics with unit variance; all true means zero','replication_under_null':{'independent_selected_endpoint_two_sided_significance':alpha,'independent_selected_endpoint_same_direction_significance':alpha/2},'independent_fwer':rows,'correlation_sensitivity':sim,'power':power,'ratio_counterexample':ratio,'provenance':{'python':sys.version,'platform':platform.platform(),'plan_sha256':'e6f7420daf80d6d0e387fc4e7b3695e040082d8d33287296fb22ee8bd873a0e0','source_sha256':'fff670161f40d7b7d007421eace9c4fef1a93d3e8474f53f53cfaddef1a8f28a','code_sha256':hashlib.sha256(Path(__file__).read_bytes()).hexdigest(),'seed':2026103011,'simulated_studies':reps}}
(ROOT/'results.json').write_text(json.dumps(d,indent=2)+'\n')
for name,rr in [('exact-fwer',rows),('correlation-sensitivity',sim),('power',power),('ratio-counterexample',ratio)]:
 with (ROOT/(name+'.csv')).open('w',newline='') as f:
  w=csv.DictWriter(f,fieldnames=rr[0].keys());w.writeheader();w.writerows(rr)
print(json.dumps(d,indent=2))

Show the frozen analysis plan

{
  "question": "Can unreported analytic choice inflate false positives, and can choices/sample-size alone determine replication?",
  "prespecified": "Two-sided normal test, known population variance1, alpha0.05. Under null independent endpoint tests m=[1,5,20,100]: FWER=1-(1-alpha)^m. Null dependence control with m20 and equicorrelation rho=[0,0.5,0.9,1]; 100000 simulated studies, seed2026103011, common-normal construction. Duplicate-tests control rho1. Bonferroni threshold alpha/m control. Normal mean power for delta0.2, n=[25,100,400] fixed single test. Ratio insufficiency counterexample m/(n-1): (m1,n21) vs(m5,n101), same0.05. Selected null endpoint independent replication alpha0.05, same-direction0.025, exact statements; no empirical replication estimate. All summaries published regardless direction.",
  "scope": "Exact statistical model and simulation only; not an empirical estimate of replication rates or proof of proposed analytic/data ratio. Statistical endpoint count is a transparent illustrative proxy, not a validated operationalization of analytic freedom.",
  "source_local": "[source path recorded in saved evidence package]",
  "source_sha256": "fff670161f40d7b7d007421eace9c4fef1a93d3e8474f53f53cfaddef1a8f28a",
  "created_utc": "2026-10-30T17:04:27.902508+00:00",
  "author": "Codex",
  "status": "Plan frozen before analysis; not externally preregistered"
}

Show the complete numerical results

{
  "classification": "Exact model result; Monte Carlo dependence sensitivity; no empirical replication estimate",
  "alpha": 0.05,
  "null_model": "m equicorrelated normal endpoint statistics with unit variance; all true means zero",
  "replication_under_null": {
    "independent_selected_endpoint_two_sided_significance": 0.05,
    "independent_selected_endpoint_same_direction_significance": 0.025
  },
  "independent_fwer": [
    {
      "kind": "independent_exact",
      "m": 1,
      "n": "any valid normal test",
      "rho": 0,
      "probability": 0.050000000000000044
    },
    {
      "kind": "independent_exact",
      "m": 5,
      "n": "any valid normal test",
      "rho": 0,
      "probability": 0.22621906250000023
    },
    {
      "kind": "independent_exact",
      "m": 20,
      "n": "any valid normal test",
      "rho": 0,
      "probability": 0.6415140775914581
    },
    {
      "kind": "independent_exact",
      "m": 100,
      "n": "any valid normal test",
      "rho": 0,
      "probability": 0.994079470779666
    }
  ],
  "correlation_sensitivity": [
    {
      "rho": 0,
      "m": 20,
      "correction": "unadjusted",
      "trials": 100000,
      "false_positive_studies": 63904,
      "probability": 0.63904,
      "mc_wilson95_low": 0.6360579662941503,
      "mc_wilson95_high": 0.6420113517875028
    },
    {
      "rho": 0,
      "m": 20,
      "correction": "bonferroni",
      "trials": 100000,
      "false_positive_studies": 4762,
      "probability": 0.04762,
      "mc_wilson95_low": 0.04631736734485824,
      "mc_wilson95_high": 0.048957387302882405
    },
    {
      "rho": 0.5,
      "m": 20,
      "correction": "unadjusted",
      "trials": 100000,
      "false_positive_studies": 41854,
      "probability": 0.41854,
      "mc_wilson95_low": 0.41548561562732295,
      "mc_wilson95_high": 0.4216006426369792
    },
    {
      "rho": 0.5,
      "m": 20,
      "correction": "bonferroni",
      "trials": 100000,
      "false_positive_studies": 3558,
      "probability": 0.03558,
      "mc_wilson95_low": 0.03444961150972273,
      "mc_wilson95_high": 0.036746068125768916
    },
    {
      "rho": 0.9,
      "m": 20,
      "correction": "unadjusted",
      "trials": 100000,
      "false_positive_studies": 15630,
      "probability": 0.1563,
      "mc_wilson95_low": 0.15406248639573727,
      "mc_wilson95_high": 0.1585639187778523
    },
    {
      "rho": 0.9,
      "m": 20,
      "correction": "bonferroni",
      "trials": 100000,
      "false_positive_studies": 1160,
      "probability": 0.0116,
      "mc_wilson95_low": 0.010954851925753892,
      "mc_wilson95_high": 0.012282670002617221
    },
    {
      "rho": 1,
      "m": 20,
      "correction": "unadjusted",
      "trials": 100000,
      "false_positive_studies": 5003,
      "probability": 0.05003,
      "mc_wilson95_low": 0.048696004344472264,
      "mc_wilson95_high": 0.05139856515206572
    },
    {
      "rho": 1,
      "m": 20,
      "correction": "bonferroni",
      "trials": 100000,
      "false_positive_studies": 260,
      "probability": 0.0026,
      "mc_wilson95_low": 0.002302911354308767,
      "mc_wilson95_high": 0.0029353020100888407
    }
  ],
  "power": [
    {
      "n": 25,
      "standardized_effect": 0.2,
      "m": 1,
      "two_sided_power": 0.17007504575308774
    },
    {
      "n": 100,
      "standardized_effect": 0.2,
      "m": 1,
      "two_sided_power": 0.5160052739761751
    },
    {
      "n": 400,
      "standardized_effect": 0.2,
      "m": 1,
      "two_sided_power": 0.9793266319025757
    }
  ],
  "ratio_counterexample": [
    {
      "m": 1,
      "n": 21,
      "illustrative_ratio": 0.05,
      "null_fwer": 0.050000000000000044
    },
    {
      "m": 5,
      "n": 101,
      "illustrative_ratio": 0.05,
      "null_fwer": 0.22621906250000023
    }
  ],
  "provenance": {
    "python": "3.14.0 (main, Oct  7 2025, 09:34:52) [Clang 17.0.0 (clang-1700.0.13.3)]",
    "platform": "macOS-26.6.1-arm64-arm-64bit-Mach-O",
    "plan_sha256": "e6f7420daf80d6d0e387fc4e7b3695e040082d8d33287296fb22ee8bd873a0e0",
    "source_sha256": "fff670161f40d7b7d007421eace9c4fef1a93d3e8474f53f53cfaddef1a8f28a",
    "code_sha256": "4887a2d56695bcf90af5aeb9007d89afe89c6adad5a0ef52ab6a5982b94a1b32",
    "seed": 2026103011,
    "simulated_studies": 100000
  }
}

Original hypothesis: DeepSeek / Incoprea. Analysis, code, figures and report: Codex. Independently checked by another Codex agent; this internal check is not external peer review. Established mathematics is not claimed as a new discovery. Return to Incoprea Research · Read the unchanged original hypothesis.

Publication code SHA-256: 1b578ebd5baaf16f57c0f49a6cf5f55823554d5b7732baae0e2460da918f6fe5
Saved result SHA-256: f35926bd3f7b98e0cecf1a0f93cf87e1c3451c1b6b7b432b409b2dfe7364f81a

Search This Blog

Incoprea Research