We stress-tested three claims from our own research notebook. The 11% storage effect became 0.7% on a 4-hour battery.
Every research notebook has a shadow metric: the number of claims it has never actually tested.
Ours was easy to count. At the end of October the notebook listed 108 completed tests, 16 reviewed reproductions, 24 Frontier Ideas and 33 theory volumes — and zero external peer reviews. The site says so plainly, which is the right thing to do, but it also means the archive is a map of questions rather than a record of settled answers.
So we did something less glamorous than adding Volume XXXIV. We took the three investigations our own review said were worth doing, and we ran them.
The results are not a triumph. One real effect got much smaller under realistic assumptions. One small effect survived a proper out-of-sample test. One popular idea did not beat a simpler baseline. That is a good week for a research notebook, and it is the week this post is about.
The storage result: real, but not the number on the box
The flagship study uses ten years of French half-hourly demand, wind and solar — 175,344 observations from RTE’s éCO2mix dataset, 2015–2024. It asks a clean question: if you hold the annual peak and the variance of net load exactly fixed, does the order of days change how much storage you need?
It does. Shuffling whole days within each month leaves the peak at 0.0 MW difference and the variance at about 3 × 10⁻¹⁶ relative difference. Only the sequence changes. The ideal annual model gives a median +11.0% extra nominal capacity (95% CI +8.0% to +16.8%), and all ten French years sit above their own reordered references.
That number is real. It is also fragile in a way the headline never quite says.
Across 20 defensible designs, the premium ranges from +1.6% to +14.0%. A ten-year sizing window halves it. Sizing variable capacity across the decade takes it to about +1%. And if you stop sizing the system and fix the asset instead — a 10 GW battery with four hours of energy, dispatched against actual demand, with unserved energy as the metric — the ordering effect is +0.74%. At ten hours it is +2.86%.
The honest claim is not “day order changes storage needs by 10.7%”. It is:
At fixed marginal statistics, day order carries information that peak-and-variance sizing rules cannot represent. The size of that information is a modelling choice: roughly 3–5% in a system with a real dispatchable fleet, and under 1% for a four-hour battery measured by unserved energy.
That is a narrower claim, and a more useful one. It says the phenomenon is real. It does not say a four-hour battery will perform 10% better or worse depending on which Tuesday follows which Monday.
What we are changing: the paper will lead with the design grid and the fixed-asset null, not the ideal annual number. The next technical step is an adequacy model with transmission, unit commitment and imports. The current result is a methods contribution — a within-month day-order null applied to a public multi-year record — and it should be read that way.
You can read the existing test, with the data and code, here: Sequence changes ideal storage accounting.
Soil moisture: small, robust, and not a missing driver
Our earlier soil analysis found a striking association. In 156,400 modeled summer days at 50 city locations, extreme heat occurred on 15.3% of days in the driest fifth of soil wetness and 0.8% in the wettest fifth. That is roughly a nineteenfold contrast.
It is also a same-day association in reanalysis data. It cannot tell you whether soil moisture improves a forecast. For that you need to predict tomorrow from what is known today, and test it on years the model has not seen.
So we used NOAA’s U.S. Climate Reference Network, which measures air temperature and soil moisture at the same station. The panel covers May–September, 2011–2025, and 31 stations. The target is a daily maximum at t+1, t+3 or t+5 above that station’s 90th percentile. The baseline includes seasonal harmonics, temperature persistence at lags 0–3, humidity, solar radiation and precipitation. The soil model adds antecedent soil moisture at three timescales.
The result: soil moisture helps, but only a little, and the help grows with lead time.
| Horizon | AUC, weather | AUC, + soil | Gain (95% CI) |
|---|---|---|---|
| t+1 | 0.892 | 0.895 | +0.003 (+0.001 to +0.005) |
| t+3 | 0.766 | 0.774 | +0.008 (+0.004 to +0.011) |
| t+5 | 0.733 | 0.740 | +0.007 (+0.003 to +0.011) |
The 2020–2025 holdout reproduces the pattern. At station level, 23 of 30 stations are positive at t+3. The confidence intervals exclude zero.
But the practical effect is smaller than the AUC gain suggests. At t+3, the top decile of predicted risk contains 29.0% heat extremes under the weather-only model and 29.2% under the soil model. The Brier skill score improves by about 0.006. Soil moisture improves ranking modestly; it does not transform the decision.
What this means: soil moisture is a modest incremental input at three to five days, not the missing driver of heat waves. The nineteenfold association is still mostly a same-day, same-air-mass story until someone shows otherwise. That is a publishable result precisely because it is small and out-of-sample — and because it warns against building a product on the association alone.
The earlier association is here: Drier modeled soil coincides with more extreme heat days.
Model ensembles: shared errors are real, but diversity is not a free lunch
The third test began with a mathematical result. Ensemble error variance is affine in the correlation between member errors, so the benefit of adding a member falls as that correlation rises. The simulation gave a threshold surface: with a new member one quarter as variable as the existing members, it must be correlated below 0.69 (three members), 0.39 (five), 0.18 (ten) or 0.06 (thirty) to help.
That is a statement about a model of errors, not about real models. So we ran nine real model families — logistic regression, SVM, random forest, extra trees, gradient boosting, k-nearest neighbours, a small neural network, naive Bayes and a decision tree — on four classification tasks, and recorded exactly which test examples each one got wrong.
Real models do share errors, substantially. Mean pairwise error correlation ranged from 0.36 on the digits task to 0.81 on iris. On breast cancer it was 0.55; on wine, 0.71.
Then we compared selection strategies. The top three models by validation accuracy averaged 0.9718 test accuracy. A diversity filter that preferred models with low error correlation averaged 0.9700. Greedy diversity averaged 0.9699. The paired difference was 0.0018 with a confidence interval that includes zero. Accuracy selection won in 35% of splits; diversity selection was not reliably better or worse.
Adding all nine models was worse than the best single model in 82.5% of splits, by an average of 0.0066.
What this means: the shared-error claim survives contact with real models. The stronger claim — that selecting for diversity beats selecting for accuracy — does not. Accuracy is the first-order variable. Diversity is a second-order refinement that did not pay for itself here.
Caveat: these are classical classifiers, not large language models. The LLM version is the next experiment, and it needs a fixed question set and a sample large enough to detect a one-point difference. We are not going to claim more than this test shows.
The original ensemble study is here: Testing ensemble errors: a correlation floor, not a universal cutoff.
The corrections we owe the reader
Running the tests exposed several smaller problems that matter more than they look:
- The weathering-coupon idea cites its supporting paper as Geoderma; the linked paper is in Applied Geochemistry. Correct the citation.
- The tactile-force idea implies no calibration method exists. NIST documented robotic-hand force calibration against external load cells in 2018, and later work reports calibrated tactile-force estimates. Narrow the novelty claim.
- The experiment-optimal-stopping idea implies the broader problem is largely unaddressed. It is not; Bayesian experiment design and optimal stopping are mature literatures. The novelty has to be the specific comparison of AI inference costs with measurement costs.
- The storage decade-level interpretation was not reproduced independently. The disputed part is the balance rule, not the arithmetic.
- Many mathematical results reproduce known theory. Keep them in a methods section, not a discovery section.
None of these is fatal. They are the normal corrections a serious notebook makes in public. The risk is that they accumulate faster than they are fixed.
What happens next
Three things, in order:
- Publish the storage result as a methods paper. Lead with the design grid, the fixed-asset null and the sentence “this is not a storage forecast”. Add an adequacy model.
- Publish the soil result as a short predictive study. The claim is narrow: antecedent soil moisture improves out-of-sample heat-extreme ranking at three to five days by about 0.008 AUC in USCRN. Do not describe it as a causal discovery.
- Publish the ensemble result as a replication and a null. Shared errors are real; diversity selection did not beat accuracy selection. Then run the LLM version.
And one editorial change: stop generating new hypothesis volumes. The archive’s marginal value is now negative. Another volume makes the one real paper harder to find. The site’s own editorial triage already says this. Make it permanent.
The full follow-up report, with the code, derived data and result files, is available from the research page: incoprea.com/research.
If you want the broader context, the 24 proposals are here: Frontier Ideas.
The lesson
A research programme is not a pile of hypotheses. It is a set of claims that have survived attempts to kill them.
We now have three more attempts on the record: one effect that shrank, one that held at a smaller size than we hoped, and one that tied with a simpler baseline. That is not a failure. It is the first time in a while that the notebook has learned something it did not already believe.
### Links- Research notebook: https://incoprea.com/research/- Storage test, data and code: https://incoprea.com/research-library/tests/storage-chronology-20261030/- Soil-moisture association (the earlier study): https://incoprearesearch.blogspot.com/2026/10/drier-modeled-soil-coincides-with-more.html- Ensemble error-correlation study: https://incoprearesearch.blogspot.com/2026/10/testing-ensemble-errors-correlation.html- Frontier Ideas (the 24 proposals): https://incoprea.com/research-library/frontier-ideas-20261030/
Working research, not peer reviewed. The tests were designed and executed with AI assistance. The code and numbers are published so that someone else can check them — and disagree.
Comments
Post a Comment