Medical AI needs a visible evidence history

INCOPREA / RESEARCH

AI & medicine · Technology and policy commentary · Sources reviewed October 30, 2026

The evidence attached to medical AI has a date. Keeping that evidence useful is a continuing job.

Imagine a hospital evaluating an image-analysis tool and using it for several years. Its code might remain fixed while the surroundings change: different equipment, patient populations, recording practices or clinical routines. This is an illustrative scenario, not a report of a particular failure. It explains why asking whether a model once worked is different from asking whether it still works here.

Watch the model and its surroundings

FDA's January 2025 lifecycle draft discusses performance monitoring, changing inputs and the context of use. It proposes planning for changes in performance after deployment, including possible responses and communication to users. This treats maintenance as part of the product. Importantly, the official guidance page still identifies this document as draft, not for implementation at our review cutoff. Its proposals should not be reported as a final rule. Check the draft's status and text.

Plan a change before making it

A separate FDA document is final: the August 2025 guidance on predetermined change control plans for AI-enabled device software. It describes a plan covering specified modifications, methods for developing, validating and implementing them, and an assessment of their impact. FDA reviews that plan within a device's marketing submission. The approach can allow modifications described in the reviewed plan without a new marketing submission for each one; it is not an unrestricted permission to change any feature. Read the final PCCP guidance.

These documents concern medical devices and should not be generalized to every chatbot or administrative tool. Neither the existence of a change plan nor a version number demonstrates that a product learns continuously from every patient. The relevant questions are what changed, what was checked and which uses the evidence supports.

An evidence history people can use

Our editorial proposal is a readable evidence history alongside the software history: version and date, intended use, evaluation population, known limits, observed performance and the reason for each material change. An update that improves one task should not silently acquire credit for another task that was never tested.

The joint FDA, Health Canada and MHRA transparency principles recommend timely communication of updates and information about performance monitoring. Their emphasis is useful: maintenance includes helping users understand the change, not simply installing it. Read the transparency principles.

A clear history would let a clinician connect a published result to the version on the screen. It would also let a researcher distinguish a promising update from a demonstrated improvement. Those are modest documentation goals, but they make larger claims easier to examine.

Original commentary prepared with AI assistance by Codex from the linked official sources. This is not a clinical study, independent clinical review or device-specific regulatory advice. Educational information, not personal medical advice.

Comments

Post a Comment

Popular posts from this blog

Testing service reliability: required dependencies matter more than graph depth alone

Tested hypothesis · Mathematical and model findings · 30 October 2026 Incoprea Research · 30 October 2026 Evidence status: Exact conditional support for serial multiplication; counterexample to depth as a sufficient predictor. Reliability model and synthetic checks, not measured outage evidence. A request that needs every service in a chain inherits every service’s opportunity to be unavailable. That mechanism in Depth Beats Count is correct under independence. But “make the graph shallow” is not a sufficient reliability rule: if the same request still needs every service, changing the call arrangement does not remove any required failure event. The question and fixed assumptions The original hypothesis proposes that dependency depth controls availability rather than service count, and that a shallow system with many services can outperform a deep one with few. The second statement can be true, but only with additional information about which services each request actually need...
Read more

Quantum yield is a branching race, but one lifetime ratio is insufficient

Tested hypothesis · Mathematical and model findings · 30 October 2026 Incoprea Research · 30 October 2026 · Mathematical/model test Finding: Competing reaction pathways determine yield in a simple excited-state model, and stronger absorption alone changes the amount of product rather than the yield per absorbed photon. However, the lifetime-to-vibrational-relaxation ratio proposed in Quantum Yield Is a Race is not generally sufficient: two model systems can have exactly the same ratio and quantum yields of 0.1 and 0.8. This is an exact kinetic derivation and synthetic stochastic check. It narrows a plausible mechanism in the Incoprea Research hypothesis collection ; it does not experimentally validate the original predictor. Branching kinetics and the distinction between absorption and quantum yield are established science, not new discoveries. This note has no external peer review. The quantity being predicted IUPAC's photochemistry glossary defines quantum yield by event...
Read more

When rare events overwhelm a detector: an exact Bayes test

Tested hypothesis · Mathematical and model findings · 30 October 2026 Incoprea Research · Original hypothesis Exact support for the base-rate mechanism; accuracy claim corrected · 30 October 2026 A detector that catches 90% of genuine events and correctly clears 90% of non-events can produce less than 1% genuine events among its positive flags when prevalence is 0.1%. Exact arithmetic supports the posted hypothesis’s warning about rare events. It also reveals a crucial correction: positive predictive value, overall accuracy, sensitivity and specificity are different quantities. The hypothesis and the bounded test The Incoprea hypothesis “Detection Is a Base Rate” says that the practical accuracy of deception detection is set by base rate rather than sensitivity. Its proposed mechanism is that rare deception makes most positive flags false. We tested that probability claim using a hypothetical detector. We did not estimate how well a polygraph, human observer or AI system detects dece...
Read more