A real signal
can lose the search.
More candidates can raise the winning score while making the genuinely better candidate less likely to win.
One candidate really has 65% accuracy. Its competitors have 50%. They share some of the same easy and difficult cases. This experiment asks whether the search recovers the real advantage—and what would justify accepting the winner.
← Three threads notebook · Argument and finite derivation · Claim-to-evidence ledger
The best-looking result
need not be the best candidate.
Every search scores all candidates on 64 shared items and selects the highest score. Ties are broken uniformly. Increasing the number of candidates adds null competitors; there is still exactly one real signal. The selector is not told which candidate it is.
The last quantity averages 65% when the signal wins and 50% when a null wins. It is a property of the specified generating model, not a performance estimate supplied to the selector.
Scroll sideways on a narrow screen to read the full figure.
With independent candidates, moving from 2 to 100 candidates raises the expected winning score from 65.15% to 67.86%, while the selected candidate's average true accuracy falls from 64.36% to 57.04%. The search has more opportunities to promote a lucky null.
Why dependence helps selection in these particular scenarios
Each item is independently easy or difficult. Conditional success probability is p + δ on an easy item and p − δ on a difficult one. All candidates experience the same difficulty sign, while their remaining outcomes are conditionally independent. Increasing δ preserves the marginal abilities of 65% and 50%.
At 100 candidates, raising δ from 0 to 0.20 increases signal selection from 46.93% to 54.92%. Under this additive mechanism, the common difficulty component partly cancels when candidates are compared. This is not a general claim that correlation improves selection. Other dependence structures can have different consequences.
Selecting a winner
does not settle its claim.
Compare three rules after the same selection step. Nominal reuse tests the winning score at a nominal 5% cutoff. Bonferroni reuse adjusts that cutoff for the number of candidates. Fresh confirmation tests the one selected winner on 64 new independent items, after every search, with no requirement to pass reused testing first.
Every percentage below uses all searches as its denominator. Accepted signal, accepted null and no acceptance partition those searches. “No acceptance” still includes a selected winner; it means the applicable test did not pass.
| Acceptance rule | Accepted signal | Accepted null | No acceptance | Unique items | Candidate-item evaluations |
|---|
Fresh confirmation uses 128 unique items in total; the reuse rules use 64. Its additional evidence is part of the comparison. No candidate is reselected after the fresh result.
Any null crossing is a different event
A null can cross the threshold while the real signal scores higher and wins. The chance that any null crosses is therefore different from the chance that the selected null is accepted.
| Rule | Any null crosses | Selected null accepted |
|---|
The fresh familywise entry is not applicable: only the selected winner is tested on the fresh batch, not every unselected null. The ordinary single-null test has an attainable 2.9971% rejection probability here; discreteness makes it lower than the nominal 5%.
Scroll sideways on a narrow screen to read the full figure.
At 100 candidates and δ = 0.20, nominal reuse accepts a null in 37.54% of searches and the signal in 52.23%. Bonferroni reduces accepted nulls to 2.16%, while accepting the signal in 15.05%. Fresh confirmation accepts a null in 1.35% and confirms the signal in 39.10%, using the extra 64 items.
These are numerically evaluated finite laws, not simulated or empirical frequencies. Bonferroni's union bound needs valid marginal null tests, but does not require independent candidates. Fresh confirmation needs independence from the selection batch. This experiment neither covers arbitrary sequential adaptivity nor implements the private reusable-holdout algorithm.
What redirects the inquiry
may not settle the new question.
The unexpected winner can be the real signal. Nothing in this calculation makes surprise worthless. The difficulty is that some of the apparent confirmation has already been used to choose what deserves attention. Its history remains part of the evidence.
For Circulatory Epistemology, this sharpens a proposed discipline of the return: remember what the evidence helped select, and what the next encounter could actually test. The force of recognition can open an inquiry while leaving its explanation to be examined.
The framework claims that truth becomes actual through living recognition between sensor and instrument. These probabilities model neither experience nor recognition. They challenge a shortcut from a compelling selected pattern to a justified general claim. A distinct CE contribution would have to improve how a living inquiry handles that passage beyond the statistical practices already available.
Where this model stops
One fixed 65% signal; exchangeable 50% nulls; independent items; a specified shared difficulty sign; conditional independence across candidates; uniform selection ties; valid binomial null tests; one independent fresh confirmation batch. This excludes persistent null ability differences, unknown or misspecified dependence, trained candidate generation and reuse of the final confirmation results. It establishes neither a best rule at equal total budget nor a measured advantage for CE.