How this study can fail
A failure mode listed in prose is easy to forget. Each one here has a detector that runs on any dataset — synthetic now, human later — and a threshold at which it fires.
On the synthetic reference cohort 3 of 13 detectors fire. That is the intended result, not an embarrassment: the reference cohort was generated with motor persistence, running-mean regression and a confounded naive estimate, so the detectors for those three should fire, and they do. The others staying quiet is equally informative — no timing anomaly, no wraparound bug, no non-engagement, no unexplained heavy tails.
Fig. 1 Detector results on the reference cohort
SIMULATED DATA
| Failure mode | Statistic | Value | Threshold | Result |
|---|---|---|---|---|
| timing anomalyBrowser frame scheduling, background tab throttling, low refresh rate. | P(|actual-requested| > 50 ms) | 0.000 | 0.05 | quiet |
| device incompatibilityMobile or low-refresh displays; orientation dial behaves differently. | max median frame interval (ms) | 16.700 | 34.00 | quiet |
| randomisation imbalanceA memoryless observer looks serially dependent when the sequence is autocorrelated. | max |circular autocorr(lag 1)| | 0.062 | 0.15 | quiet |
| motor persistenceMimics response-history bias exactly; controlled by randomising the dial start. | group c_motor (dial-start pull)95% CI [0.043, 0.048], p=0.000 | 0.045 | 0.00 | FIRED |
| circular wraparound bugManufactures huge errors near the 0/180 boundary that correlate with history. | mean |error| near boundary / elsewhere | 0.917 | 1.50 | quiet |
| non engagementDetected by the circular concentration of the response error. | min circular R of error across observers | 0.927 | 0.35 | quiet |
| response quantisationCoarse dial or keyboard stepping compresses the error distribution. | P(response within 0.25 deg of a 5-deg grid point) | 0.098 | 0.30 | quiet |
| model non identifiabilityCorrelated recovery errors; the design, not the data, is the limit. | corr(DoG stimulus basis, DoG response basis)cue-congruent 0.88, incongruent -0.23 | 0.308 | 0.80 | quiet |
| history confoundingDifference between the unadjusted and adjusted amplitude. | group a_stim (unadjusted - adjusted), deg95% CI [0.391, 0.876], p=0.000 | 0.687 | 0.00 | FIRED |
| insufficient trialsAmplitude estimates are biased and high-variance below ~600 trials. | median analysed trials per observerBelow 600 the amplitude estimator is biased and high-variance (results/parameter_recovery.parquet). | 600.000 | 600.00 | quiet |
| overfit tuning curvePeak location pinned at the parameter boundary. | fraction of observers with DoG peak at a bound | 0.033 | 0.10 | quiet |
| bimodal respondingHeavy-tailed or bimodal error distribution not captured by the lapse term. | fraction of observers whose error kurtosis exceeds their own fitted noise+lapse model's 97.5th percentilemedian observed excess kurtosis 12.9; a fixed threshold here would fire on a correctly specified lapse model | 0.000 | 0.10 | quiet |
| central tendencyThe single largest false-positive route; see the red-team study. | group c_mean (running-mean pull)95% CI [0.044, 0.048], p=0.000 | 0.046 | 0.00 | FIRED |
results/failures.parquet · run failures-e756953-8Failures found while building this
Four bugs were caught by the validation studies before any result was reported, and each is now pinned by a test. Models in a comparison were scored on different trials, so the multi-lag model won even on data with no history. A running-mean regressor that did not exist on the first trials of a block let central tendency leak into the serial-dependence amplitude. A bootstrap interval at small sample sizes could exclude its own point estimate. And fitting one uncertainty condition at a time redefined “the previous trial” as the previous trial of the same condition. They are listed in research/decision_log.md because a pipeline that reports its own bugs is easier to trust than one that reports none.