Which model explains the data
Ten models, from no history at all to a Bayesian observer, scored on trials they were not fitted to.
Each synthetic observer’s six blocks are held out one at a time; the model is fitted on the other five and scored by log-likelihood on the held-out block. History predictors are built within a block, so no predictor on a test trial can depend on a training trial. Every model is scored on the identical set of trials — an early version of this pipeline let deeper-lag models drop the first trials of each block, and they won every comparison, including on data with no history at all.
On the reference cohort the best model is M4_multilag, which beats the no-history model by 0.170 nats per trial and does so for 30 of 30 observers. The cohort was generated with three lags of geometrically decaying stimulus history plus every nuisance process, so the multi-lag model is the right answer.
Fig. 1 Held-out log-likelihood relative to no history
SIMULATED DATA
Δ held-out log-likelihood per trial vs M0 · vertical rule: no-history baseline
results/model_comparison.parquet · run model_comparison-280346c-4Table view (9 rows)
| model | Δ vs M0 (nats/trial) | 95% CI | observers improved | parameters | mean AIC |
|---|---|---|---|---|---|
| M4_multilag | +0.1702 | +0.1600 to +0.1811 | 30 / 30 | 13 | 3992.9 |
| M3adj_full | +0.1661 | +0.1563 to +0.1757 | 30 / 30 | 12 | 3998.2 |
| M5_uncertainty | +0.1656 | +0.1563 to +0.1749 | 30 / 30 | 13 | 3998.8 |
| M6_bayes | +0.1547 | +0.1467 to +0.1634 | 30 / 30 | 11 | 4012.6 |
| M3_both | +0.0293 | +0.0229 to +0.0362 | 28 / 30 | 8 | 4161.0 |
| M1_stim | +0.0287 | +0.0223 to +0.0358 | 28 / 30 | 6 | 4163.2 |
| M2_resp | +0.0080 | +0.0061 to +0.0103 | 28 / 30 | 6 | 4187.9 |
| M1lin_stim | +0.0045 | +0.0028 to +0.0066 | 22 / 30 | 5 | 4192.5 |
| M0_no_history | +0.0000 | +0.0000 to +0.0000 | 0 / 30 | 4 | 4197.6 |
Reading this honestly
Held-out likelihood rewards the model that predicts best, which is not always the model that is true. How often this procedure picks the generating model, when the generating model is known, is measured on the red team and recovery pages. The posterior predictive checks in the manuscript test whether the winning model reproduces the serial-dependence curve itself, not just the spread of errors.