How many observers, how many trials
Power here is a property of the design and the estimator under an assumed effect size. It is not a promise about a real study.
For each amplitude and session length, 250 independent synthetic observers were simulated and fitted. A study of N observers was then formed by resampling N of them and running the same observer-level test as the primary analysis, 4,000 times. At a true amplitude of zero the rejection rate across all cells ranges from 4.3% to 6.0% (nominal 5%); resampling from a finite pool treats the pool’s own mean as the truth, so these are slightly conservative about calibration.
With 600 trials per observer, 80% power is reached at 24 observers for 0.5°; 8 observers for 1.0°; 8 observers for 1.5°; 8 observers for 2.0°; 8 observers for 3.0°. The literature reports amplitudes over a wide range; the grid spans it rather than betting on one value, and nothing here should be read as the effect size a human study will find.
100%probability of detecting the effect with the correct sign
Fig. 1 Power by number of observers, 300 trials each
SIMULATED DATA
- 0°
- 0.5°
- 1°
- 1.5°
- 2°
- 3°
results/power.parquet · run power-8964a2e-5000Table view (48 rows)
| true a (°) | observers | power | any rejection |
|---|---|---|---|
| 0.00 | 8 | 6.0% | 6.0% |
| 0.00 | 12 | 5.6% | 5.6% |
| 0.00 | 16 | 5.3% | 5.3% |
| 0.00 | 20 | 5.4% | 5.4% |
| 0.00 | 24 | 5.0% | 5.0% |
| 0.00 | 30 | 5.6% | 5.6% |
| 0.00 | 40 | 5.1% | 5.1% |
| 0.00 | 60 | 5.1% | 5.1% |
| 0.50 | 8 | 28.1% | 28.1% |
| 0.50 | 12 | 38.0% | 38.0% |
| 0.50 | 16 | 47.4% | 47.4% |
| 0.50 | 20 | 57.1% | 57.1% |
| 0.50 | 24 | 63.5% | 63.5% |
| 0.50 | 30 | 73.6% | 73.6% |
| 0.50 | 40 | 85.7% | 85.7% |
| 0.50 | 60 | 95.3% | 95.3% |
| 1.00 | 8 | 78.5% | 78.5% |
| 1.00 | 12 | 90.0% | 90.0% |
| 1.00 | 16 | 94.5% | 94.5% |
| 1.00 | 20 | 97.9% | 97.9% |
| 1.00 | 24 | 99.0% | 99.0% |
| 1.00 | 30 | 99.9% | 99.9% |
| 1.00 | 40 | 100.0% | 100.0% |
| 1.00 | 60 | 100.0% | 100.0% |
| 1.50 | 8 | 97.2% | 97.2% |
| 1.50 | 12 | 99.8% | 99.8% |
| 1.50 | 16 | 100.0% | 100.0% |
| 1.50 | 20 | 100.0% | 100.0% |
| 1.50 | 24 | 100.0% | 100.0% |
| 1.50 | 30 | 100.0% | 100.0% |
| 1.50 | 40 | 100.0% | 100.0% |
| 1.50 | 60 | 100.0% | 100.0% |
| 2.00 | 8 | 99.7% | 99.7% |
| 2.00 | 12 | 100.0% | 100.0% |
| 2.00 | 16 | 100.0% | 100.0% |
| 2.00 | 20 | 100.0% | 100.0% |
| 2.00 | 24 | 100.0% | 100.0% |
| 2.00 | 30 | 100.0% | 100.0% |
| 2.00 | 40 | 100.0% | 100.0% |
| 2.00 | 60 | 100.0% | 100.0% |
| 3.00 | 8 | 100.0% | 100.0% |
| 3.00 | 12 | 100.0% | 100.0% |
| 3.00 | 16 | 100.0% | 100.0% |
| 3.00 | 20 | 100.0% | 100.0% |
| 3.00 | 24 | 100.0% | 100.0% |
| 3.00 | 30 | 100.0% | 100.0% |
| 3.00 | 40 | 100.0% | 100.0% |
| 3.00 | 60 | 100.0% | 100.0% |
Fig. 2 Power by number of observers, 600 trials each
SIMULATED DATA
- 0°
- 0.5°
- 1°
- 1.5°
- 2°
- 3°
results/power.parquet · run power-8964a2e-5000Table view (48 rows)
| true a (°) | observers | power | any rejection |
|---|---|---|---|
| 0.00 | 8 | 5.5% | 5.5% |
| 0.00 | 12 | 5.1% | 5.1% |
| 0.00 | 16 | 4.5% | 4.5% |
| 0.00 | 20 | 4.3% | 4.3% |
| 0.00 | 24 | 5.0% | 5.0% |
| 0.00 | 30 | 4.7% | 4.7% |
| 0.00 | 40 | 5.9% | 5.9% |
| 0.00 | 60 | 5.9% | 5.9% |
| 0.50 | 8 | 40.0% | 40.1% |
| 0.50 | 12 | 51.6% | 51.6% |
| 0.50 | 16 | 65.7% | 65.7% |
| 0.50 | 20 | 73.9% | 73.9% |
| 0.50 | 24 | 82.5% | 82.5% |
| 0.50 | 30 | 89.6% | 89.6% |
| 0.50 | 40 | 95.9% | 95.9% |
| 0.50 | 60 | 99.4% | 99.4% |
| 1.00 | 8 | 94.5% | 94.5% |
| 1.00 | 12 | 99.6% | 99.6% |
| 1.00 | 16 | 100.0% | 100.0% |
| 1.00 | 20 | 100.0% | 100.0% |
| 1.00 | 24 | 100.0% | 100.0% |
| 1.00 | 30 | 100.0% | 100.0% |
| 1.00 | 40 | 100.0% | 100.0% |
| 1.00 | 60 | 100.0% | 100.0% |
| 1.50 | 8 | 100.0% | 100.0% |
| 1.50 | 12 | 100.0% | 100.0% |
| 1.50 | 16 | 100.0% | 100.0% |
| 1.50 | 20 | 100.0% | 100.0% |
| 1.50 | 24 | 100.0% | 100.0% |
| 1.50 | 30 | 100.0% | 100.0% |
| 1.50 | 40 | 100.0% | 100.0% |
| 1.50 | 60 | 100.0% | 100.0% |
| 2.00 | 8 | 100.0% | 100.0% |
| 2.00 | 12 | 100.0% | 100.0% |
| 2.00 | 16 | 100.0% | 100.0% |
| 2.00 | 20 | 100.0% | 100.0% |
| 2.00 | 24 | 100.0% | 100.0% |
| 2.00 | 30 | 100.0% | 100.0% |
| 2.00 | 40 | 100.0% | 100.0% |
| 2.00 | 60 | 100.0% | 100.0% |
| 3.00 | 8 | 100.0% | 100.0% |
| 3.00 | 12 | 100.0% | 100.0% |
| 3.00 | 16 | 100.0% | 100.0% |
| 3.00 | 20 | 100.0% | 100.0% |
| 3.00 | 24 | 100.0% | 100.0% |
| 3.00 | 30 | 100.0% | 100.0% |
| 3.00 | 40 | 100.0% | 100.0% |
| 3.00 | 60 | 100.0% | 100.0% |