abeval — eval sample-size calculator
Description: How many eval items do you need before a difference between two LLM runs is detectable? Paired-design sample-size calculator.

# abeval — sample-size calculator

How many items does your eval need before the difference you care about is distinguishable from noise? Paired design: both runs scored on the same items. Same formulas as `abeval power`.

Baseline accuracy (%) where run A sits today
Difference to detect (points) the improvement you'd act on
Item correlation 0.5 typical for two variants of one system
Power (%) chance of detecting a real effect
– paired items needed

Smallest detectable difference at each eval size (log-scaled items axis)

Items in your eval | Smallest detectable difference
--- | ---

Two-sided test at 95% confidence on the mean of per-item differences, normal approximation. Verify locally: `pip install abeval` then `abeval power --baseline 0.75 --delta 0.03`. MIT-licensed — github.com/mohammadi-hadi/abeval.