Hello! 🦆 I’ve been busy working on an international move,
so this issue has been quite delayed even though I started
working on at the start of May.
So apologies if you were expecting it to come out sooner;
if it’s any consolation, I made the same mistake.
This issue covers Stabilizer: Statistically Sound Performance
Evaluation (PDF) (ASPLOS 2013) by Charlie Curtsinger and Emery Berger.
You can also check out the code for the paper,
although, that has bit-rotted by this point.
The code targets LLVM 3, whereas the latest
LLVM release is LLVM 14. For context, the paper came out in 2013,
so the bit-rot is not surprising.
The pace of LLVM development is relentless,
and industrial users of LLVM have enough engineering power
to keep up with breaking changes.
Academics on the other hand, perhaps not so much.
Getting back to the main topic, the paper covers a few things:
- It discusses why accounting for the positions in memory
of code and objects is important for robust benchmarking.
- It describes Stabilizer, which is a compiler pass plus a runtime library
developed by the authors that randomizes the positions of code and objects in memory.
- It describes how powerful statistical tests can be
used to analyze the results after benchmarking code with Stabilizer.
- It delivers a spicy result:
the performance benefits of LLVM 3’s -O3 optimizations
over -O2 on the SPEC CPU2006 benchmarks are indistinguishable
from random noise.