How to Write Accurate Benchmarks in Go

  • Not resetting or pausing the timer
  • Making wrong assumptions about micro-benchmarks
  • Not being careful about compiler optimizations
  • Being fooled by the observer effect

General Concepts

$ go test -bench=.
cpu: Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
BenchmarkFoo-4 73 16511228 ns/op
$ go test -bench=. -benchtime=2s
BenchmarkFoo-4 150 15832169 ns/op

Not Resetting or Pausing the Timer

Making Wrong Assumptions about Micro-Benchmarks

cpu: Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
BenchmarkAtomicStoreInt32
BenchmarkAtomicStoreInt32-4 197107742 5.682 ns/op
BenchmarkAtomicStoreInt64
BenchmarkAtomicStoreInt64-4 213917528 5.134 ns/op
BenchmarkAtomicStoreInt64
BenchmarkAtomicStoreInt64-4 224900722 5.434 ns/op
BenchmarkAtomicStoreInt32
BenchmarkAtomicStoreInt32-4 230253900 5.159 ns/op
$ go test -bench=. -count=10 | tee stats.txt
cpu: Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
BenchmarkAtomicStoreInt32-4 234935682 5.124 ns/op
BenchmarkAtomicStoreInt32-4 235307204 5.112 ns/op
// ...
BenchmarkAtomicStoreInt64-4 235548591 5.107 ns/op
BenchmarkAtomicStoreInt64-4 235210292 5.090 ns/op
// ...
$ benchstat stats.txt
name time/op
AtomicStoreInt32-4 5.10ns ± 1%
AtomicStoreInt64-4 5.10ns ± 1%

Not Being Careful about Compiler Optimizations

cpu: Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
BenchmarkPopcnt1-4 1000000000 0.2858 ns/op
  1. During each loop iteration, assign the result to a local variable (local in the context of the benchmark function).
  2. Assign the latest result to a global variable.
cpu: Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
BenchmarkPopcnt1-4 1000000000 0.2858 ns/op
BenchmarkPopcnt2-4 606402058 1.993 ns/op

Being Fooled by the Observer Effect

Computing the sum of the first eight columns
cpu: Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
BenchmarkCalculateSum512-4 81854 15073 ns/op
BenchmarkCalculateSum513-4 161479 7358 ns/op
cpu: Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
BenchmarkCalculateSum512-4 1116 33547 ns/op
BenchmarkCalculateSum513-4 998 35507 ns/op
❤️ Enjoying my work? You can consider becoming a GitHub sponsor: https://github.com/sponsors/teivah.

Conclusion

  • Use time methods to preserve the accuracy of a benchmark.
  • Increasing benchtime or using tools such as benchstat can be helpful when dealing with micro-benchmarks.
  • Be careful with the results of a micro-benchmark if the system that ends up running the application is different from the one running the micro-benchmark.
  • Make sure the function under test leads to a side effect, to prevent compiler optimizations from fooling you about the benchmark results.
  • To prevent the observer effect, force a benchmark to re-create the data used by a CPU-bound function.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Teiva Harsanyi

Teiva Harsanyi

3.5K Followers

Software Engineer @Docker 🐳 | 📖 100 Go Mistakes author | 改善