Hyperfine: Easy benchmarking

18 Jul 2020

Very often, when I want to test the performance of something, most of the time I end up using time. time, however, lacks a certain amount of scientific rigor, it only runs the thing you’re testing once and gives three different times which can mean slightly different things in different contexts.

Enter, Hyperfine. Here are three things I love about it:

1. It runs commands multiple times.

This ensures that there is no random variation in your testing, it even warns you if there are major outliers:

$ hyperfine 'ls'
Benchmark #1: ls
  Time (mean ± σ):       1.6 ms ±   0.3 ms    [User: 0.6 ms, System: 0.8 ms]
  Range (min … max):     1.2 ms …   5.1 ms    516 runs

  Warning: Command took less than 5 ms to complete. Results might be inaccurate.
  Warning: Statistical outliers were detected. Consider re-running this benchmark on a quiet PC without any interferences from other programs. It might help to use the '--warmup' or '--prepare' options.

2. Parameters

You can provide a lister of parameters (or even ranges) and they can be substituted in, letting you see how a program scales with different variables.

$ hyperfine --parameter-list duration 1,2,3 'sleep {duration}'
Benchmark #1: sleep 1
  Time (mean ± σ):      1.009 s ±  0.001 s    [User: 1.0 ms, System: 1.8 ms]
  Range (min … max):    1.006 s …  1.011 s    10 runs

Benchmark #2: sleep 2
  Time (mean ± σ):      2.008 s ±  0.002 s    [User: 1.2 ms, System: 2.3 ms]
  Range (min … max):    2.005 s …  2.011 s    10 runs

Benchmark #3: sleep 3
  Time (mean ± σ):      3.009 s ±  0.003 s    [User: 1.3 ms, System: 2.4 ms]
  Range (min … max):    3.005 s …  3.012 s    10 runs

  'sleep 1' ran
    1.99 ± 0.00 times faster than 'sleep 2'
    2.98 ± 0.00 times faster than 'sleep 3'

3. It’s just so simple

It’s just as easy as running time command, except it is far more accurate and repeatable, and if you want to use some more advanced features, they’re super simple too.