We use PkgBenchmark.jl to conduct benchmarks, coupled with BenchmarkCI.jl to automate benchmark runs with GitHub Actions on certain PRs. Currently, PRs will not run benchmarks by default. Only PRs with the label
run benchmark will trigger benchmarks on push, adding a comment with the benchmark results.
To locally run benchmarks, you can use PkgBenchmark.jl with the following code:
# make sure PkgBenchmark and BenchmarkTools # are installed globally before running it using Meshes, PkgBenchmark benchmarkpkg(Meshes)
It will include
benchmark/benchmarks.jl and look for a
BenchmarkGroup variable named
SUITE, that it will run for you with a nice printing.
The full list of options is detailed in the PkgBenchmark documentation.
Sometimes you may prefer to run the suite manually, especially during interactive development. All you have to do is include the
benchmark/benchmarks.jl file, and run the
Note that because GitHub Actions may use different runners between benchmarks, you are likely to see fluctuations and performance changes that may not always be relevant. This makes it somewhat unreliable for tracking regressions.
Benchmark results can be very useful for validation, for example when optimizing or modifying existing features, but you are encouraged to benchmark important changes locally.