If you want to benchmark your code, the Java Microbenchmark Harness is the tool of choice.
In our example, we will use the refill-rate-limiter project.
Since refill-rate-limiter uses Gradle, we will use the following plugin for Gradle:
plugins {
...
id "me.champeau.gradle.jmh" version "0.5.3"
...
}We will place the benchmark in the jmh/java/io/github/resilience4j/ratelimiter folder.
Our benchmark should look like this:
package io.github.resilience4j.ratelimiter;
import io.github.resilience4j.ratelimiter.internal.RefillRateLimiter;
import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.infra.Blackhole;
import org.openjdk.jmh.profile.GCProfiler;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;
import java.time.Duration;
import java.util.concurrent.TimeUnit;
import java.util.function.Supplier;
@State(Scope.Benchmark)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
@BenchmarkMode(Mode.All)
public class RateLimiterBenchmark {
private static final int FORK_COUNT = 2;
private static final int WARMUP_COUNT = 10;
private static final int ITERATION_COUNT = 10;
private static final int THREAD_COUNT = 2;
private RefillRateLimiter refillRateLimiter;
private Supplier<String> refillGuardedSupplier;
public static void main(String[] args) throws RunnerException {
Options options = new OptionsBuilder()
.addProfiler(GCProfiler.class)
.build();
new Runner(options).run();
}
@Setup
public void setUp() {
RefillRateLimiterConfig refillRateLimiterConfig = RefillRateLimiterConfig.custom()
.limitForPeriod(1)
.limitRefreshPeriod(Duration.ofNanos(1))
.timeoutDuration(Duration.ofSeconds(5))
.build();
refillRateLimiter = new RefillRateLimiter("refillBased", refillRateLimiterConfig);
Supplier<String> stringSupplier = () -> {
Blackhole.consumeCPU(1);
return "Hello Benchmark";
};
refillGuardedSupplier = RateLimiter.decorateSupplier(refillRateLimiter, stringSupplier);
}
@Benchmark
@Threads(value = THREAD_COUNT)
@Warmup(iterations = WARMUP_COUNT)
@Fork(value = FORK_COUNT)
@Measurement(iterations = ITERATION_COUNT)
public String refillPermission() {
return refillGuardedSupplier.get();
}
}Let’s now check the elements one by one.
By using benchmark scope, all of the threads used on the benchmark scope will share the same object. We do so because we want to test how refill-rate-limiter performs in a multithreaded scenario.
We would like our results to be reported in microseconds, therefore we shall use the OutputTimeUnit.
@OutputTimeUnit(TimeUnit.MICROSECONDS)On JMH, we have various benchmark modes depending on what we want to measure.
Throughputis when we want to measure the number of operations per unit of time.AverageTimeis when we want to measure the average time per operation.SampleTimeis when we want to sample the time for each operation including min, max time, and more than just the average.SingleShotTimeis when we want to measure the time for a single operation. This can help when we want to identify how the operation will do on a cold start.
We also have the option to measure all of the above.
Those options configured on the class level will apply to the benchmark methods we will add.
Let’s also examine how the benchmark will run. We will specify the number of threads by using the @Threads annotation.
@Threads(value = THREAD_COUNT)Also, we want to warm up before we run the actual benchmarks. This way our code will be initialized, online optimizations will take place, and our runtime will adapt to the conditions before we run the benchmarks.
@Warmup(iterations = WARMUP_COUNT)Using @Fork we will instruct how many times the benchmark will run.
@Fork(value = FORK_COUNT)Then we need to specify the number of iterations we want to measure:
@Measurement(iterations = ITERATION_COUNT)We can start our test by just using:
The results will be saved in a file.
...
2022-10-28T09:08:44.522+0100 [QUIET] [system.out] Benchmark result is saved to /path/refill-rate-limiter/build/reports/jmh/results.txt
..Let’s examine the results.
Benchmark Mode Cnt Score Error Units
RateLimiterBenchmark.refillPermission thrpt 20 13.594 ± 0.217 ops/us
RateLimiterBenchmark.refillPermission avgt 20 0.147 ± 0.002 us/op
RateLimiterBenchmark.refillPermission sample 10754462 0.711 ± 0.025 us/op
RateLimiterBenchmark.refillPermission:refillPermission·p0.00 sample ≈ 0 us/op
RateLimiterBenchmark.refillPermission:refillPermission·p0.50 sample 0.084 us/op
RateLimiterBenchmark.refillPermission:refillPermission·p0.90 sample 0.125 us/op
RateLimiterBenchmark.refillPermission:refillPermission·p0.95 sample 0.125 us/op
RateLimiterBenchmark.refillPermission:refillPermission·p0.99 sample 0.209 us/op
RateLimiterBenchmark.refillPermission:refillPermission·p0.999 sample 139.008 us/op
RateLimiterBenchmark.refillPermission:refillPermission·p0.9999 sample 935.936 us/op
RateLimiterBenchmark.refillPermission:refillPermission·p1.00 sample 20709.376 us/op
RateLimiterBenchmark.refillPermission As we can see, we have the modes listed.
Countis the number of iterations. Apart from throughput where we measure the operations by time, the rest is the time per operation. Throughput, Average, and Single shot are straightforward. Sample lists the percentiles. Error is the margin of error.
That’s it! Happy benchmarking!
