Select Page

Use JMH for Your Java Applications With Gradle

Emmanouil Gkatziouras
Published: November 3, 2022

If you want to benchmark your code, the Java Microbenchmark Harness is the tool of choice.Java logo
In our example, we will use the refill-rate-limiter project. 

Since refill-rate-limiter uses Gradle, we will use the following plugin for Gradle:

plugins {
...
  id "me.champeau.gradle.jmh" version "0.5.3"
...
}

We will place the benchmark in the jmh/java/io/github/resilience4j/ratelimiter folder.

Our benchmark should look like this:

package io.github.resilience4j.ratelimiter;
 
import io.github.resilience4j.ratelimiter.internal.RefillRateLimiter;
import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.infra.Blackhole;
import org.openjdk.jmh.profile.GCProfiler;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;
 
import java.time.Duration;
import java.util.concurrent.TimeUnit;
import java.util.function.Supplier;
 
@State(Scope.Benchmark)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
@BenchmarkMode(Mode.All)
public class RateLimiterBenchmark {
 
    private static final int FORK_COUNT = 2;
    private static final int WARMUP_COUNT = 10;
    private static final int ITERATION_COUNT = 10;
    private static final int THREAD_COUNT = 2;
 
    private RefillRateLimiter refillRateLimiter;
 
    private Supplier<String> refillGuardedSupplier;
 
    public static void main(String[] args) throws RunnerException {
        Options options = new OptionsBuilder()
                .addProfiler(GCProfiler.class)
                .build();
        new Runner(options).run();
    }
 
    @Setup
    public void setUp() {
 
        RefillRateLimiterConfig refillRateLimiterConfig = RefillRateLimiterConfig.custom()
                                                                                 .limitForPeriod(1)
                                                                                 .limitRefreshPeriod(Duration.ofNanos(1))
                                                                                 .timeoutDuration(Duration.ofSeconds(5))
                                                                                 .build();
 
        refillRateLimiter = new RefillRateLimiter("refillBased", refillRateLimiterConfig);
 
        Supplier<String> stringSupplier = () -> {
            Blackhole.consumeCPU(1);
            return "Hello Benchmark";
        };
 
        refillGuardedSupplier = RateLimiter.decorateSupplier(refillRateLimiter, stringSupplier);
    }
 
    @Benchmark
    @Threads(value = THREAD_COUNT)
    @Warmup(iterations = WARMUP_COUNT)
    @Fork(value = FORK_COUNT)
    @Measurement(iterations = ITERATION_COUNT)
    public String refillPermission() {
        return refillGuardedSupplier.get();
    }
 
}

Let’s now check the elements one by one.

By using benchmark scope, all of the threads used on the benchmark scope will share the same object. We do so because we want to test how refill-rate-limiter performs in a multithreaded scenario.

We would like our results to be reported in microseconds, therefore we shall use the OutputTimeUnit.

@OutputTimeUnit(TimeUnit.MICROSECONDS)

On JMH, we have various benchmark modes depending on what we want to measure.

  • Throughput is when we want to measure the number of operations per unit of time.
  • AverageTime is when we want to measure the average time per operation.
  • SampleTime is when we want to sample the time for each operation including min, max time, and more than just the average.
  • SingleShotTime is when we want to measure the time for a single operation. This can help when we want to identify how the operation will do on a cold start.

We also have the option to measure all of the above.

Those options configured on the class level will apply to the benchmark methods we will add.

Let’s also examine how the benchmark will run. We will specify the number of threads by using the @Threads annotation.

@Threads(value = THREAD_COUNT)

Also, we want to warm up before we run the actual benchmarks. This way our code will be initialized, online optimizations will take place, and our runtime will adapt to the conditions before we run the benchmarks.

@Warmup(iterations = WARMUP_COUNT)

Using @Fork we will instruct how many times the benchmark will run.

@Fork(value = FORK_COUNT)

Then we need to specify the number of iterations we want to measure:

@Measurement(iterations = ITERATION_COUNT)

We can start our test by just using:

The results will be saved in a file.

...
2022-10-28T09:08:44.522+0100 [QUIET] [system.out] Benchmark result is saved to /path/refill-rate-limiter/build/reports/jmh/results.txt
..

Let’s examine the results.

Benchmark                                                         Mode       Cnt      Score   Error   Units
RateLimiterBenchmark.refillPermission                            thrpt        20     13.594 ± 0.217  ops/us
RateLimiterBenchmark.refillPermission                             avgt        20      0.147 ± 0.002   us/op
RateLimiterBenchmark.refillPermission                           sample  10754462      0.711 ± 0.025   us/op
RateLimiterBenchmark.refillPermission:refillPermission·p0.00    sample                  ≈ 0           us/op
RateLimiterBenchmark.refillPermission:refillPermission·p0.50    sample                0.084           us/op
RateLimiterBenchmark.refillPermission:refillPermission·p0.90    sample                0.125           us/op
RateLimiterBenchmark.refillPermission:refillPermission·p0.95    sample                0.125           us/op
RateLimiterBenchmark.refillPermission:refillPermission·p0.99    sample                0.209           us/op
RateLimiterBenchmark.refillPermission:refillPermission·p0.999   sample              139.008           us/op
RateLimiterBenchmark.refillPermission:refillPermission·p0.9999  sample              935.936           us/op
RateLimiterBenchmark.refillPermission:refillPermission·p1.00    sample            20709.376           us/op
RateLimiterBenchmark.refillPermission    

As we can see, we have the modes listed.

Countis the number of iterations. Apart from throughput where we measure the operations by time, the rest is the time per operation. Throughput, Average, and Single shot are straightforward. Sample lists the percentiles. Error is the margin of error.

That’s it! Happy benchmarking!

Source: dzone.com