[Benchmark] JMH simple entry

JMH simple entry

What is JMH

JMH is Java Microbenchmark Harness acronym. Chinese meaning roughly “JAVA micro benchmark suite.” First to understand what is the “benchmark.” Baidu Baike to is defined as follows:

It refers to the design benchmark scientific testing methods, test tools and test systems to achieve a performance of a particular type of test object and quantitative comparison test.

Performance testing software can be a simple analogy to our common master Lu computer, or cell phone used to run sub software security Bunny and the like. They are certain benchmarks or go on to test the performance of an object in a particular condition, such as graphics, IO, CPU and the like.

Why use JMH

Benchmark characteristics are summarized as follows:

①, Repeatability: Repeatability test can be carried out, this is conducive to compare each test result, long-term trends of performance results, a reference for the system tuning and capacity planning on the front line.

②, observability: through a full range of monitoring (including testing start to finish, perform, server, database), understand and analyze what happened during the test.

③, may be illustrative: Related art can understand intuitively obvious Test Results (web interface, dashboard, line tree, etc).

④, authenticity: the test results reflect the real situation to the customer experience (true and accurate business scenarios + production + configuration consistent with reasonable correct test method).

⑤, enforceability: personnel can quickly modify the test to verify tuning (which can be positioned Analysis).

Once seen to be done in line with the characteristics of the benchmark, it is very complicated and very difficult. External factors can easily affect the final test results. Especially for JAVA benchmarks.
    Some articles will tell us is written in JAVA C ++, JAVA write better programs in general are less likely than writing C ++ code to run efficiently. But in some scenes JAVA C ++ run more efficiently than it did. Do not think fantasy. In fact, JVM With years of development has become very smart, it will continue to optimize during operation.
    This is good for our program is, but for performance testing on a headache. The number of times you run at different times and different results may be obtained, it is difficult to obtain a stable result. In this case, there is a solution to a large number of repeat calls, and also to a certain warm-up before the real test, the results are as accurate as possible.

In addition to these, the results we need a good display, allows us to determine the quality of performance by these displays.

And these JMH have! ?

How to use JMH

Here we have several ways for example using string concatenation JMH to benchmark.

1. Import dependence

JMH is JDK9 comes, if you are JDK9 previous version can also import openjdk



2. Directory Structure

├── pom.xml
└── src
   ├── main
   │  └── java
   │     └── cn
   │        └── coder4j
   │           └── study
   │              └── demo
   │                 └── jmh
   │                    ├── benchmark
   │                    │  └──
   │                    └── runner
   │                       └──
   └── test
      └── java
         └── cn
            └── coder4j
               └── study
                  └── demo

3. Specific Code

 * Copyright (C) 2013-2018 All Rights Reserved.

import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

 * @author buhao
 * @version, v 0.1 2018-12-25 09:53 buhao
public class StringBuilderRunner {

    public static void main( String[] args ) throws RunnerException {
        Options opt = new OptionsBuilder()
                // 导入要测试的类
                // 预热5轮
                // 度量10轮

        new Runner(opt).run();


 * Copyright (C) 2013-2018 All Rights Reserved.

import org.openjdk.jmh.annotations.Benchmark;

 * @author buhao
 * @version, v 0.1 2018-12-25 09:29 buhao
public class StringConnectBenchmark {

     * 字符串拼接之 StringBuilder 基准测试
    public void testStringBuilder() {
        print(new StringBuilder().append(1).append(2).append(3).toString());

     * 字符串拼接之直接相加基准测试
    public void testStringAdd() {
        print(new String()+ 1 + 2 + 3);

     * 字符串拼接之String Concat基准测试
    public void testStringConcat() {
        print(new String().concat("1").concat("2").concat("3"));

     * 字符串拼接之 StringBuffer 基准测试
    public void testStringBuffer() {
        print(new StringBuffer().append(1).append(2).append(3).toString());

     * 字符串拼接之 StringFormat 基准测试
    public void testStringFormat(){
        print(String.format("%s%s%s", 1, 2, 3));

    public void print(String str) {


4. Run results

# Run progress: 93.33% complete, ETA 00:00:15
# Fork: 3 of 3
objc[12440]: Class JavaLaunchHelper is implemented in both /Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/bin/java (0x106a7d4c0) and /Library/Java/JavaVirtualMachines/jdk1.8.0_91.jdk/Contents/Home/jre/lib/libinstrument.dylib (0x106af74e0). One of the two will be used. Which one is undefined.
# Warmup Iteration   1: 747281.755 ops/s
# Warmup Iteration   2: 924220.081 ops/s
# Warmup Iteration   3: 1129741.585 ops/s
# Warmup Iteration   4: 1135268.541 ops/s
# Warmup Iteration   5: 1062994.936 ops/s
Iteration   1: 1142834.160 ops/s
Iteration   2: 1143207.472 ops/s
Iteration   3: 1178363.827 ops/s
Iteration   4: 1156408.897 ops/s
Iteration   5: 1123123.829 ops/s
Iteration   6: 1086029.992 ops/s
Iteration   7: 1108795.147 ops/s
Iteration   8: 1125522.731 ops/s
Iteration   9: 1120021.744 ops/s
Iteration  10: 1119916.181 ops/s

Result "":
  1132633.183 ±(99.9%) 16252.303 ops/s [Average]
  (min, avg, max) = (1082146.355, 1132633.183, 1182418.648), stdev = 24325.684
  CI (99.9%): [1116380.879, 1148885.486] (assumes normal distribution)

# Run complete. Total time: 00:03:57

Benchmark                                  Mode  Cnt          Score         Error  Units
StringConnectBenchmark.testStringAdd      thrpt   30   63728919.269 ±  906608.141  ops/s
StringConnectBenchmark.testStringBuffer   thrpt   30  112423521.098 ± 1157072.848  ops/s
StringConnectBenchmark.testStringBuilder  thrpt   30  110558976.274 ±  654163.111  ops/s
StringConnectBenchmark.testStringConcat   thrpt   30   44820009.200 ±  524305.660  ops/s
StringConnectBenchmark.testStringFormat   thrpt   30    1132633.183 ±   16252.303  ops/s

The code analysis

  • StringBuilderRunner

The role of this class runner, is to start benchmarking.
    JMH There are usually two ways to start, one is through the command line using the maven command. This case is suitable for large benchmarks, like those many, many times you want to run, and run a very long time. You can directly play a jar package, sent to the server, knocking command would not have cared, over tens of minutes, a few hours, a few days back to see the results.

But in many cases, we simply want to test a small function, but also do not need to run servers. JMH it also provides a way to run through the Main method, as shown in the above code.

In the Main method, to run through org.openjdk.jmh.runner.Runner class instance can org.openjdk.jmh.runner.options.Options. The focus here is to construct Options object. Official provides a OptionsBuilder objects to build. The Builder objects are streaming in. It is commonly used method and the corresponding annotation form:

Method name



The corresponding notes

To run the benchmark tests the simple name like eg. StringConnectBenchmark

Specifies the benchmark class to run

Do not run benchmarks simple name like eg. StringConnectBenchmark

Do not run specified benchmark class of

The number of iterations preheated

Specifies the number of iterations preheated

Preheat the batch size

Specify the size of the batch preheating

Preheat mode: INDI, BULK, BULK_INDI

Designated warm-up mode

Preheat mode

Designated warm-up mode

Preheating time

Designated warm-up time

The number of iterations of the test

Specifies the number of iterations of the test

The size of the test batch

Specify the size of the test batch

The test of time

Time specified test

Test mode: Throughput (throughput), averageTime (mean time), the SampleTime (in the test, the random sampling execution time), SingleShotTime (Processed calculated in each execution), All

Specifies the test pattern

warmupIterations @Warmup
warmupBatchSize @Warmup
warmupForks @Warmup
warmupMode @Warmup
warmupTime @Warmup
measurementIterations @Measurement
measurementBatchSize @Measurement
measurementTime @Measurement
mode @BenchmarkMode
  • StringConnectBenchmark

This is the real implementation of the benchmark class, much like the unit test classes, each test method you write test code to be executed. But here the @Test replaced @Benchmark comment.
    And added to this is the benchmark method specified in this method, when the Main method Runner class runs, it will find those annotated modified method, and then specify the rule to benchmark. Of course, different approaches may sometimes need different rules, this time in the form of notes can correspond by the above method to specify a method to rule alone.

6. analytical results

The result is divided into three parts.
    The first part “#Warmup Iteration ….” this form of content. This suggests that the results of each iteration of the warm-up.
    Another part to “Iteration …” form content, which shows the results of each benchmark iteration.

The last portion of the content “Result …” form, which is all iterations finish the final result. The results of the first paragraph tells us the maximum, minimum, average value information.
    The information most of the final table structure is the focus of our analysis, but the results it outputs a bit misplaced, the beginning I have been tangled Error is ± 906608.141 what does it mean, google found a circle, it is actually nothing Error output and Score is 63728919.269 ± 906608.141. I used a little table row board, explained as follows:

Methods benchmarks executed

Test mode, here is the throughput

How many times running




Benchmark Mode Cnt Score Error Units
StringConnectBenchmark.testStringAdd thrpt 30 63728919.269 ±  906608.141 ops/s
StringConnectBenchmark.testStringBuffer thrpt 30 112423521.098 ± 1157072.848 ops/s
StringConnectBenchmark.testStringBuilder thrpt 30 110558976.274 ±  654163.111 ops/s
StringConnectBenchmark.testStringConcat thrpt 30 44820009.200 ±  524305.660 ops/s
StringConnectBenchmark.testStringFormat thrpt 30 1132633.183 ±   16252.303 ops/s

in conclusion:

StringBuffer> = StringBuilder> String added directly> StringConcat >> StringFormat

可见 StringBuffer 与 StringBuilder 大致性能相同,都比直接相加高几个数量级,而且直接相加与 Concat 方法相加差不多。但是这里不管哪种都比 StringFormat高 N 个数量级。所以 String的 Format方法一定要慎用、不用、禁用!!!

Related Links

Reference links

  1. openjdk official DEMO

  2. openjdk official DEMO (translated version)

  3. On Benchmark

  4. What is the benchmark

  5. JMH very good study notes ←

  6. JMH done using JAVA benchmarks

  7. JMH do JAVA benchmarks

Code link

    DEMO Code link

No public concern “KIWI Sui Suinian” share more than just technology

Leave a Reply