Elasticsearch Benchmarks

Results

Overview

On this page are the results of the Elasticsearch nightly benchmarks based on the main branch as of that point in time. The Apache Software Foundation also provides a similar page for the Lucene nightly benchmarks.

Benchmarks

Benchmarking Methodology

All benchmarks are run by Rally against the Elasticsearch main branch as of that date. The benchmark uses four bare-metal server-class machines. On one we run the benchmark driver (Rally), on the other three the benchmark candidate (one to three Elasticsearch nodes, one per machine). All machines are connected via a dedicated 10 GBit switched network.

The benchmarks are intentionally not scalability benchmarks but rather show the performance characteristics of Elasticsearch ranging from one node to at most three nodes and are mainly intended to help the development team spot performance regressions.

Benchmarking Environment

High shard count environment

Benchmarks run on ephemeral AWS environments using 3x c6gd.8xlarge instances as dedicated master nodes and 6x m6gd.4xlarge instances for the hot tier. Don't use those numbers to compare with other benchmarks! Indeed:

  • Those benchmarks are meant to detect regressions in Elasticsearch: we make sure performance with high shard counts stays the same (or gets better) over time.
  • The AWS instance types used have vastly different performance characteristics than our dedicated servers.

ARM environment

We run nightly Graviton2 benchmarks using the m6gd.metal AWS instance type. Don't use those numbers to compare x86 vs. ARM performance! Indeed:

  • Those benchmarks are meant to detect regressions in Elasticsearch: we make sure they're stable, not fast.
  • The ARM instance types that we use have vastly different performance characteristics than our dedicated servers.

Dedicated servers

All other benchmarks are run on bare-metal machines with the following specifications:

  • CPU: Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz
  • RAM: 32 GB
  • SSD: Micron_1100_MTFDDAK512TBN
  • OS for Rally load driver servers:
    • Linux kernel version 5.4.0-65
  • OS for Elasticsearch target servers:
    • Linux kernel version 5.4.0-65
    • /sys/kernel/mm/transparent_hugepage/enabled = always
    • /sys/kernel/mm/transparent_hugepage/defrag = always
    • /sys/devices/system/cpu/intel_pstate/no_turbo = 1
  • JVM: JDK bundled with Elasticsearch.

All machines are connected via a dedicated 10 GBit switched network.

History of changes

  • 2022-11-07:
    • Added a new dashboard for so_vector track under Search section.
  • 2022-11-01:
    • Added two new dashboards for so-transform and so-frequent-items challenges under Machine Learning section.
  • 2022-08-28:
    • Updated both arm64 and x86_64 AWS AMIs for ubuntu-bionic-18.04-{arch}-server from 20201027 (arm64, Ubuntu 18.04.5, kernel 5.4.0-1029-aws) and 20200610 (x86_64, Ubuntu 18.04.4, kernel 5.3.0-1023-aws), respectively, to 20220810 (Ubuntu 18.04.6, kernel 5.4.0-1083-aws).
  • 2022-08-18:
    • Replaced data disk on one target machine used by the tracks pmc and http_logs with another identical model (Micron_1100_MTFDDAK512TBN).
  • 2021-11-08:
    • Enabled all the tracks and challenges to be run with security on.
  • 2021-10-03:
    • Replaced data disk on one target machine used by the tracks nested, noaa, pmc, http_logs with another identical model (Micron_1100_MTFDDAK512TBN).
  • 2021-08-23:
    • Replaced data disk on one target machine used by the tracks geonames, geopoint, percolator, nyc_taxis with another identical model (Micron_1100_MTFDDAK512TBN).
  • 2021-06-01:
    • Replaced data disk on one target machine used by the tracks nested, noaa, pmc, http_logs with another identical model (Micron_1100_MTFDDAK512TBN).
  • 2021-04-21:
    • Added new benchmarking hardware and geopointshape benchmarks.
  • 2021-02-07:
    • Upgraded all Rally load driver and Elasticsearch target servers to Ubuntu 18.04 Bionic, Linux Kernel 5.4.0-65
  • 2021-01-13:
    • Replaced data disk on one target machine used by the tracks geonames, geopoint, percolator, nyc_taxis with another identical model (Micron_1100_MTFDDAK512TBN).
  • 2020-05-18:
    • Replaced data disk on one target machine used by the tracks geonames, geopoint, percolator, nyc_taxis with another identical model (Micron_1100_MTFDDAK512TBN).
  • 2020-04-30:
    • Run nightly benchmarks with JDK bundled with Elasticsearch.
  • 2020-04-06:
    • Replaced data disk on all target machines used by the tracks nested, noaa, pmc, http_logs with another identical model (Micron_1100_MTFDDAK512TBN).
  • 2020-02-07:
    • Performed OS package upgrade for load driver server used for tracks nested, noaa, pmc, http_logs and switched to kernel 4.15.0-76-generic.
  • 2020-02-04:
    • Performed OS package upgrade for load driver server used for tracks geonames, geopoint, percolator, nyc_taxis and switched to kernel 4.15.0-76-generic.
  • 2019-12-12:
    • Replaced data disk on first target machine used by the tracks nested, noaa, pmc, http_logs with another identical model (Micron_1100_MTFDDAK512TBN).
  • 2019-04-25:
    • Switch to new benchmarking hardware and add benchmarks for default distribution
  • 2019-04-03:
  • 2018-10-02:
    • Replaced data disk (Samsung MZ7LN512HMJP-00000) on first target machine with another model (Micron_M600_MTFDDAK512MBF) resulting in higher performance in some benchmarks.
  • 2018-04-12:
    • Set transparent_hugepage enabled and defrag settings to always. This makes it consistent with the settings before the upgrade to 4.13.0.
    • Upgrade kernel from version 4.13.0-36 to 4.13.0-38.
  • 2018-02-21: Upgrade kernel from version 4.13.0-32 to 4.13.0-36.
  • 2018-02-06: Upgrade kernel from version 4.8.0-53 to 4.13.0-32.
  • 2017-10-06:
  • 2017-06-29: Upgrade JDK from version 1.8.0_101-b13 to 1.8.0_131-b11
  • 2017-06-29: Upgrade kernel from version 4.4.0-72 to 4.8.0-53
  • 2017-04-13: Upgrade kernel from version 4.4.0-38 to 4.4.0-72