All about Elasticsearch updates

Pedro, Database Architect

May 5, 2021

Pedro is a Database Architect at The Workshop. As the Founder and current leader of the User Group Malaga Databases, Pedro loves learning about technology and exploring his passion for databases. In this article, he provides some pointers on how to manage Elasticsearch upgrades. If you are working with Elasticsearch, you have probably realised that the company publishes new versions of its software very frequently. To give you more context, let’s compare it to what other database technologies have published in the last year:

Elasticsearch shines with 16 releases — that’s more than one a month! Most other companies release one per quarter on average.

Versions in Elasticsearch

Let’s see an example of how versions, or releases, are defined in Elasticsearch:

7.12.1

  • 7 is called a major release
  • 12 is called a minor release
  • 1 is called a maintenance release

You may have experienced a minor release create some incompatibility issues. These issues may happen between components inside the Elastic stack, or ELK stack, or may occur with the application using Elasticsearch. It means that something that worked before has stopped working after applying a minor release. The most common reason is a breaking change.

Generally with other database technologies, broken compatibility changes happen with major releases. So we could consider a minor release from Elasticsearch as similar to a major release from other databases systems.

Elastic compensates for these incompatibility issues with its Maintenance Policy and a lot of documentation offering supporting during the upgrade process. The high-level steps to follow whenever you are thinking of taking upgrading to a new release are:

  1. The Preparation
  2. The Upgrade

1. The Preparation

Follow these steps at the Preparation stage of an Elasticsearch upgrade:

  1. Check the deprecation log from the Elasticsearch cluster nodes
      This can be provided by a DBA, but ideally this information will be available to the Engineering teams.
  2. Review the breaking changes
      Check the breaking changes on every minor release between the current version and target one.
  3. Review Plugins
      Plugins are quite common, eg: Monitoring.
  4. Check the code
      In the applications or services interacting with Elasticsearch, check the code and make the necessary changes.

In some cases, you might be required to do some re-indexing whenever there are indices created in previous versions. The Elasticsearch engine does accept indices from the previous major version, but it does not from older ones. For example:

  • Indices created in the 6.X version are compatible in 7.X, but 5.X indices are not.
  • Indices created in the 7.X version will be compatible in 8.X, but 6.X indices won’t work.

Indeed, if the re-indexing is not done before the upgrade, the whole Elasticsearch cluster won’t work at all.

2. The Upgrade

The recommended steps for the Upgrade stage are:

  1. Create a backup of all indices in the Elasticsearch cluster.
  2. Stop all the Machine Learning jobs, if any.
  3. Upgrade all the ELK stack components:
    1. Elasticsearch Hadoop
    2. Elasticsearch DB cluster by rolling upgrade all the nodes in the Cluster
    3. Kibana
    4. Logstash
    5. Beats
    6. APM Server
  4. Upgrade the drivers used by each application using Elasticsearch, or any of the related ELK components.

Needless to say, this upgrade process should be tested in non-productive environments first with the applications running in background and cover any defined functional and performance tests.

Maintenance Policy in Elasticsearch

Lets remind ourselves of the two key points we have covered so far:

  • Elasticsearch has a high release frequency with 16 releases last year, six of which were minor releases.
  • Every minor release has some breaking changes, which means some Elasticseach functionalities change and/or stop working.

Elastic believe that not all users can keep up with their upgrade pace. This is the reason why Elastic keeps the updates of the latest minor release in the previous major version. For example:

  • Current Major Version: 7.X. Current maintained stream is 7.12.X only. Note that 7.11.0 was released 2 months earlier.
  • Previous Major Version: 6.X. Maintained stream is 6.8.X. Note this stream does not change since version 7 was released, on April 2019.

A trusted source told me that Elastic will keep the current version till 7.15 or 7.16, so version 8 could be released around the end of this year, or the beginning of 2022…

Wrapping Up!

I hope that these explanations can help you form your own conclusions about which major Elasticsearch version you want to run, or about its upgrade process as a whole. Clearly, many users will need to consider these Upgrade notes when version 8 is released.

See you soon for my next article about databases!

NB: This is a translated and updated post from another one on my personal blog in Spanish, bonisql.

References:

Share this article

Contact Us

If you have any questions, just fill in the form and we’ll get back to you.

Controller: The Workshop Technologies Ltd. Purpose: To provide the services offered through the website or to handle other types of relationships that may arise with The Workshop Technologies Ltd as a result of the requests, procedures, or formalities performed by the user through the website. Legal basis: Consent of the data subject as provided in Regulation (EU) 2016/679 and the LOPDGDD 3/2018. Recipients: Internal automated file of The Workshop Technologies Ltd and third parties for the development, maintenance, and control of the legal relationship established when there is legal authorization by the user to do so. Rights: Access, rectification, transfer, opposition, and deletion. Additional information: You can obtain all the additional and detailed information you need about the processing and protection of your personal data at the link Privacy Policy.

We Come From

Like what you see?

Join your comrades or add a new flag!

Thanks!

Your message was sent. We will get back to you shortly.