Pedro is a Database Architect at The Workshop. As the Founder and current leader of the User Group Malaga Databases, Pedro loves learning about technology and exploring his passion for databases. In this article, he provides some pointers on how to manage Elasticsearch upgrades.
If you are working with Elasticsearch, you have probably realised that the company publishes new versions of its software very frequently. To give you more context, let’s compare it to what other database technologies have published in the last year:
Elasticsearch shines with 16 releases — that’s more than one a month! Most other companies release one per quarter on average.
Versions in Elasticsearch
Let’s see an example of how versions, or releases, are defined in Elasticsearch:
7.12.1
- 7 is called a major release
- 12 is called a minor release
- 1 is called a maintenance release
You may have experienced a minor release create some incompatibility issues. These issues may happen between components inside the Elastic stack, or ELK stack, or may occur with the application using Elasticsearch. It means that something that worked before has stopped working after applying a minor release. The most common reason is a breaking change.
Generally with other database technologies, broken compatibility changes happen with major releases. So we could consider a minor release from Elasticsearch as similar to a major release from other databases systems.
Elastic compensates for these incompatibility issues with its Maintenance Policy and a lot of documentation offering supporting during the upgrade process. The high-level steps to follow whenever you are thinking of taking upgrading to a new release are:
- The Preparation
- The Upgrade
1. The Preparation
Follow these steps at the Preparation stage of an Elasticsearch upgrade:
- Check the deprecation log from the Elasticsearch cluster nodes
- This can be provided by a DBA, but ideally this information will be available to the Engineering teams.
- Review the breaking changes
- Check the breaking changes on every minor release between the current version and target one.
- Review Plugins
- Plugins are quite common, eg: Monitoring.
- Check the code
- In the applications or services interacting with Elasticsearch, check the code and make the necessary changes.
In some cases, you might be required to do some re-indexing whenever there are indices created in previous versions. The Elasticsearch engine does accept indices from the previous major version, but it does not from older ones. For example:
- Indices created in the 6.X version are compatible in 7.X, but 5.X indices are not.
- Indices created in the 7.X version will be compatible in 8.X, but 6.X indices won’t work.
Indeed, if the re-indexing is not done before the upgrade, the whole Elasticsearch cluster won’t work at all.
2. The Upgrade
The recommended steps for the Upgrade stage are:
- Create a backup of all indices in the Elasticsearch cluster.
- Stop all the Machine Learning jobs, if any.
- Upgrade all the ELK stack components:
- Elasticsearch Hadoop
- Elasticsearch DB cluster by rolling upgrade all the nodes in the Cluster
- Kibana
- Logstash
- Beats
- APM Server
- Upgrade the drivers used by each application using Elasticsearch, or any of the related ELK components.
Needless to say, this upgrade process should be tested in non-productive environments first with the applications running in background and cover any defined functional and performance tests.
Maintenance Policy in Elasticsearch
Lets remind ourselves of the two key points we have covered so far:
- Elasticsearch has a high release frequency with 16 releases last year, six of which were minor releases.
- Every minor release has some breaking changes, which means some Elasticseach functionalities change and/or stop working.
Elastic believe that not all users can keep up with their upgrade pace. This is the reason why Elastic keeps the updates of the latest minor release in the previous major version. For example:
- Current Major Version: 7.X. Current maintained stream is 7.12.X only. Note that 7.11.0 was released 2 months earlier.
- Previous Major Version: 6.X. Maintained stream is 6.8.X. Note this stream does not change since version 7 was released, on April 2019.
A trusted source told me that Elastic will keep the current version till 7.15 or 7.16, so version 8 could be released around the end of this year, or the beginning of 2022…
Wrapping Up!
I hope that these explanations can help you form your own conclusions about which major Elasticsearch version you want to run, or about its upgrade process as a whole. Clearly, many users will need to consider these Upgrade notes when version 8 is released.
See you soon for my next article about databases!
NB: This is a translated and updated post from another one on my personal blog in Spanish, bonisql.
References:
