War Unfolding for Control of Elasticsearch

Amazon Web Services on Monday announced that it’s partnering with Netflix and Expedia to champion a new Open Distro for Elasticsearch due to concerns of proprietary code being mixed into the open source Elasticsearch project. Elastic, the company behind Elasticsearch, responded by accusing Amazon of copying code, inserting bugs into the community code, and engaging with the company under false pretenses.

Amazon Web Services unveiled the new Open Distro for Elasticsearch yesterday. In a blog post, Adrian Cockcroft, the vice president of cloud architecture strategy for AWS, says the new project is a “value added” distribution that’s 100% open source, and that developers working on it will contribute any improvements or fixes back to the upstream Elasticsearch project.

“The new advanced features of Open Distro for Elasticsearch are all Apache 2.0 licensed,” Cockroft writes. “With the first release, our goal is to address many critical features missing from open source Elasticsearch, such as security, event monitoring and alerting, and SQL support.”

Elastic uses a three-tiered system that’s a mix of open and proprietary licenses to distribute its Elastic Stack. The core elements of the Elastic Stack, which include Elasticsearch, Kibana, Logstash and Beats, are free and distributed under a permissive Apache 2.0 license. In February 2018, Elastic made some changes to its open source policy when it elected to expose the source code for X-Pack, which is a collection of add-ons that is now known as Elastic Stack Features.

Even though Elastic Stack Features are distributed under Elastic’s proprietary license, the company says they are nonetheless open. The more basic Elastic Stack Features, such as security, monitoring, and alerting, are free to use, while more advanced Elastic Stack Features, like machine learning, graph database capabilities, and SQL functionality, must be purchased. All Elastic Stack Features, however, are developed in the open, Elastic says.

“We now have three tiers: open source and free, free but under a proprietary license, and paid under a proprietary license,” Elastic CEO Shay Banon told Datanami in an interview a couple of weeks ago. “Under these three tiers, the code itself is open and visible for introspection.”

AWS is seizing upon Elastic’s actions in creating this three-tiered system – not to mention the merger of X-Pack into Elastic Stack proper with the version 6.3 release of the Elastic Stack last summer – in justifying the creation of Open Distro for Elasticsearch.

“Unfortunately, since June 2018, we have witnessed significant intermingling of proprietary code into the code base,” Cockroft wrote. “While an Apache 2.0 licensed download is still available, there is an extreme lack of clarity as to what customers who care about open source are getting and what they can depend on.”

Cockroft says there’s no clear documentation in the Elasticsearch release notes over what’s open source and what’s proprietary. “Enterprise developers may inadvertently apply a fix or enhancement to the proprietary source code,” he wrote. “This is hard to track and govern, could lead to breach of license, and could lead to immediate termination of rights (for both proprietary free and paid).”

Banon today responded to AWS in a blog post, in which he leveled a variety of accusations at the cloud giant.

“Our products were forked, redistributed and rebundled so many times I lost count,” Banon wrote. “There was always a ‘reason’ [for the forks, redistributions, and rebundling], at times masked with fake altruism or benevolence. None of these have lasted. They were built to serve their own needs, drive confusion, and splinter the community.”

Elastic’s commercial code may have provided an “inspiration” for others to follow, Banon wrote, but that inspiration didn’t necessarily make for clean code. “It has been bluntly copied by various companies and even found its way back to certain distributions or forks, like the freshly minted Amazon one, sadly, painfully, with critical bugs,” he wrote.

According to Cockcroft, any improvements or changes that developers make in the Open Distro for Elasticsearch will be merged with the “upstream” Elasticsearch project. “… [O]ur intention is not to fork Elasticsearch,” he wrote, adding that maintaining an “internal forked version of a project causes extra wasted effort…”

AWS believes that maintainers of open source software have an obligation to keep open source and proprietary code separate. “Unfortunately, we are seeing other examples where open source maintainers are muddying the waters between the open source community and the proprietary code they create to monetize the open source,” Cockroft wrote.

AWS reached out to Elastic, but there was no deal, Cockroft wrote. “We have discussed our concerns with Elastic, the maintainers of Elasticsearch, including offering to dedicate significant resources to help support a community-driven, non-intermingled version of Elasticsearch,” he wrote. “They have made it clear that they intend to continue on their current path.”

Banon accused AWS copying code and co-opting the Elasticsearch product for its own use.

“When companies came to us, seeing our success, and asked for special working relationship in order to collaborate on code, demanding preferential treatment that would place them above our users, we told them no,” he wrote. “This happened numerous times over the years, and only recently again, this time with Amazon.”

In the interview two weeks ago, Banon told Datanami he had no plans to push back away from open source licenses, as Confluent, MongoDB, and Redis Labs have done in response to Amazon and other cloud vendors. In recent months, those three companies have restricted their open source licenses in an attempt to dissuade cloud vendors from launching their own services based on the open source versions of their products.

Elastic’s issue with AWS is a bit different. In any event, Banon says he has no regrets over allowing developers to view the source code for all of Elastic’s products (even if they’re not distributed under an open source licenses).

“We did the exchange more than half a year ago, and so far it’s going well. This is just us being more open,” he says. “There’s a reason why we’ve been successful as a company and it would be pretty foolish of us on our end to get away from it.”

The code repository for the new Open Distro for Elasticsearch can be found at opendistro.github.io/for-elasticsearch.

Related Items:

Elastic Eyes Security and APM for Expansion Effort

Elastic to Release Source Code for X-Pack