Storage and Networking Bytes: Elasticsearch Logging Goes Kubernetes

LogDNA, which offers an ultra-scalable distribution of the open source Elasticserach log management system that is based on Kubernetes, this month unveiled version 2 of its LogDNA Agent.

According to product manager Andrew Konoff, the new V2 agent was built to handle file logging “in even the most intensive Kubernetes deployments.” The company chose to develop the new agent in the low-level language Rust, which it says makes it “far more reliable” than the V1 agent. It also consumes fewer resources and has the ability to track “even the most minor, rapid changes to file logs,” Konoff writes in the LogDNA blog.

LogDNA provides a centralized repository for ingesting, storing, and analyzing logs gathered from applications developed with a variety of language (Ruby, Java, PHP, etc.), running on a variety of cloud and on-prem sources (Linux, MacOS, and Windows, etc.). It supports multiple ingestion methods for logs, including syslog, code libraries, AWS, Google Cloud, and an ingestion REST API, in addition to the aforementioned V2 agent, which installs on Kubernetes systems.

Once the log data is brought in, LogDNA provides tools to enable administrators to view their log data, as well as tools to develop alerts, boards, and graphs. The San Francisco company, which was founded in 2016 by Chris Nguyen and Lee Liu , has raised $33.4 million to date, and touts customers like Asics, Lyme, Virgin, IBM Cloud, and Mantl.

The company’s roots lie within Elasticsearch, a log management system built on Lucene. As the company explains in a white paper, “while an Elasticsearch cluster can scale to handle truly impressive amounts of data, it eventually reaches a saturation point.”

For LogDNA, this saturation point occurs when a cluster reaches between 30,000 and 50,000 shards, the company says. “Large clusters experience more operational overhead due to having to constantly synchronize node state, as well as having to manage the sheer number of indices and shards across each node,” it states. By housing Elasticsearch within containers managed by Kubernetes, the company is able to scale around that limitation while still maintaining a high level of automation.

Predictable Data Egress

Cloud storage is a growing business, thanks to cheap and abundant storage capacity. But one aspect of cloud storage that customers don’t like is data egress costs, or the fee that cloud platforms charge customers for moving data off the cloud.

LucidLink’s Filespaces network device makes data on the cloud appear to be stored locally

One vendor that’s come up with a solution to that dilemma is LucidLink, the San Francisco-based startup that develops a distributed file system called Filespaces that makes S3-compatible object stores look and function like a fast local file system.

Last week, the company unveiled LucidLink Performance, a new solution that combines the Filespaces gateway together, 1TB of Amazon S3 storage, and 1TB of data egress outside of S3 for $85 per month. The company says that, by giving customers better prediction of data egress fees, it makes cloud services more accessible and predictable.

“We recognized that Amazon S3’s massive scale is well suited to absorb usage spikes and ensure a consistent user experience,” LucidLink CEO and co-founder Peter Thompson says in a press release. “However, we also heard a great deal of frustration from our customers around the unpredictable nature of egress fees. By evaluating the usage patterns of our user base, we were able to come up with a bundle that will satisfy most customer requirements, assisting in cost predictability.”

NMVe Over Fabric

Infinidat this week rolled out enhancements to its line of all-flash storage arrays, including a new NVMe-over-fabric option, three-site replication, elastic pricing, and a 100% data availability guarantee.

The Waltham, Massachusetts-based company sells a range of high-end storage arrays that can store up to 8PB of data within a single 42-inch rack. Infinidat supports block and file storage over Fibre Channel and standard Ethernet networks, and with the version 5.6 update to the InfiniBox software, Infinidat has introduced NVMe-over-fabric option through its support for support for the NVMe protocol over TCP.

Infinidat storage solutions can hold 8PB of data in a full 42-inch rack

The company says that it selected NVMe/TCP as its first supported NVMe protocol to enable customers to leverage their existing Ethernet investments and because NVMe/TCP “is compatible with visions of a software-defined data center.” It has plans to support NVMe over Fibre Channel and RoCE as customer demands dictate.

The software update also brings support for three-site replication, which will allow an active-active replica to be simultaneously replicated asynchronously to a third InfiniBox system, thereby providing protection from data loss in the event of a disaster.

Infinidat also rolled out its new Elastic Pricing model. Under this new model, customers purchase Base Capacity and rent Burst Capacity, thereby eliminating the need for customers to choose between low cost and flexibility.

The company has also rolled out its Data Availability Guarantee, whereby the company pledges that “customers will never experience downtime caused by Infinidat.” Should a covered customer ever suffer an unexpected downtime event, the company promises to provide the appropriate remedy depending on the business model in place, and do root-cause analysis to ensure that the downtime event will be a one-off event, the company says.

Related Items:

Storage and Networking Bytes: 3D XPoint Lawsuit

Blurred Storage Lines: Clouds That Appear Like On-Prem

Storage and Networking Bytes — January 17, 2020