The Past and Future of In-Memory Computing

(Connect world/Shutterstock)

When Nikita Ivanov co-founded GridGain Systems back in 2005, he envisioned in-memory computing going mainstream and becoming a massive category unto itself within a few years. That obviously didn’t pan out, but on the eve of the In-Memory Computing Summit 2020 taking place later this week, the GridGain CTO is still bullish on the future in-memory computing, particularly for powering stream processing.

“When I started this journey close to 20 year ago, there was a general belief that in-memory computing will be a massive category, as has been with cloud compute,” Ivanov says. “And it didn’t turn out this way. In memory computing hasn’t become a massive category. It’s still an important category, but it’s a little bit different.”

Instead, in-memory computing “kind of morphed into something else,” Ivanov says. In particular, in-memory computing technologies–specifically in-memory data grids (IMDGs) that GridGain Systems and other vendors develop–have become a core element underlying large stream processing setups. IMDGs and stream computing are not the same thing, obviously, but the share similarities.

“I can see the parallels,” Ivanov tells Datanami. “If you look at in-memory computing 10 years ago, that was this ungodly spaghetti mess of approaches and frameworks and project. Nomenclature was very hard to comprehend…Today it’s a little bit more organized. And I believe that’s exactly what’s going to happen with streaming, to a certain degree. Right now, it’s an ungodly mess for sure. You ask any architects to give you standardized streaming setup, and they’ll struggle.”

“So I believe streaming probably will not only get a little less complicated, but it’s also going to morph into something in the next concept, the next philosophical view of data processing,” he continues.

GridGain has found a certain degree of success with Apache Ignite, which is an open source platform for storing and computing on large volumes of data across a cluster of nodes. Ignite essentially is the free and open version of GridGain’s enterprise-level in-memory computing platform, which it donated to the Apache Software Foundation back in 2014.

Today, Ignite is the fifth most popular open source project at the ASF, Ivanov says, behind Spark, Kafka, Hadoop, and Cassandra. The project has matured to the point where it no longer has to prove itself, and is widely considered as a core technological building block when developers absolutely need the fastest transactional performance.

Another IMDG vendor that will be showcasing its technology at the first virtual In-Memory Computing Summit this week is ScaleOut Software. According to ScaleOut’s CEO William Bain, who will be delivering a keynote at the summit on Wednesday, IMDGs are flourishing in massive real-world use cases, such as tracking data for a million e-commerce shoppers or a fleet of rental cars.

Thanks to their ability to store fast-changing data in memory, IMDGs will be essential for acting upon live streams of data when the latencies involved with data lakes and other big data systems are too great, Bain says.

In-memory technology is an essential element for stream processing, including transactions and analytics (spainter_vfx/Shutterstock)

“With the explosion in the adoption of IoT (which is soon to be catalyzed by 5G wireless networking), countless data sources in our daily life now generate continuous streams of data that need to be mined to save lives, improve efficiency, avoid problems and enhance experiences,” Bain says in an email to Datanami. “Now we can track vehicles in real-time to keep drivers safe, ensure the safe and rapid delivery of needed goods, and avoid unexpected mechanical failures. Health-tracking devices can generate telemetry that enables diagnostic algorithms to spot emerging issues, such as heart irregularities, before it becomes urgent. Web sites can track e-commerce shoppers to assist them in finding the best products that meet their needs.”

IMDGs aren’t ideal for all streaming or IoT use cases. But when the use case is critical and time is of the essence, IMDGs will be have a role in orchestrating the data and providing fast response times.

“The combination of memory-based storage, transparent scalability, high availability, and integrated computing offered by IMDGs ensures the most effective use of computing resources and leads to the fastest possible responses,” Bain writes. “Powerful but simple APIs enable application developers to maintain a simplified view of their data and quickly analyze it without bottlenecks. IMDGs offer the combination of power and ease of use that applications managing live data need more than ever before.”

GridGain Systems CTO and co-founder Nikita Ivanov

GridGain is organizing the virtual In-Memory Computing Summit 2020, which is free to attend. In addition to ScaleOut Software, representatives from Dell, IBM, Oracle, MemVerge, and M&T Bank will be speaking at the event.

In the early days of in-memory computing, it was almost exclusively relegated to financial services firms. Companies in that industry make up the heart of GridGain’s customer base, but it’s also widely adopted in other industries, including telecommunications and retail.

“When I started this company 15 year ago, we were struggling to prove ourselves to every customer,” Ivanov says. “We were dreaming about a time when we could walk into a customer and not do anything, when a customer would know us by name and trust us. We are just now entering this place where we’re realizing, we don’t have to do all these POCs. GridGain and Apache Ignite have a reputation of being mature. It just tells me that the customer base is widening. That naturally means the market is growing, and adoption of this technology is growing.”

In-Memory Computing Summit 2020 takes place October 28 and 29. For a list of keynote speakers, click here.

Related Items:

Filling Persistent Gaps in the ‘Big Memory’ Era

Stream Processing Is a Great Addition to Data Grid, Hazelcast Finds

GridGain Expands Data Persistence for In-Memory DB