The State of Storage: Cloud, IoT, and Data Center Trends

(spainter_vfx/Shutterstock)

One of the biggest storylines in data storage this year is the explosion of the cloud and the simplicity that cloud storage can bring. But cloud storage isn’t the whole story – not by a long shot – as IoT data generation ramps up and on-prem storage technology continues its evolution.

Data storage demands continue to increase at a geometric rate, thanks largely to the global spread of the Internet and the digitization of large swaths of the lives of people, who numbered more than 7.5 billion in 2017.

Human-generated data from smart phones – think selfies shared on social media sites — account for a fair amount of the data. (Videos, of course, are the largest data.) But the real bend in the curve comes from machine-generated data from IoT devices.

That’s the conclusion of the IDC, which last year estimated there would be about 40 zettabytes (ZBs) of data in the world this year. By 2025, IDC estimates that the “global datasphere” will hit 175 ZB, with 90 ZB of that total coming from edge devices.

Much of the data created on edge devices won’t escape the memory or local storage of the device itself. Instead of writing it down permanently, about 30% of this data will be consumed (or analyzed) in real time, according to the IDC.

Faced with this onslaught of data on the edge, AI developers are looking to train machine learning models locally on devices, rather than pay the high cost of moving all that data back to the data center or the cloud.

(Source: IDC)

While lots of data will appear and disappear on the edge, that doesn’t mean the market for longer term storage will be hurting. IDC estimates the storage industry will ship 42 ZB of total capacity between 2018 and 2025, so clearly a good chunk of data is being written down.

The other big story is the rise of cloud storage, which will account for nearly half (49%) of data storage by 2025, the IDC says.

With hundreds of massive data centers spread across the planet, cloud vendors like Amazon Web Services, Microsoft Azure, Google Cloud, IBM Cloud, and Alibaba have the scale and global reach to satisfy the capacity and data locality demands of enterprises all over the world.

Cloud vendors offer a range of storage options to clients. On the hardware front, that means selecting from hard disk drives (HDDs) or solid state drives (SSDs). Tape is also used on clouds, since it provides such a cost-effective backup medium.

The more crucial selection is the storage software interface. The cloud vendors offer block, file, and object storage interfaces, providing customers with a range of options for performance, capacity, and cost.

Meanwhile, in the data center, the shift to SSDs is well underway. According to Gartner, the high-end storage array market is transitioning from first-generation NAND flash technology that travelled over established SAS and SATA disk adapters to more advanced NVMe technology that attach directly to the PCIe bus.

Gartner says solid-state arrays that use NVMe internally will represent more than 30% of revenue by 2021, up from less than 1% in 2017. Meanwhile, arrays based on another advanced technology, NVMe over Fabrics (NVMe-oF), will account for nearly 30% of the storage array market by 2021, up from about 1% in 2018.

(spacedrone808/Shutterstock)

Leaders in the solid-state array market, such as Pure Storage, NetApp, HPE, IBM, Dell EMC, Hitachi Vantara, and Kaminario, are all looking to capitalize on the adoption of NVMe technology in 2019. According to Gartner, the all flash array market has been growing around 30% annually, and there’s no sign it will let up any time soon.

While NVMe adoption in the enterprise and the cloud is currently ramping up, there are even more advanced storage technologies on the horizon. Enterprise storage administrators should keep an eye on so-called storage class memory (SCM) technologies, which are just now starting to come to market.

One of the early SCM technologies is Phase Change Memory (PCM), which offers performance somewhere between a NAND flash drive and DRAM. Intel Optane is an example of one type of PCM, which Intel dubs 3D XPoint. Micron, which worked with Intel on PCM technology, has its own PCM offering, and IBM does work in the area too.

Meanwhile, suppliers are bolstering their traditional NAND storage offerings with new manufacturing techniques that can extend the capacity of the technology, such as 3D-NAND. Some manufacturers expect to be able to layer up to 500 levels of NAND storage, which will keep NAND storage tech in play even as SCM blur the lines between storage and memory in the years to come.

Related Items:

Global DataSphere to Hit 175 Zettabytes by 2025, IDC Says

Which Type of SSD is Best: SATA, SAS, or PCIe?