Data Management Predictions for 2024

(Yurchanka Siarhei/Shutterstock)

In the world of data storage and unstructured data management, much has changed in the past 12 months. Cloud storage strategies are on the radar with rising costs and increased pressure on IT budgets during volatile economic times, generative AI is creating new data storage and governance requirements, data migrations are increasingly complex but necessary in an era of data center consolidation and IT organizations are under intense pressure to contain costs and deliver greater data value.  What to do about all this?  We have some predictions below for IT organizations and data storage teams, beginning with AI and unstructured data management.

AI Will Enrich Unstructured Data for Better Outcomes

Unstructured data is vast and to this day, has not been usable for a few reasons: it is difficult and costly to search, classify, segment and move to AI engines and analytics tools. As AI tools and services have evolved to become more affordable and consumable by many, not just the largest organizations with deep pockets, there is increased demand to harness this data for new business value.

(Paper piper/Shutterstock)

Here is the challenge: Researchers and data scientists that want to send data to AI don’t have easy ways to do that securely. It requires writing manual scripts and days or weeks of work. Separately, AI and ML technologies are still too inaccurate and introduce bias and false outcomes.

However, we predict an increased demand for solutions that create a workflow where AI can quickly find the desired data, enrich it and verify outcomes. The workflow could consist of first using an AI tool that scans data in a cloud data lake or data center to find the desired data types for a project, such as all mammography images from 2022. The AI then enriches the metadata by scanning file contents and tagging files (such as “containing marker X for diagnostic follow up”) and returns a data set that can be verified by a human as the correct output. An unstructured data management with a searchable global file index which can connect by API to AI tools to further identify and enrich data is invaluable: it saves time, creates efficiencies and improved accuracies for AI projects.

From Cloud First to Data First

Cloud-first strategies were all the rage during the height of the global pandemic but today, those plans have come under the microscope.  IT organizations have created flexible, hybrid cloud and multi-cloud environments using multiple vendor technologies suited to different workloads.  Some organizations have been burned in the cloud as they discovered that they were not only not saving enough but sometimes even spending more versus keeping their data in-house.

There are many reasons behind this reality but the idea that having most or all of your workloads in the cloud for ultimate cost savings has not panned out. IT organizations will choose from the many storage options on the market—whether on-premises or in the cloud—based on the performance, cost and security needs of their data through its lifecycle. The ability to easily move data as requirements change or better technologies become available is paramount.

(Rawpixel.com/Shutterstock)

Therefore, data management tools which allow massive amounts of unstructured data to move without vendor lock-in will be increasingly valuable.

Unstructured Data Migrations Get More Intelligent, Automated

Enterprise data migrations have traditionally been complex, hands-on, and require ample professional services, especially when it comes to the massive volumes of unstructured data involved. Automation and AI will change this, enabling intelligent, efficient data migrations that no longer need IT managers to babysit them and they will also be adaptive.

These tools will know how to solve problems on the fly and self-remediate. As their knowledge grows, advanced migration planning tools will recommend optimal storage tiers for different workloads and use cases. This is a timely development, as data migrations are dependent upon the customer’s changing environment: their firewall, their network connections and security configurations. Enterprise customers will look for solutions that provide an order of magnitude faster migrations with better long-term outcomes and fewer instances of data loss, errors and security risks.

Storage IT Careers: FinOps and Cross-Silo Skills Required

With all the trends above, storage IT teams will need to obtain additional expertise to be more cost-effective, efficient and align with business and departmental needs. The term FinOps will be part of the storage architect’s nomenclature in 2024. As storage becomes more software and services-centric, managing hardware is now less of a requirement. Instead, managing vendors, contracts and delivering secure, cost-efficient data services to departments and users will take up the bulk of storage professionals’ time.

As well, enterprises are moving away from being single vendor shops. Therefore, storage administrators must hop between different technologies rather than specialize in one platform. This requires broader skills and knowledge in networking, security, cloud architecture, cost modeling, and data analytics.

To boot, storage-specific job titles will be replaced by data titles, such as “data insights engineer” or “data management architect.” In mature infrastructure teams, managers responsible for storage will work more closely with data science and AI teams to procure AI-ready infrastructure and devise plans for data classification and data workflows to analytics platforms.

About the author: Kumar Goswami is the CEO and Co-founder of Komprise.

Related Items:

2024 GenAI Predictions: Part One

Fourteen Big Data Predictions for 2024

What Will 2024 Bring to Advance Analytics?