Why cold storage and the new data era go hand in hand
Author: Beatrice
By Timothy Sherbak, Enterprise Products and Solutions Marketing at Quantum.
The phrase ‘digital transformation’ has been bandied around a lot over recent years, as many companies adopt technology into their processes to meet changing business and market requirements. The shift in the way companies operate means that the scale of data being generated has dramatically increased. With the amount of unstructured data growing at a rate of up to 60% per year and expected to make up 80 to 90% of all data by 2025 – the are no signs of things slowing down.
To put this into perspective, every year, a semiconductor manufacturer produces over a billion image scans documenting the manufacturing process of 4000 wafers every week, which results in petabytes of data that must be stored for several years. Another example is autonomous cars, an autonomous car produces up to 2TBs of data every hour of use that must be maintained for several years for potential safety analysis or remodelling.
As data growth increases, companies’ storage budgets and data storage capacities are struggling to keep up. The issue of storing unstructured and inactive data needs to be addressed – this is where cold storage comes in.
What is cold storage all about?
Like every great story, the lifecycle can also be divided up into three parts: the beginning, the middle and the end. The importance of data to a project and how often it needs to be accessed dictates how data can be classified – ‘hot’, ‘warm’ or ‘cold’ data. Data that is ‘on demand’ and being used every day for a project, is referred to as ‘hot’ data. Data that needs to be accessed on a regular basis but not daily is called ‘warm’ data. Then, last but by no means least, there is ‘cold’ data or ‘dormant’ data that is no longer or very rarely needed by organisations for projects they are working on. This data makes up over 60% of all stored data.
Many companies have had to preserve cold data, not out of choice but of necessity to comply with strict regulations or internal policies around storing data. However, attitudes towards cold data are starting to change. More companies are deciding to hold on to cold data because of the long-term strategic value it holds, the future potential of the data to be re-used for projects, as well as the chance to enrich the data over time to augment its value.
Traditionally, cold data has been written to tape media, moved offline, and transferred to a storage facility where it can be accessed for future use. Cold data storage is a highly durable, secure, and inexpensive way to store data in the long term. The emergence of new data analysis methods, increase in computing power, and the rapid acceleration of digital transformation, means there is a greater demand for data to be stored and readily available online. Despite this, there are still concerns around cold storage’s durability, level of security, and storage costs that need to be resolved.
Is a cold storage strategy really necessary?
It is predicted that data created during the next three years will amount to more than all of the data created during the past 30 years. As organisations are realising the value of cold data, combined with the surge in the volume of data being generated, the case for cold storage is clear – if companies want to stay ahead of the data curve, they must put act quickly and implement a cold storage strategy. Companies must also prioritise building meaningful and scalable solutions to harvest data-driven innovations and opportunities to reduce the common problems associated with massive data accumulation and continued data growth.
What are the perks of cold storage?
Recent innovations in cold data storage, have improved performance, data durability, and storage efficiency, raising the standard overall expectations. Storing vast amounts of data can be costly for organisations, however, cold data storage is considerably cheaper than the NVMe and solid-state disk technologies deployed for high performance access to hot data. One of the key reasons for this is because cold data can be stored on lower performing and cheaper storage infrastructures, either in-house or in the cloud. This means organisations can cost-effectively store more of their growing data sets.
Most modern cold storage archives have been developed by some of the world’s largest cloud solution providers, but with emerging architectures and services, cold storage solutions are now deployable within an organisation’s data centre, colocation facility or hosted IT environment that meets data sovereignty and data residency requirements. As a result, the data is more easily accessible with no access fees and retrieval times are reduced from days to minutes. Additionally, new erasure coding algorithms are now optimised specifically for cold storage which reduces the storage overhead compared to the traditional method of storing multiple copies.
Unlocking the value of data
Cold storage enables organisations to safely store data and maintain in-house control of these invaluable assets and easily retrieve them for future projects. It increasingly clear in this new data era that organisations cannot just disregard data if it doesn’t serve any immediate purpose. Certain data must be stored, both for its historic relevance and value for future projects. Advances in cold storage, mean companies don’t have to sacrifice inactive data for active data. Whether an organisation is looking to reduce storage costs or boost the value of its data, cold storage provides a real solution to the problem of data accumulation.
