Evolution of Storage: A Technical Perspective
When it comes to storage, the problems faced by consumers and enterprises are not all that different. As an iPhone user, my constant worry is that I’ll run out of storage on my phone while recording a very important (soon to be viral) video.
Gone are the days when we stressed over floppy/CD/HDD corruption and data loss. Public cloud storage lets consumers take data availability for granted; I am not afraid of giving my iPhone to my 1-year-old nephew anymore, because even if the commodity hardware suffers damage the valuable photos, playlists, and critical emails are safe in the cloud. Cloud storage does not mean a new type of storage hardware: it is simply an intelligent software managing your data on massively large numbers of drives.
Enterprises have the same set of problems, albeit at a much larger scale. Data availability, security, and scalability are the biggest concerns. Enterprise storage has come a long way in the last 60 years, but data explosion, due to IoT, AI, and social media has completely transformed storage requirements. It will soon be cliché to mention that “Data is King”. It’s not about data storage anymore, it’s a race to extract maximum intelligence out of the data, evolve applications, and gain a competitive advantage. Data storage solutions are already expanding toward data management and analysis. Like dinosaurs, traditional storage solutions are too rigid to keep pace with modern demands and will soon be extinct. Let’s explore how storage has evolved over past few decades.
The first production IBM hard disk drive, the IBM 350 disk storage unit, shipped in 1957 as a component of the IBM 305 RAMAC system. It was approximately the size of two medium-sized refrigerators and stored five million six-bit characters (3.75 megabytes) on a stack of 50 disks. Even though Direct Attached Storage (DAS) drives have improved to terabytes of storage and are now just a couple of inches in form-factor, they are not suitable for modern applications, which assume five or six nines availability. With DAS, your applications get handcuffed to local monolithic storage, and scaling DAS for large capacity requirements is simply not possible.
NAS / SAN
Storage Area Network (SAN) and Network Attached Storage (NAS) solutions connected storage arrays over the network and resolved many issues associated with DAS. These solutions allowed storage admins to eliminate the web of wires needed to support DAS solutions. Storage arrays could now be shared among multiple application servers, and the storage layer could be scaled independently of compute. Both NAS and SAN solutions provided different ways to consume storage and were suitable for applications with specific requirements, such as file access and block access, respectively. The delight of storage admins was short-lived though. Both of these solutions involved large CapEx investments and also had limitations when it came to scaling beyond a certain capacity, and stretching installations across distant locations was generally not supported.
Hyper Converged Infrastructure solutions came into existence with the goal of making the life of infrastructure administrators simpler. These solutions combined the best parts of DAS and NAS/SAN solutions by consolidating compute and storage layers in the same server so that resources could be shared. Also, multiple servers were allowed to connect over the network so that they could be part of the same cluster. You did not have to worry about managing different servers for compute and storage anymore — you could scale capacity by adding more servers, and reduce management and support headaches by eliminating multiple vendors. But, HCI was still very expensive and did not solve the problem of data movement across distant locations.
Cloud or nothing
Whether it’s consumer or enterprise storage, cloud is the new norm: high availability, on-demand scalability and cost-effectiveness are the key features required from any storage provider. For enterprise storage, gone are the days of purpose-built storage solutions designed as short-term solutions, leading to the rip-and-replace of an entire infrastructure every 3 to 4 years. Modern applications are now being designed with cloud-native and cloud-first models, which demands that enterprise storage blur the lines between private cloud and public cloud infrastructure.
Hedvig was founded with one goal in mind: to make storage cloud-easy. Bring the capabilities that consumer storage already enjoys to the enterprise. Ingest data from anywhere and access it from anywhere: this should be the new standard, without stressing over security, 24×7 availability, and cost. As Avinash Lakshman, founder of Hedvig once said, “Product should be designed and capable of being deployed in an evolutionary and revolutionary fashion”.