Why storage is the last technology to be disrupted by software-defined

By Avinash Lakshman | | Big Data & IoT

I was in London a few weeks ago meeting with customers and it dawned on me: enterprises are ready for a modern approach to infrastructure, and storage in particular. We’ve seen this for years in the web space and we’ve been talking about it for years in the enterprise space, but not much has actually changed. It’s behavioral and IT must want to change. The customers I met had enthusiasm that went beyond mere willingness for a new approach. They explicitly said they want to disrupt the way they do storage – completely. This was music to my ears.

When we launched Hedvig last March, one of the things I talked about that resonated most with customers, analysts and the media was my belief that there hadn’t been any fundamental innovation in enterprise storage in the last 10 years. And by fundamental innovation, I mean an entirely new way of solving a problem, an approach that reflects a wholesale shift in perspective. In fact, I would argue that NetApp was the last disruptor in storage when they consolidated a number of storage protocols and brought to market a solution that changed the way enterprises could solve storage challenges. To be sure, there have been sizable incremental innovations in our industry such as all-flash arrays, cloud storage gateways, and object storage software. But none of these have been a monumental step forward.

Our industry had become so used to “how things were always done.” When we began detailing the specifics of the Hedvig Distributed Storage Platform to customers we were told more than once, “There’s just no way you can do all that.” Well, it turns out we can. While other major areas of the IT stack (compute and networking come readily to mind) have already embraced a more modern architecture, why should it be that enterprise storage is the last “legacy” part of the stack to be modernized? After all, it’s not as though there’s a dearth of really smart people in Silicon Valley or in other tech centers in the world.

The answer is twofold.


Storage is arguably the hardest technology to get right because it’s so “easy” to get wrong. It’s easy to miss a write, read the wrong bit, corrupt a file, or simply perform too slowly. If a packet is lost, it can be resent. If an application doesn’t work, it can be restarted. In both cases this works because there is an assumption that the underlying data – the source of truth – can be reaccessed. Storage is that source of truth. You have to architect redundancy to survive disk failures and you often need multiple copies to restore from more catastrophic failures.

As a result, the industry has labored under a drawn-out evolution in storage because the preeminent concern is, “how do I protect the data.” Large legacy storage vendors have iterated around this concern for decades. It’s the Innovator’s Dilemma. If legacy storage vendors make even a small, incremental change, it could impact thousands of installations at thousands of customers. As a result, it’s nearly impossible for them to incorporate hardware innovation without disrupting build cycles (which takes years) and customer installations. Consider IBM’s recent claim of a breakthrough in phase-change memory that could replace DRAM. This multilevel PCM innovation complements NVMe, 3D NAND and many other rapidly evolving technologies that organizations want to incorporate. But traditional storage products won’t do so for at 12 to 18 months, maybe longer.


Data has become the most important business asset, after an organization’s people. Companies today seek to develop entire new revenue streams and business models by extracting insights from the petabytes of information they already store. As such, for a vendor to come calling and offer a disruptive way to store data demands a large leap of faith indeed.

Take Tesla as an example of innovation anxiety. I argue Tesla is a fundamental disruption to the automobile industry. The company decided that incremental innovations like turbo diesels or biofuel engines weren’t good enough. The company instead chose electricity and designed fundamentally different motors and batteries accordingly (it also doesn’t use car dealerships). Yet think about how “range anxiety” has entered our lexicon. Given how critical the automobile is to our daily lives, then this is but common consumer resistance to innovation and change.

Now let’s turn this analogy back to storage. Storage is a critical component of our daily “business lives.” It’s the most stateful part of your IT architecture and responsible for maintaining consistency. You can change virtually every other part – compute, networking and others – but you will always know that the state, or consistency, of your application is maintained. Asking companies to adopt a new storage approach can be as disruptive as asking them to adopt electric cars. However, “data anxiety” is no more a challenge than range anxiety if embraced.


As I alluded to earlier, this is not to say incremental improvements in storage are a bad thing and cannot yield substantial benefits. Hedvig has and continues to benefit from many of these innovations. Take, for example, what’s happening with all-flash arrays pioneered by Pure Storage; hyperconverged solutions pioneered by Nutanix; and, going further back, deduplication pioneered by Data Domain. While these incremental improvements produced great dividends, they weren’t, in my view, fundamental innovations.

So why now? What has changed to make storage ripe for disruption by a fundamental innovation?

I see three things:

  • Distributed systems have matured operationally. Distributed systems as a concept has been validated by companies in industries across the globe. The IT industry has had more than a decade of operational experience with distributed computing and architecture inside the largest and most successful companies in the world, such as Amazon, Facebook and Google.
  • Big data, IoT, and BI tools add to exponential data growth. More sophisticated software platforms like Hadoop and Spark enable organizations to extract more value from their data than ever before. This is compounded by the rise of sensors, smart infrastructure, and other Internet of things (IoT) data sources. All of this prompts enterprises to store even more, forcing a new approach.
  • A true “cloud like” mentality finally pervades the enterprise. Companies are ready to embrace a more cloud-like architecture, which I define as automation and self-provisioning on top of a fully virtualized infrastructure, be it hypervisor- or container-based. Moreover, these companies are also comfortable buying commodity hardware. These demand-side dynamics are further fueled by supply-side dynamics, such as EMC buying Dell, which forces companies to rethink infrastructure refreshes.

Suffice to say that since we launched Hedvig, many of the concerns and fears we heard – that enterprises would remain saddled with the solutions they’ve had for 15 years and reluctant to adopt an entirely new architecture – were unfounded. Storage is now a distributed systems problem, and the “old way” is freighted with exponential data growth and untenable economics. Organizations need a distributed systems approach to solve their data challenges. I, for one, foresee a good deal more fundamental innovation around the corner. This is what we believe we’re doing at Hedvig – applying true distributed systems thinking and architecture to storage. We’re harnessing these aggregate parts to create a platform that solves today’s storage problems from a totally different angle. And we’re gratified that more and more customers are finding the value in our approach.

If you’d like to learn more, check out our recent Storage Field Day presentations. I elaborate on this concept of incremental versus fundamental innovation, and my team provides interesting demos and whiteboards on our how our distributed systems approach is disruptive.

Watch Videos