Advanced Computing in the Age of AI | Friday, March 29, 2024

The Open Source World of Today: How We Got Here 

Is the closed-source system on its last legs? Maybe not just yet, but we’re pretty close to it. Consider the phenomenal growth in the last five years of new open source technologies and processes, such as containers, Hadoop and databases like MongoDB, ElasticSearch and Redis. An entirely new set of architectures, most of them open source, have sprung up, spawning fresh business models and robust ecosystems.

The open source stack and all that it engenders is driving the closed source, proprietary stack toward irrelevance and economic infeasibility. Take, for example, the great success of Docker with containers and its resulting ecosystem. Its popularity is largely a result of its open source model that reflects the ascendance of software engineers in the creation and deployment of software. And Docker is giving closed source VMware a headache as a result.

If Docker had instead opted to emerge from stealth with a proprietary version, it’s likely we wouldn’t have seen such prolific adoption of containers. In fact, as many of us know, we’ve seen enterprises broadly moving toward open infrastructure stacks. Starting with Linux in the late 1990s, and continuing now with Hadoop and others, the industry has undergone a paradigm shift, with developers increasingly running the show. Just recently Microsoft, historically the most anti-open company imaginable, revealed that its flagship database, Microsoft SQL Server, would be available to run on Linux for the first time, showing the diminishing role of closed source operating systems in the modern IT stack. How did we get here?

Twenty-five years ago, because virtually all systems were proprietary, software developers had to wait until IT operations approved their projects. ITOps ensured their stacks were robust and everything was locked down before granting permission. Today, however, with the rapid proliferation of open source technologies, developers aren’t waiting for ITOps’ go-ahead because open source platforms have becoming increasingly more scalable, robust and secure. It’s this feedback loop between the developer community wanting to build a vast range of apps for today’s connected world and the work on open source platforms that has allowed this approach to flourish, leaving closed systems increasingly challenged to keep up.

Two examples of this dynamic can be seen in the Hadoop space, with Hortonworks and Cloudera. Hortonworks has adopted an entirely open source model. The kernel that Hortonworks uses and everything it builds on top of it are open source. So, how does the company make money? By monetizing services and support — charging its customers to deploy and manage their Hadoop implementations. Cloudera, on the other hand, has embraced a hybrid model. It uses the open source Hadoop kernel yet builds and sells proprietary capabilities and functions on top of it, ensuring that its software is enterprise- and production-ready. Two different business models, but all based on open source technologies.

These examples raise the perennial question of making money in an increasingly open source world. As companies try to overcome “The Software Paradox” — the idea that as software becomes more important to running modern business software company profits decrease — we’re seeing new business models emerge. Time will tell which ones prove to be most profitable.

What is clear is that the entire IT stack is now becoming open. Thirty-five years ago, companies like IBM built and sold their own closed hardware and proprietary software running on top. Then hardware and software began to separate as Intel’s x86 microarchitecture took hold in the server market amid the transition to a client-server model. This gave birth to proprietary software stack companies such as Oracle, SAP and others. Obviously, those two companies aren’t going out of business, and there is still a need — albeit smaller — for closed-source systems.

There are, too, concerns around open source security, scalability and robustness. In the past, organizations bought a closed hardware/software stack because it was hardened and guaranteed by the vendor. How are these concerns mitigated today? Of course, we’ve already seen this question answered. Cloudera, for example, assures customers who buy its open source products that the code, value-added capabilities and services are hardened for enterprise use.

To be sure, there are still some companies that can be successful in today’s open world with a closed model. Amazon, for example, has built an amazing business by creating an entire infrastructure stack on a proprietary system that paradoxically enables open source development and adoption of other open source technology. Even so, I believe the number of these types of companies will become smaller and smaller over time.

For the first time in our information technology age, we can now build an entire infrastructure stack composed of x86 architecture, commodity components and an open source stack. The fastest growth, as we know, is happening among open source companies. Developers today play a far more influential role in application development as the monolithic architectures break down. Demand for microservices, developer-centric workflows, containers, open source, big data are all part of the larger current driving information technology today.

This is the open source direction of the software industry today that has already crossed the tipping point. Long live open source.

Mohit Bhatnagar is vice president of product at ClusterHQ.

EnterpriseAI