Advanced Computing in the Age of AI | Friday, March 29, 2024

Apache Mesos Emerges as Datacenter OS 

The rise of Linux container technology as a means of packaging and running web applications is creating an opening for open source software needed to distribute and orchestrate app containers across datacenters. Hence, the emerging ecosystem around Apache Mesos is transforming what is now an abstraction layer in the cloud stack into what proponents call a "datacenter operating system."

"Many of the folks vetting [Mesos] are looking to understand how this technology will play in their container strategy," Twitter software engineer Dave Lester noted in a recent blog post updating the project.

Apache Mesos was originally designed to abstract CPU, memory, storage and other compute resources away from both physical and virtual machines, enabling the design of "elastic" distributed systems. Lester argues that Mesos is evolving beyond a mere tool to what he called "operating system kernel."

When a set of application frameworks including Apache Aurora and Google Kubernetes are built on top of it. "The Mesos ecosystem works like a data center operating system," Lester maintained.

The Mesos framework called Aurora is intended to run applications and services across a shared pool of machines. Kubernetes is used to manage clusters of Linux containers across multiple hosts.

Lester is beating the drum for Apache Mesos in advance of the open source project's second annual conference, MesoCon, scheduled for Aug. 20-21 in Seattle. Twitter was an early adopter of Apache Mesos, with Lester serving as chief evangelist and program chair of the August 2015 conference. Mesos continues to gain traction as hyperscale operators like AirBnB, eBay, Netflix and PayPal adopt the datacenter technology.

“The vision of Mesos frameworks is to make it as easy to write and run applications in the datacenter as it is for applications on your desktop or cell phone,” Lester told Linux.com.

Twitter has been running its cloud infrastructure for several years on Apache Mesos. It is used to abstract compute resources away from servers and cloud instances as a way of simplifying and scaling resource management and scheduling across datacenters and public clouds.

Benjamin Hindman, who founded the Mesos project at the University of California at Berkeley and later served as lead engineer on Twitter's Mesos project, left last September to join Mesophere. The startup was formed in 2013 to commercialize Mesos technology. The company announced a $36 million funding round in December, led by new investor Khosla Ventures.

Investor Vinod Khosla said the emerging datacenter operating system promises to fill the gap for delivering applications: "Existing virtualization or orchestration software are insufficient to deliver on the end goals: abstract complexity, optimize utilization and simplify or eliminate human intervention in the face of dynamic failures, demand surges, constant provisioning and upgrades."

Mesosphere has so far raised $50 million in venture capital to commercialize Apache Mesos technology. Mesosphere beta launched its Data Center Operating System in December following several years of development.

Since then, the open source community has been rallying around the Apache Mesos project as it emerges as a next-generation cloud technology that promises to merge compute and batch workloads through an abstraction layer proponents say is morphing into a datacenter operating system capable of managing both workloads.

EnterpriseAI