Advanced Computing in the Age of AI | Friday, March 29, 2024

StackEngine Brings Automation To Docker Containers 

The rush to commercial Docker software container technology for Linux systems is well underway and very likely getting close to a hype cycle, but the fact remains that Docker is a transformational tool that is hitting the IT market in precisely the right way at exactly the right time. A startup called StackEngine has just uncloaked from stealth mode to help bring Docker into the enterprise, and no doubt many others with experience in managing various kinds of virtualization will be adding Docker support to their wares as fast as possible.
The issue that StackEngine is tackling is the same one that has turned VMware into a $6 billion company. While the underlying ESXi hypervisor has some value as it abstracts away the hardware, the real value with full-on server virtualization is that it makes hardware more malleable and allows for the automation of software workloads in a production environment. StackEngine co-founders Eric Anderson, who is CTO, and Bob Quillin, who is CEO, explained to EnterpriseTech that there are tools from Docker itself to manage the creation of applications running inside of Docker containers, there are not yet sophisticated tools for deploying Docker software in production and managing the entire lifecycle of a Docker software stack. Telco and cloud provider CenturyLink has created Panamax from a bunch of open source tools, and Google, Mesos, IBM, Microsoft, Red Hat, and Docker are working together on the Kubernetes container scheduler tool that Google opened sourced earlier this year. But no one has, as yet, created the vCenter management console analog.

StackEngine wants its eponymous tool to be that analog, and Anderson says that the company is working on integrating Kubernetes into StackEngine. "We would like to be able to leverage Kubernetes and Mesos as scheduling engines, but these are only one piece of the puzzle for managing Docker," says Anderson, adding that the company will very likely have a plug-in approach for schedulers and other Docker tools that will allow customers to pick and choose their schedulers of choice as they become available. (You can read our report on Mesos at this link.)

StackEngine takes a more holistic approach to Docker container management, says Anderson. It provides complete visibility into the Docker estate, including what is inside of the containers, what bare metal or virtual machine hosts the containers are running on, the dependencies between the hosts (either virtual or physical), and so on. System administrators build a desired running state for each container in terms of what budget it has to run on particular hardware configurations, what security it needs, and what quality of service it requires, and StackEngine then looks at the entire infrastructure estate and finds the best place to plunk that container to meet all of the criteria. As workloads move around and as the underlying infrastructure changes, the matchup between a container and its place will also change, and the idea is to automate all of this placement and movement as well as the updating of the software inside of the containers once it is deployed into production.

This is precisely the kind of work that system administrators are doing mostly by hand today with Docker and used to do by hand with full-on server virtualization hypervisors before the ESXi, Xen, and KVM stacks matured a few years ago. And this is also why VMware has promised to bring Docker management into vCenter, although precisely how it will do this remains unclear.

"Big organizations are now dealing with the IT operations bottleneck," explains Quillin. "Developers are diving in and using Docker in droves, and many of the tools out there are designed to get applications up and running because that is where the early activity is. What we have found is that many organizations are stuck in a delay mode in that they can get it working in development but there are no operational tools on the flip side for deploying Docker applications on thousands of servers and tens of thousands of containers."

The interesting bit, of course, is that many organizations are deploying Docker containers atop of server virtualization hypervisors, providing more layers of abstraction and control for the underlying infrastructure and allowing for finer-grained control of resource allocation and the secure mixing of workloads on top of shared hardware. This is, in fact, how Google deploys containers on its own Compute Engine public cloud. Google lays down containers (called cgroups and not based on Docker) on its bare metal, fires up a KVM virtual machine on top of them, and then puts another layer of cgroups containers inside of that where end users actually deploy their workloads. If you want to deploy a fully operating system, you drop it into the KVM virtual machine. For its internal workloads, where Google has its own variant of Linux, it presumably does not nest virtualization technologies because it need not worry about security and isolation issues as it must in the public cloud.

stackengine-mesh

StackEngine itself is written in the Go programming language, one that is favored by Google and was used to create the Kubernetes scheduler/orchestrator. (If you are a Go programmer looking for work, StackEngine has a job for you.) StackEngine, the tool, creates what Anderson calls a "management mesh" that can discover all of the Docker containers and their hosts. The communication protocol that is used to gather data from the containers is based on the Raft distributed consensus protocol, something akin to the Paxos protocol used by Google to manage its geographically distributed systems. Anderson says that StackEngine has tested the tool scaling up to managing 20,000 containers in its labs. The tool has an operations dashboard that does container inventory and provides a top-level health status of containers. You drill down into the dashboard to look at the configuration and health of specific groups of containers or a single container. You can start, stop, pause, and unpause containers, sort them in various ways, and identify zombie or stopped containers and configure and reconfigure the containers.

The tool is a beginning, not an end. To get rolling, StackEngine, which is based in Austin, Texas, has lined up $1 million in seed funding from Silverton Partners and LiveOak Venture Partners. As the company is coming out of stealth today, there are 15 companies that are in the alpha testing program. Several of them have been using LXC Linux containers for a while and are moving to Docker containers. Others are moving from VMware ESXi supporting Linux workloads to Docker. It is about a 50-50 split between customers moving new to any type of container and getting rid of VMware in favor of Docker. While VMware may not be thinking of Docker as an ESXi replacement for Linux workloads, clearly some big customers are. (Azul Systems, which has created a Java environment tuned for Docker, is seeing the same phenomenon.) Quillin says that there are some cloud and hyperscale customers testing out StackEngine as well as a few banking, insurance, and healthcare companies. In some cases, they want to put containers on top of their virtual machine environments for more application flexibility.

StackEngine 1.0 is expected to be generally available by the end of the year, and the company is looking at adopting LXC containers as the next obvious step. The company is also looking at how to integrate the management of other virtualization environments, and some customers are even asking StackEngine to see how they might manage the underlying bare metal. ESXi and Xen are on the radar, says Anderson, but so far there is not much demand for KVM, oddly enough. Integrating StackEngine with the OpenStack cloud controller is probably something that needs to be considered, too, and clearly it is vital to get schedulers integrated with the tool so it is not just used for monitoring and configuration, but for automating workflows. StackEngine is not announcing prices as yet, but say that it will be similar to the commercial versions of the Chef and Puppet system configuration tools, with per-host or per-VM fees. StackEngine will have a freebie version to get developers going and then a paid-for version with all of the bells and whistles.

EnterpriseAI