Stonebraker Seeks to Invert the Computing Paradigm with DBOS

(Bruce Rolff/Shutterstock)

In the current tech paradigm, databases run on top of operating systems. But what if that stack was inverted, with an operating system running on top of the database? That’s the idea behind database guru Mike Stonebraker’s new startup DBOS, or Database-Oriented Operating System, which launched its commercial service on AWS today as well as an $8.5 million round of funding.

Stonebraker–who led the teams that created several databases (Ingres, Postgres, Vertica, VoltDB, SciDB) over the years and also won a Turing Award for his work–is known for out-of-the-box thinking and having a little bit of a contrarian streak. For instance, when most of the computing world was singing the praises of Hadoop back in 2014, he was pointing out its flaws five years before the big yellow elephant floundered and fell.

“And I was completely right,” Stonebraker said last week in an interview with Datanami.

But running an operating system inside of a database? OSes have always been the software abstraction sitting closest to the bare metal. They have been relied on to control everything in the computer. Why on Earth would Stonebraker want to flip it around and put the database in charge of the hardware, and turn the operating system into just another service offered by the database?

It turns out Stonebraker has given the matter a great deal of thought, which isn’t surprising. The answer to “why” emerges out of three main reasons.

The first has to do with the huge amount of OS data being generated in large clusters today. As distributed computing has grown, the volume of node-to-node communications in a cluster has grown by an enormous amount, Stonebraker said.

DBOS co-creator Mike Stonebraker

“The operating system state, which is all the data you have to keep track of if you’re the operating system, is basically proportional to the resources you have at hand and that’s gone up by six orders of magnitude in the last 40-ish years,” he said. “So without me saying another word, keeping track of operating system state is a database problem. So that was the inspiration, number one.”

The second reason was how fast OLTP databases have become. Stonebraker may have wanted to put the OS in the DB in the past, but they just weren’t up to the task. That’s no longer true. “OLTP databases have gotten wildly faster in the last 15 years, so my supposition was that you could run the operating system on top of the database, and it would work out just fine,” he said.

The third reason stems from a talk that Stonebraker happened to hear. Apache Spark creator and Databricks co-founder Matei Zaharia talked about the difficulty of managing OS state in the cloud clusters that Databricks’ runs on behalf of customers.

“He said Databricks is routinely orchestrating a million Spark subtasks on a sizable cloud, and he said it was very clear that scheduling a million subtasks cannot be done with conventional operating system techniques,” Stonebraker said. “He said he put all the scheduling information into a Postgres database and a Postgres application is doing the scheduling.”

Maintaining operating system state in the operating system is basically impossible for any cluster at Databricks scale, Stonebraker said. “So we started chatting, and he and I sort of got going on the DBOS project,” he said.

The DBOS project commenced at MIT and Stanford, with code openly shared on its GitHub page. Led by the two computing legends, the team of university scientists quickly hammered out what the project would look like. It would provide the essential services that every OS needs, such as a file system, a scheduling engine, and a messaging system. These were coded in SQL.

The first version of DBOS was written in Java and used VoltDB, the fast relational database created by Stonebraker over a decade ago. But early feedback from interested parties said a proprietary system was a no-go, so the commercial version was rewritten to use FoundationDB, a fast key-value store acquired by Apple nearly a decade ago. Java was jettisoned in favor of Typescript.

Users can utilize the FoundationDB database exposed by DBOS, or they can choose to run any Postgres-compatible OLTP database on top of DBOS, such as CockroachDB, YugabyteDB, Citus, and others. DBOS itself runs on AWS and uses Firecracker, its lightweight virtualization software for serverless computing. Users are given an SDK to develop applications in TypeScript. That’s the commercial offering that the DBOS company is now selling on AWS. An open source version is available too.

But it’s more telling what DBOS does not contain. “Linux is nowhere to be seen. Kubernetes is nowhere to be seen,” Stonebraker said. “And if you have a transactional file system in your stack right now, there’s no need for it since we provide one automatically. So bunches of stuff go away. Life is a lot simpler.”

That simplicity brings several main advantages, the biggest one being security improvements. Without Linux, Kubernetes, and a host of security packages intended to address security weaknesses in the architecture, DBOS presents a much smaller attack surface than the traditional stack.

“Most shops are a complete mess because they have Linux running everywhere, they have Kubernetes running everywhere. They’ve got a bunch of security packages running on top of that,” Stonebraker said. “You have a huge attack surface, so it’s easy to break in. Because it’s very hard with a very complicated requirement to make sure you’ve closed all the doors. And we just get rid of all that stuff. So it’s a simpler system administration world. You get a much more secure world. And you get a much better debugging world.”

Keeping OS state in the DB also enables DBOS applications developed with the TypeScript SDK to time travel. Stonebraker explained:

“If this is fast enough for OS stuff, it’s certainly fast enough for your application,” he said. “So if you put all of your application state in the database, then you can time travel everything. So if there was a ransomware attack 15 minutes ago, you just back up everything 16 minutes, single step around the problem, and you’re back up and running instantaneously.”

The time travel function also helps with debugging. Users can back up their applications, then single-step it forward while changing variables to see what breaks, Stonebraker said. This is particularly helpful when trying to track down issues occurring among large number of parallel micro-operations, he said.

“We give you a much better debugging experience that avoids a lot of the parallel problems that come with that territory,” he said. “So fancy debugger. Simplified systems administration. Much better security story. That’s what we have to offer.”

DBOS as it exists currently is cloud-only, which is where users tend to run into large-scale issues. Initial users look to be governmental agencies that demand the highest levels of security’ “adventuresome” folks in financial services who can benefit from DBOS support for “once and only once” distributed transactions semantics; and “West Coast startups who want the next shiny thing rather than yesterday’s shiny thing,” Stonebraker said.

Companies are dragging along 50 years’ worth of legacy code, and are getting tired of it, he said. As the cloud beckons, they’re faced with one of two choices: Move that legacy into a cloud environment, which keeps the same complexity, cost, and security challenges that existed on-prem, or take the time to rewrite the application for the cloud. DBOS represents a once-in-a generation opportunity to refactor that application for the cloud and deliver a vastly superior product, he said.

DBOS will resist being on-prem and will resist being POSIX compliant, Stonebraker said. But the company is open to what early adopters want, and if they demand on-prem and POSIX, then that’s what they’ll give them. They may also want things like support for Python and Java programming environments, and support for running in Azure and GCP, which will be determined in the future. “If there are exotics services that we don’t provide that enough people want, we’ll of course support them in SQL,” he said.

“The idea is to get a product out there as quickly as we can,” Stonebraker said. “What we wanted to do is see if the dog eats the dog food. And we will get very clear very quickly why they do or do not salute.”

Related Items:

Postgres Expands Its Reach

Array Databases: The Next Big Thing in Data Analytics?

Breaking Down the Seven Tenets of Data Unification