A MongoDB Secret Weapon: Aggregation Pipeline

(Aurora72/Shutterstock)

MongoDB is best known for creating a document database that Web and mobile developers love to use. But developers and analysts alike may be interested in a little-known MongoDB feature called the aggregation pipeline. What’s more, the aggregation pipeline just got easier to use with MongoDB 4.0.

The aggregation pipeline presents a powerful abstraction for working with and analyzing data stored in the MongoDB database. According to MongoDB CTO and co-founder Eliot Horowitz, the composability of the aggregation pipeline is one of the keys to its power.

“You start with data and build little things on top of each other,” Horowitz said in his keynote address today at MongoDB World 2018 in New York City. “You’re building this as you go, which also makes it really easy to debug, because you chop off that pipeline at any point and it still works. It’s incredibly powerful. You can do almost anything with the pipeline.”

So where did the MongoDB aggregation pipeline come from? It was added in 2012, according to Horowitz, and was actually modeled on the Unix pipeline, which allows developers to string together a series of processes that work together.

“I don’t know about you, but I think the Unix pipeline is one of the greatest pipelines out there,” Horowitz said. “It’s a combination of little things that together are an incredibly powerful tool.”

With today’s announcement of MongoDB 4.0, the company is debuting the Aggregation Pipeline Builder, the new offering in the Compass management interface that gives users a GUI for constructing data aggregation pipelines in MongoDB.

MongoDB is shipping a new Aggregation Pipeline Builder in its Compass management UI that makes it easy to build aggregation pipelines

According to MongoDB, the Aggregation Pipeline Builder in MongoDB Compass provides the ability to create aggregation pipelines to process data. “With aggregation pipelines, documents in a collection pass through multiple stages where they are processed into a set of aggregated results,” the company says. “The particular stages and results can be modified depending on your application’s needs.”

The new software gives users the capability to create aggregations in a simple and real-time manner. As users select different processes to execute from drop-down boxes on the left side of the screen, the results of those actions are shown on the right side of screen. Users can add as many stages as they want, and save or clone pipelines as needed.

When a user is done building a pipeline, they can copy it to the clipboard. The resulting JSON code can then be executed against the MongoDB database using the shell environment.

Horowitz admitted that the MongoDB Aggregation Pipeline “flies under the radar” for a lot of people. “We’ve done poor job explaining it.” However, that could be changing with the launch of the Aggregation Pipeline Builder in MongoDB 4.0 and Compass 1.14.

“I have built and debugged a lot of really bizarre aggregation pipelines over the last two years,” Horowitz says, “and I cannot tell you how much easier it is with this tool, where you can take a whole big thing, put it in here, work with it and play with it, then export it to a language and be done.”

Horowitz said his favorite feature in MongoDB 4.0 is the Aggregation Pipeline Builder. “When I show this around the office, people have been incredibly excited about it,” he says.

The Aggregation Pipeline Builder is one of many enhancements that MongoDB unveiled today at MongoDB World.

Related Items:

MongoDB Fleshes Out Mobile and Cloud Strategies

Full ACID Support Means ‘Game Over’ for Relational, MongoDB Says