MongoDB Users Discuss Their NoSQL Journeys

(Joe Techapanupreeda/Shutterstock)

NoSQL databases account for a very small fraction of the overall database market, yet this segment is growing much faster than the overall market. What’s driving this big demand in NoSQL? Last week at the MongoDB user conference, several of the database company’s customers, including Experian Health, HSBC, and Envision, shared their NoSQL stories.

HSBC is the world’s seventh largest bank, with assets valued at $2.3 trillion. Like most global firms of this size, HSBC heavily relies on a variety of technologies to run its businesses, including proven platforms like the IBM z/OS mainframes and Power Systems servers running the IBM i operating system.

More recently, the London-based company’s Investment Bank Technology division has begun adopting newer technologies, including NoSQL databases, to power parts of its extensive operation. At last week’s MongoDB World 2017 show in Chicago, Illinois, HSBC Development Manager Nick Maybin and Equities Architect Andrew Matthews told the audience about a new operational data store (ODS) HSBC is building atop MongoDB.

The new ODS is designed to be the “single source of truth” for its global equities and fixed income trading systems, they said. “It’s an operational data store that we’re building to enable, equip, and engage our trading teams, salespeople, and operations across the globe [to serve] up data consistently and reliably,” Maybin said.

The plan called for letting users pull data out of the ODS via the microservices model, where the data store is connected to a “view server” that sends the requested data to the user over the Web via sockets. This lightweight, Web-based approach lets HSBC users around the globe tap into the data service as they need it, while minimizing overhead and latency – an approach that values simplicity over complexity.

However, news of the move to MongoDB did not initially go over well when the project started 18 months ago. “Initially in my department, there was some concern and reservation around moving to Mongo and NoSQL and that was about the perceived lack of a schema,” Maybin said.

The big concern was that a schema-less approach would lead parts of the system to “spin out of control,” he said. “But it wasn’t long after I started using MongoDB that I realized that there was a schema, and it’s implicit,” he added.

As HSBC developers’ comfort level with NoSQL grew, they found other benefits, including improving transparency for regulatory compliance. The availability of MongoDB was another selling point.

“In the old world where we had systems that were reliant on relational databases operating in two data centers…waiting that manual failover to occur can be quiet an excruciating experience,” Maybin said. But resiliency and failovers come out of the box with MongoDB. “There’s not much more than a couple lines of configuration,” he said.

One of the biggest challenges for HSBC has been how it delivers ACID (atomicity, consistency, isolation, and durability) control in its transactional systems. “ACIDity is a big deal for us,” said Mathews, the equities architect. “We do need it. We just have to solve it in a slightly different way with MongoDB.”

Because MongoDB doesn’t provide all the ACIDity that HSBC requires, the company has looked to developers to deliver what they need.

“You have to support it at the application level. And that’s not a terrible thing,” Mathews said. “We’ve been supporting transactionality at the application level and service level for some time, in all sorts of ways, before Mongo was around. That is a perfectly valid solution in my opinion.”

MongoDB Does Healthcare

Another big company adopting MongoDB for critical business applications is Experian Health, the Franklin, Tennessee-based division of the $4.5 billion consumer data company that plays a critical role as a clearinghouse for medical data and informational middleman between healthcare providers and insurance companies.

Hospitals and physician groups rely on Experian Health to pull up the details of their patients’ insurance coverage, and to basically ensure that they’ve dotted all their “i”s and crossed all their “t”s when it’s time to submit a claim to the insurance company. That involves integrating data from a wide range of hospital information systems (HIS) and electronic health record (EHR) systems.

Recently the company selected MongoDB to power its Universal Identification Manager, a new application Experian Health uses to uniquely identify healthcare customers. Many medical groups have moved away from using Social Security Numbers to identify patients, which drove the need for a new way to positively identify people.

To determine which database should power this new application, Experian Health held a bakeoff between Oracle‘s MySQL, EnterpriseDB‘s Postgres, Microsoft SQL Server, and MongoDB. “It was hands down Mongo,” said Experian Health CTO Mike Ochs.

This particular application and the way it needed to store data about people, it was a great fit for MongoDB, Ochs said. “It was hands [MongoDB] down because the way the data is organized lends itself very well to the way Mongo manages data,” Ochs told Datanami. “Because it’s not really relational. It serves itself well for a document data store.”

If Experian Health had developed the same system on a relational database, it would require executing 10 SQL joins in order to make a positive match on somebody’s identity, whereas with MongoDB, it’s a single query. “It was way simpler from a development perspective, and performance wise….it is essentially 4x more performant.”

Ochs is now overseeing a much bigger and more complex MongoDB implementation that’s due to go live next month. “We’re in the process of re-architecting our primary workflow engine out of SQL,” Ochs said. “That’s a big transition, because it’s going from C#, Windows, and MS SQL to Java, Linux, and Mongo inside of Docker and Kubernetes.”

Moving this primary data store – which holds petabytes of data on more than 300 million Americans and their hospital and doctor’s visits – to MongoDB makes sense because of the underlying format of the data. “It’s not really relational data,” Ochs said. “It’s really a lot of documents, and so it lends itself real well” for a NoSQL store.

Experian Health picked MongoDB to store data that isn’t easily expressed in a relational database, according to CTO Mike Ochs

That doesn’t mean the company is throwing out relational databases or SQL. “We’re trying to make sure we have the right use case for the right technology,” Ochs said. “We’re not going to stop using SQL. We’re just going to use it for the right things.”

The transition from Windows to Linux, it turns out, has been the bigger trouble spot for Experian Health (the switch from C# to Java was practically painless). As part of the re-platforming of the core database onto MongoDB, the company is also looking to embrace new development constructs, such as agile and DevOps. Part of that means taking business logic out of the database and putting it into the hands of the programmers.

Ochs said it’s a tradeoff. “I would say we have more developers than we do DBAs, so they would tend to be a bottleneck in that respect,” Ochs said. “Very early on there was way more business logic in the database than we would like. And the compute is cheaper on the application side for us.”

The biggest reason for moving from SQL Server to MongoDB, however, lies in the fact that the old system was hitting up against technical limits of the system. At times, Experian Health had 30,000 concurrent query connections from Windows to the clustered SQL Server database. By moving to MongoDB, the company is essentially pairing down those queries.

“You just make one connection and get a wider query,” Ochs said. “We’ve already reduced the connections significantly by doing that work. Mongo doesn’t gain us anything in that respect. It just so happens that you don’t put a lot of business logic in Mongo anyway, so the path we were moving in was a really easy transition.”

Developers, Developers, Developers

The migration from Postgres to MongoDB has paid dividends for InVisionApp, which develops a popular cloud-based Web and mobile application design tool.

InVision essentially lets developer whiteboard their application flows, and keeps track of all the designs in a database. It’s not that you couldn’t keep copies of all these screens and the metadata that describes the workflow, says CTO Bjorn Freeman-Benson. But it would be a lot harder.

“Fundamental our data is prototypes of designs, screens and metadata about the screen and transitions between the screens,” he told Datanami. “I could see how you [could store that] in rows and columns. But it turns out you need a really complicated schema to make that work. Instead, we just store it as documents, and you don’t need all that crap.”

While efficiency of data storage is important to a startup, developer productivity is even more important, Freeman-Benson said. And since the development team at InVision requested MongoDB, that was a big checkmark in its favor.

“You’ve got to have sufficient performance in your apps. But the cost efficiency isn’t a factor here. It’s the speed of development,” he said. “Primarily the reason we chose Mongo was it did everything it needed to do, and the developers liked working with it. And given a choice, I’d always rather choose something the developers like working with, because then they’re going to be happier and more productive.”

Most of InVision’s app is written in Go and Node, and anything that can accelerate their productivity gives InVision a leg up against its competitors. “You could have written the entire thing that we’ve written in Fortran,” Freeman-Benson said. “But it would have taken you a long time, because you’d really have to stretch yourself to figure out how to fit it all into that archaic, old model.

“We want to use tools that allow us to move quickly,” he continued. “We want to go have an idea on Monday and roll out a new feature on Friday. In order to have that kind of velocity, you can’t be spending a lot of time in the mechanics of putting data together.”

Related Items:

MongoDB Pivots to Analytics

MongoDB Takes Another Big Step Into Clouds