Altoros Report Compares Two NoSQL Databases

Altoros Systems has released a report comparing the performance, in terms of latency and throughput, of two modern NoSQL database-as-a-service products, Couchbase Capella (the release as of March 2022) and MongoDB Atlas v5.0.

Altoros is a Sunnyvale, California-based IT services provider that helps enterprises evaluate technology and implement technological solutions spanning AI, blockchain, and operational databases. The company has been running periodic benchmark testing on NoSQL databases since late 2018, and readers should be aware that it is an active contributor to Couchbase Server, an open source, distributed NoSQL document-oriented database. Also, Couchbase has compensated Altoros for past benchmark reports, though it is not immediately clear if that is true in this case.

Based on the results, Altoros concludes that Couchbase Capella outperforms MongoDB Atlas when running the same workloads on the same AWS cloud resources at a 40% lower operating cost.

Source: Altoros

Configuration Specs

Altoros says this evaluation was conducted on three different cluster configurations— 6, 9, and 18 nodes—as well as under four different workloads. “The first workload performs update-heavy activity, invoking 50% reads and 50% updates of the data. The second workload performs a short-range scan that involves 95% scans and 5% updates, where short ranges of records are queried instead of individual ones. The third workload represents a query with a single filtering option to which an offset and a limit are applied. Finally, the fourth workload is a JOIN query with grouping and ordering applied,” as stated in the report’s executive summary.

The benchmark was performed on Amazon Elastic Compute Cloud (EC2) instances. For consistency in this evaluation, the company utilized the Yahoo! Cloud Serving Benchmark (YCSB), which it names as an open source specification and program suite for evaluating retrieval and maintenance capabilities of computer programs. Technical specifications for the Couchbase Capella instance environment included 8 vCPUs, 64GB memory, 200GB Amazon Elastic Block Store storage, with an IOPS rating of 5,700. The MongoDB Atlas instance, type M60, was set up in the same way but with 200GB of SSD storage.

Source: Altoros

Pricing

As far as monthly cost is concerned, for Couchbase Capella, pricing for 6, 9, and 18 nodes was $5,284, $7,926, and $15,851, respectively. For MongoDB Atlas, prices include $1 per server per hour with $0.000182 per GB per hour for data storage and $0.01 per GB for data transfer. Six nodes came to $9,756, nine nodes came to $14,292, and 18 nodes came to $28,050.

The report notes that operating costs are different between the two databases, and Capella is priced at 40% less than Atlas due to different architectures and customizations intended to create fair comparison conditions for the benchmark.

Results

The first workload is an update-heavy workload that simulates the typical actions of an e-commerce application. Results concluded that Capella outperformed Atlas on a 6-node cluster with a throughput of 355,520 ops/sec vs. Atlas’s 31,030 ops/sec. Capella also surpassed its rival by processing up to 417,980 ops/sec on a 9-node cluster and 523,020 ops/sec on an 18-node cluster, while Atlas processed 38,340 ops/sec throughput on a 9-node cluster and 55,870 ops/sec. on an 18-node cluster. For each test, Couchbase Capella showed a latency of a millisecond: 1.5 ms on a 6-node cluster and less than 1 ms on 9-node and 18-node clusters. The latency of MongoDB Atlas was halved from 15 ms on a 6-node cluster to 7 ms on an 18-nodes cluster. The report states that both Atlas and Capella showed stable results without failed operations.

The second workload is a short-range scan workload, simulating threaded conversations, in which short ranges of records are queried instead of individual ones. Capella demonstrated a throughput of 30,388 ops/sec on 6-node clusters while Atlas had a lower rate of 20,532 ops/sec. On 9-node clusters, Capella showed a rate of 53,092 ops/sec, and Atlas showed 21,220 ops/sec. Capella continued to increase the throughput up to 98,808 ops/sec on 18-node clusters while Atlas demonstrated 23, 276 ops/sec. Again, Capella increased its throughput as the number of nodes increased, and latency decreased with more nodes from 7 ms on a 6-node cluster to 3.2 ms on an 18-node cluster. Atlas demonstrated the lowest latency on 18 nodes at 21 ms, and it decreased from 29 ms on 6-node clusters.

Source: Altoros

The third workload is a pagination workload with a single filtering option, an offset, and a limit, and it simulates a selection by field with pagination workload used for e-commerce listings or search engine results. Atlas showed a lower throughput on a 6-node cluster at 36,392 ops/sec versus Capella’s 51,952 ops/sec. Atlas slightly decreased throughput on a 9-node cluster with a rate of 35,325 ops/sec while Capella increased its throughput at 60,680 ops/sec. Capella performed best on 18-node clusters with 65,228 ops/sec, while Atlas managed a throughput of 29,300 ops/sec on the same cluster size. Capella again gradually increased throughput by the number of nodes, but latency rose from 3 ms on a 6-node to 6 ms on an 18-node cluster. The report notes that Atlas had the inverse result, performing progressively worse as the cluster size increased, and latency increased from 11 ms on a 6-node cluster, to 12.4 ms on a 9-node cluster, and 14.4 ms on an 18-node cluster.

Finally, the fourth workload is a JOIN query with grouping and ordering applied, simulating a selection of complex child/parent relationships with categorization. Results for this workload showed Capella’s performance at 6, 9, and 18 nodes was 34,448 ops/sec, 49, 624 ops/sec, and 56, 828 ops/sec, respectively. Atlas demonstrated 884 ops/sec, 196 ops/sec, and 60 ops/sec on the same number of nodes. Compared to Atlas, Capella showed higher throughput and lower latency on all cluster sizes for this workload, with a consistent latency of 9 ms across sizes.

Source: Altoros

Conclusion

The report concludes with the idea that no single DBaaS is perfect for every scenario, and each has unique advantages and disadvantages. However, in Altoros’s estimation, Couchbase Capella better performed in all workloads compared to MongoDB Atlas in all three cluster size categories. “As clusters and data sets grow in size, Couchbase Capella ensures a high level of scalability across these operations,” the report’s conclusion states. “MongoDB Atlas produced comparatively decent results. The database is scalable enough to handle increasing amounts of data and cluster extension. Under this benchmark, the only issue we observed was that MongoDB Atlas did not support JOIN operations on sharded collections out of the box. This resulted in poor performance in JOIN Workload.”

“As in previous benchmarks, Couchbase Capella demonstrated better performance than Atlas due to its active-active, all-worker nodes architecture,” said the report’s author, Ivan Shyrma, a data engineer at Altoros. “Capella is also easier to query due to its SQL support. These factors translate to a better price-to-performance ratio in real-world environments.”

To explore the full technical scope of the report, download a copy at this link.

Related Items:

NoSQL Database Smackdown Takes to the Clouds

Vendors Compete to Make Serverless NoSQL in the Cloud Drop-Dead Simple

Benchmarking NoSQL Databases