Coralogix Brings ‘Loggregation’ to the CI/CD Process

(Paolo-De-Gasperis/Shutterstock)

Most developers by now are familiar with the concept of continuous integration/continuous delivery (CI/CD). The ability to automatically push a large number of relatively small updates into production provides much of the basis for today’s DevOps productivity. But how does log data play into CI/CD? One of the leaders in this field, Coralogix, says it’s all about the “loggregatation.”

Coralogix develops a software as a service (SaaS) application that analyzes log data flowing from the CI/CD process to provide the DevOps professionals with insights about the state of their application, including possible errors and even security concerns. The software, which is built on the Elasticsearch, Logstash, and Kibana (ELK) stack, uses a mix of machine learning and visualization techniques to accelerate the identification of problems in the application, which of course are expressed through the logs.

“Logs are the first encounter of your code with reality. They’re the bed source of proof you have in your organization,” says Coralogix CEO Ariel Assaraf. “I think we’re the first company ever to look at log data as a way to improve the CI/CD process and not just for monitoring and analytics.”

As an AWS app, Coralogix hooks into Jenkins other popular CI/CD tools to ensure that it’s seeing all the updates that are pushed down to production systems. It doesn’t matter whether the application is on-prem or running in the cloud, uses Syslog or Amazon’s JSON format – Coralogix likely has an integration ready to go to bring the logs into its customers S3 data repository.

The software builds upon AWS’s Open Distro for Elasticsearch and brings proprietary algorithms to bear on log data. Coralogix says it takes a couple of days after installation for the machine learning algorithms to get a handle on basic behavior of a given application, and a few more days after that to understand how multiple applications work together.

Whenever a piece of code is deployed through the CI/CD process, Coralogix assigns it a tag that allows it to track the code. If an application starts misbehaving or throwing up lots of errors, the tag let’s Coralogix trace that behavior to a particular piece of code. In this manner, the software helps DevOps professionals react quickly to problems and improve the quality of their code.

“We basically know everything that’s happening in your software from your logs. Parameters, different patterns, behaviors, trends – everything is now known, and then it’s tied up to your CI/CD pipeline,” Assaraf says. “So at the end of the day, what you’re getting is this better version, faster time to market, and you can basically perform CI/CD or accelerate it without comprising the quality of your production.”

Coralogix provides “loggregation” for big log data

Coralogix describes this process as “loggregation,” which is an amalgamation of logs and aggregation. While it provides its own proprietary user interface to help customers drill into the application workflow (as expressed by the logs), customers who prefer Grafana or Kibana (other components of the ELK stack) can use those, Assaraf says.

Coralogix offers different capabilities for different groups of users. Developers can use it to optimize the performance of their applications, Assaraf says, while DevOps folks will focus on the CI/CD process itself. Site reliability engineers (SRE) and support engineers will connect their Slack channels to receive alerts from Coralogix when it detects that service level agreements (SLAs) are in danger, while security pros will use it to keep on top of log-born anomalies that could be connected to the activities of cybercriminals.’

While the company elected to standardize on the popular open source ELK stack for much of the log work, it went the proprietary route for the development of machine learning algorithms for the log data.

“We didn’t find any algorithms out of the box or any package that would fit the world of problems of logs in a way that would be sufficient,” says Assaraff, who helped co-found Coralogix in 2015 and became its CEO in 2019. “We’ve seen a lot of products create false positives and issues because logs are behaving in a very certain way.”

Coralogix CEO Ariel Assaraf served in the prestigious 8200 unit of the Israel Defense Force before co-founding the company

Coralogix uses different types of algorithms to tackle different problems. The algorithms are designed to detect certain characteristics in the log data, and are designed to resist problems that can trip up other automated methods, including when application volumes spike or when application flows break.

Log data has long been identified as a big data problem, but elegant solutions for making sense of log data have been elusive. Coralogix is convinced that its “loggregation” approach has legs.

“We basically disconnect the problem of big data from log data,” Assaraf says. “Every time you run a query in Coralogix, we tell you have millions of results and 25 symptoms. So you just click a feature called loggregation. Then see those grouped results telling you this log happened 3,000 times, these are the parameters and it’s 15% of your logs. Then you can click into behavior.”

With customers like Lufthansa, KFC, and Caesars Palace, Coralogix is gaining traction. The company, which is based in San Francisco, raised $10 million in its Series A round last year.

Related Item:

How Big Data Improves Logging and Compliance