Telemetry with Scala, part 2: Kamon

Introduction

Welcome to the second part of our telemetry with Scala series! In the previous post, we introduced OpenTelemetry, a vendor-neutral open-source framework for generating, collecting, and exporting telemetry data. We explored how OpenTelemetry can help us gain visibility into the behavior of distributed systems by capturing traces, metrics, and logs. In this post, we will dive into Kamon, a popular telemetry solution for Scala applications. Kamon provides a range of powerful tools for monitoring and troubleshooting applications, including distributed tracing, metrics, and error tracking. We will walk through the key features of Kamon, how to configure it, and how to integrate it into a Scala application. So let’s explore Kamon and learn how it can help us monitor and improve the performance of our Scala applications.

Previous part recap

Before proceeding, first, let’s have a quick recap of the system under monitoring from the previous post. This is a simple task ticketing (like Jira) service written on top of Akka HTTP and Slick and provides a very basic CRUD REST API for ticket management.

It uses the following external services:

• PostgreSQL database to store tickets;
• Kafka to publish events about tickets creation and update (for example, to send notifications to users);
• ElasticSearch for tickets full-text search;
• project-service to get information about a project to which the ticket belongs (mocked service);

For this service we would like to monitor a couple of simple things:

• Number of tickets in the system;
• Traces for each request;

To simulate user traffic (create, read, update, and delete tickets) Gatling load testing tool was used.

Kamon

Overview:

Kamon is a popular telemetry solution for monitoring and troubleshooting Scala applications. It provides a suite of tools for capturing and analyzing telemetry data, including distributed tracing, metrics, and error tracking. Kamon is designed to be lightweight and easy to use, with a simple API and minimal overhead. Kamon provides, out of the box, a number of instrumentations like Akka HTTP, Spring Boot, Play Framework and Lagom. The full list of supported instrumentation’s can be found here

Similarly to OpenTelemetry, Kamon has also concepts of Tracing and Metrics. Additionally, Kamon can wire additional context, which might help to debug certain traces.

How to plug instrumentation to an application

For a particular example case, we are interested in Akka instrumentation, which is described here NOTE: At the moment of writing this blog post, the used version of Kamon is 2.6.0. The latest version can be found here

First of all, add Kamon dependencies to build.sbt:

kamon-bundle includes all instrumentation's that are available for the current version of Kamon, so we don't need to add each instrumentation separately. But if you want to add only specific instrumentation's, you can do it by adding specific dependencies, for example:

Then, we need to include Kanela agent to our application, so to enable instrumentation’s. Make sure that you have sbt-java-agent plugin in your project/plugins.sbt:

After, add Kanela instrumentation agent to build.sbt Using sbt-native-packager, so it can enable the agent at runtime.

Metrics

Kamon provides several metrics in its API. To track number of tickets we can use Gauge metric:

Kamon has tags to provide additional information about metric, similarly to OpenTelemetry’s Java API Attributes. So to track the number of created tickets per project we can use the following code:

Metrics example: Prometheus

Kamon provides metrics-specific integrations for Prometheus and InfluxDB. Here we will use Prometheus to monitor the metrics of our application.

First, add to exporter dependency to build.sbt:

Then we need to configure it in application.conf

Let’s start the local Prometheus instance using docker-compose (partial example):

Don’t forget to expose 9094 port (or specific for your needs) for tickets-service for Prometheus agent to scrap metrics. Let's start the whole setup and run Gatling tests after. On Prometheus UI at localhost:9090 we can find tickets_count metric:

As you can see, we have project_id tag for tickets_count metric, which is added by our application.

Tracing example: Zipkin

Kamon provides tracing-specific integrations for Zipkin and Jaeger. Here we will use Zipkin to monitor traces of our application.

As in the previous example, firstly we need to add dependency to build.sbt:

Then we need to configure exporter in application.conf

Last but not least, we need to enable Akka-HTTP tracing in application.conf:

Let’s start local Zipkin instance using docker-compose (partial example):

Let’s start the whole setup and create ticket with POST /tickets request. Among response headers, you can find the value trace-id - 1b05566f51be20ad which is a trace id of our request. On Zipkin UI at localhost:9411 we can find our trace:

This trace includes all requests made by our application, including requests to projects-service, queries to Postgres etc.

APM Example: Kamon APM

Integrations with Datadog and New Relic are also supported by Kamon. But, among instrumentation’s and libraries, Kamon also provides a commercial product called Kamon APM. Kamon APM is a hosted commercial solution, which has native integration with Kamon library.

To plug Kamon APM reporter you need to add the following dependency

The configuration looks pretty simple, all you need to do is to provide your API key:

That’s it — you’re ready to go. Let’s start tickets_service and Gatling tests after. You can find application metrics in Kamon APM dashboard immediately:

The great thing about Kamon APM is that it has pre-configured dashboards for JVM, JDBC, and Akka-specific metrics.

Additionally, tracing data is also available in a handy, such as finding trace by trace-id header value:

You can find more about Kamon APM here

Conclusion

In this blog post, we explored how to monitor a Scala-based backend application using Kamon framework. It is a bit hard to choose between OpenTelemetry and Kamon as telemetry solutions for Scala & Lightbend stack, because they provide similar functionality in a very convenient way, such as automatic instrumentation. Both OpenTelemetry and Kamon provide very similar metrics APIs (Kamon metrics and OTEL traces) along with tracing (Kamon traces and OTEL traces). Nonetheless, some pros and cons can be listed.

Kamon does not provide a mechanism for logs sending to external services like OpenTelemetry does with OTLP Protocol. But, there are plenty of ways to do this with other tools.
The list of OpenTelemetry’s JVM instrumentation’s is certainly bigger than of Kamon’s. Another pitfall of Kamon is the limited number of supported reporters. While OpenTelemetry is vendor-agnostic by design.
But, Kamon provides better native Scala API for creating custom Traces and Spans for instance for Futures and for Akka actors.

If you are using multiple languages across your organization, or you have an existing complex monitoring solution, then it’s probably better to stick with OpenTelemetry, as it’s language and vendor agnostic. But, if Scala and Akka ecosystem is your primary choice, then Kamon might be a better fit for you.

That’s it for this time. All the code is available on Github. In the next post, we will talk about Trace4Cats in TypeLevel stack-based projects.