microservices metrics icon

Microservices with Spring Cloud : Metrics and Monitoring

We’re nearing the end of our series about microservices with Spring Cloud. In the last post, we’ve seen how to implement logging in microservices. Today, we’ll cover metrics and monitoring.

Why do we need metrics?

After your service is developed and tested, the most important step remains: deploying it to the real world. This step can often be quite the opposite of your expectations.

Imagine this scenario: you’ve tested your service extensively, everything works great and you proudly deploy it to production. And then, all hell breaks loose!! As soon as users start hammering your service with requests, it starts falling apart. You start getting timeouts, slow queries, internal server errors, and all kinds of weird stuff. How are you going to figure out what is going on?

In the previous post, we talked about logging in microservices. If you’ve implemented that part, it can give you some insight about what is going on. But, it can’t tell you much about memory consumption, CPU load, concurrent requests, timing and other performance related stuff. For this, we need metrics.

Metrics can also help us to anticipate problems and prevent them from happening. For example, if you notice that time required to process certain requests grows over time, you can take steps to correct prevent timeouts. This can include adding more instances to reduce the load, optimizing database connections etc.

One more use case for metrics is that they enable use of alerting tools like PagerDuty. You can configure threshold values for some metrics in your services, and when metric values are close or over threshold, you will get an alert. There are a lot more similar tools available, but that is a topic for another post.

Spring Cloud metrics support

Spring Cloud is based on Spring Boot, which means it comes with out of the box support for Actuator metrics. Actuator is an add-on module which provides endpoints for application health checks and performance metrics. It is extremely easy to integrate and extend when used in Spring-based applications. In addition, it provides an easy way to implement custom metric types. The limitation of Actuator is that it comes with only 2 metric types:

  • gauge – represents a single value
  • counter – represents a difference (increment or decrement)

If you need to use more feature-rich set of metrics, you can use Dropwizard metrics library for microservices. Dropwizard is another popular framework for microservices development, and it’s metrics library was separated in it’s own module, so it can be reused without dependency on complete framework. Spring Bot can support Dropwizard metrics, and we’ll see how it is configured later in the article.

Spring Boot also provides a way to export metrics to tools like Graphite or Grafana, which allow visual representation of results. Exporting metrics works with both Actuator and Dropwizard metrics. Further in this post, we will see how to export collected metrics to one of these tools.

Adding Actuator metrics

Now it’s time to get dirty and implement metrics support in our code. We’ll start with Actuator metrics, since they should work automatically. We’ve already added required dependency spring-boot-starter-actuator when we first created our services. If it’s not there, just add the following code snippet to service POM:

By default, metrics endpoints are protected and require authentication. For our example, we won’t need this, and so we’ll turn of authentication. Simply add this code snipper to application.properties  file of each service:

management.security.enabled=false

Now rebuild the services and start complete system (Eureka + Config server + Zuul + services). If you invoke user service at http://localhost:8800/health  endpoint, you should get an output like this:

This is the default output of health endpoint that says instance is up. You can add your own health check implementation which perform custom operations to determine instance health, but we won’t cover it in this topic. Now, invoke metrics endpoint at http://localhost:8800/metrics . You should get a large output similar to this:

This is just a sample of the output, but you get the idea. Here you can see information about memory consumption and system load. Many more metrics are available. If you now invoke user service through Zuul (by sending POST to http://localhost:9000/api/v1/users ), and call metrics endpoint again, you will see that it now contains a lot more data. This is because datapoints of HTTP request were added to metrics.

Implement metrics for services

As we’ve already mentioned, Actuator has two types of metrics: counter and gauge. Counter is what the name implies, it can be incremented, decremented or reset as needed. Gauge returns an arbitrary value, You can think of it as a speed meter on your dashboard: whenever you look at it, it shows you the speed.

We’ll add both types of meters to our service, to see how they work. We’ll add a counter to user service, and gauge to messaging service. For counter, we’ll make the following changes to UserController :

Here, we autowire the reference to CounterService, which comes with Actuator. Then, we call counterService.increment("counter.user.create") . This will create a counter called counter.user.create , and increment it each time we call user controller.

Now comes important twist: Spring Cloud comes with support for Netflix Servo out of the box. This means that Actuator will automatically prepend counter.servo  to the name of your metric. To avoid this, add the following line to application.properties:

We’ll need to add the same property to messaging service as well. To add gauge metric to messaging service,  make the following changes to MessageController:

Just like in previous case, we add reference to GaugeService  which comes with Actuator. Then, we invoke gauge.submit("msg.gauge.value", Math.random() * 100) . This will create a metric called msg.gauge.value  and return random value each time controller is called.

Putting it to the test

Now is the time to test our metrics! Startup whole system and invoke Zuul at http://localhost:9000/api/v1/users . This will cause both controllers to be invoked. You can confirm by looking at the logs. Call this endpoint several times, so the counter can be incremented. Now, if you call the metrics endpoint of user service, you should see the value of counter.user.create.

This snapshot shows sample output on my machine:

spring cloud metrics data

If you call the metrics endpoint on messaging service, you should find a value gauge.msg.gauge.value . You will notice that Spring Cloud will automatically prepend gauge to the name of your metric.

Moving metrics to interceptor

In any real world application, you wouldn’t want to mix business logic with collecting metrics. It’s distracting for anyone reading your code and adds noise to he business flow. Fortunately, Spring provides several options to avoid this, like aspects, interceptors or filters.

We will use interceptor to collect metrics in after the request has been completed. In order to invoke metrics collection, interceptor will be registered for our endpoints and invoked each time request is sent to these endpoint. We will also need additional configuration classes in order to register interceptor. Interceptor for user service will look like this:

This is exactly the same logic we previously had in controller, but it is now moved to separate component. Make sure to remove call to counter from the controller, otherwise you will get duplicate values.

Configuration class will look like this:

This is there is to it. Now rebuild the service and invoke Zuul endpoint again. After that, check the metrics and you should see the counter incremented correctly. We now have the clean code and working metric.

The same logic applies to messaging service. I won’t write the code here, because it is basically the same. You can check out complete code on Github.

Adding Dropwizard Metrics

In addition to Actuator, Spring Cloud support using Dropwizard metrics. Dropwizard adds more metric types, such as histograms, meters and timers. As a result, it is sometimes preferred in production compared to Actuator. In this section, we will include Dropwizard metrics and see how they can be used with our microservices.

First we need to add required dependency. We will use Ryan Tenney’s Spring metrics library, because it provides bindings for seamless integration of Spring and Dropwizard. Add the following to user service POM:

We will need to add another configuration class to the service in order to configure Dropwiard metrics. Create the following class:

Attribute proxyTargetClass means that CGLIB will be used to create required proxy object, instead of regular JDK proxies. I’m not sure what this does, but things will not work without this attribute.

The final touch is to annotate endpoints we are interested in with metrics annotations. For example, we’ll add @Timed  annotations to our (forgotten) greeting endpoint. This will cause endpoint latency information to be recorded as metrics. The only needed change for this is to annotate the method:

Now, if you invoke /greet endpoint , and view the metrics, you should see something like this:

Dropwizard metrics example value

You can notice that metrics provide values like minimum, maximum, median, mean, percentiles, rates and so on. Thus, it’s a plethora of information from one single annotation. The only eyesore in this is $$EnhanceBySpringCGLIB$$  stuff, which is automatically added by CGLIB. I have not figured out how to remove them, so if anyone knows how to do it, please comment on this post.

Exporting metrics to Graphite

Looking at these JSON files which represent metrics gets painful after few minutes. Fortunately, good people around the world have come up with a lot of tools to visualize metrics data. One of widely used tools for such tasks is Graphite. In this section, we will export our metrics to Graphite instance, so we can visualize them.

In order to avoid setting up Graphite locally, we’ll use ready-made Docker image. You’ll need to have Docker installed to complete this part of procedure. I won’t cover installation of Docker here, since you can find a lot of resource online. Assuming that you have Docker installed, simply run the following command:

Here, option /path/to/local/dir  is the path to local directory where metrics will be stored, so Graphite can import them. We now need to configure Graphite reporter. In order to do that, we’ll update  DropwizardMetricsConfig  class:

In this snippet, we configures sender for graphite host and port, which is where Docker container runs. It would be a good idea to externalize this configuration, but for tutorial purposes it should be fine. Finally, we configure reporter with some additional options and set that metrics are sent to Graphite every 10 seconds.

Now it’s time to test the whole thing! If you visit http://localhost:8080 , you should see Graphite dashboard. If you start the system and invoke endpoints a couple of times, you should see Graphite dashboard update to display metrics. It might take some time for them to show up, but eventually you should get something similar to the sample bellow:

grpahite spring dropwizard metrics

Conclusion

This was somewhat exhausting post, but we covered a lot of stuff. You have a complete working solution to monitor microservices system. You can find the complete source code on Github. If you feel like it, you can build it yourself and play around with it. In case you missed some of the previous posts, bellow is the list of all the posts in this series.

If you have any questions, remarks or suggestions, please don’t hesitate to comment on this post and voice your opinion.

Complete lists of posts in this series

 

Leave a Comment

Your email address will not be published. Required fields are marked *