Monitoring DevOps Style With WildFly 9 And Jolokia

DevOps is among the hottest topic these days. And the wide range of topics around it makes it hard to actually find a complete description or something that covers everything on a decent granularity. One thing is for sure: One of the most important parts is to deliver the correct metrics and and information for monitoring of the application.

Java EE and JMX

The standard way of monitoring Java EE servers is JMX. This is possible with tools like JConsole, VisualVM or the Oracle Mission-Control Suite. There are a bunch of advantages to this approach and most of the operation teams actually used this a lot in the past. But it doesn't exactly works the DevOps-way. It is a separate tooling and the DevOps-teams don't have a good way to actually script this without having all the tooling and operational systems (Nagios, etc.) installed. Today it feels a lot more natural and is easier to use to have http endpoints which expose configuration and runtime information.

Jolokia - JMX To HTTP With JSON
A very convenient way to do this for JMX is to use Jolokia. Jolokia is a JMX-HTTP bridge giving an alternative to JSR-160 connectors. It is an agent based approach with support for many platforms. In addition to basic JMX operations it enhances JMX remoting with unique features like bulk requests and fine grained security policies. It comes bundled with a lot of JBoss projects lately (e.g. WIldFly-Camel subsystem) and can be easily used in your own applications.

A Simple Java EE 7 App Equipped With Jolokia
Just create a simple Java EE 7 project (maybe with Adam Bien's maven artifact) and add one dependency to it:

<dependency>
     <groupId>org.jolokia</groupId>
     <artifactId>jolokia-core</artifactId>
     <version>1.3.1</version>
 </dependency>

The next step is to configure the Jolokia AgentServlet in your web.xml and map it to a pattern which suits your needs:

  <servlet>
        <servlet-name>jolokia-agent</servlet-name>
        <servlet-class>org.jolokia.http.AgentServlet</servlet-class>
        <load-on-startup>1</load-on-startup>
    </servlet>

    <servlet-mapping>
        <servlet-name>jolokia-agent</servlet-name>
        <url-pattern>/metrics/*</url-pattern>
    </servlet-mapping>

Build your application as usual and access the relevant metrics as you need them. The complete .Jolokia reference explains the different operations and types.

Deploy Your Application To WildFly 9
Download and unzip WildFly 9 to a folder of your choice. Startup with bin/standalone.xml.

Example Metrics
While you can access every JMX MBean, that is defined in the server, here is a list of metrics, that might help you out of the box.

Heap memory usage:
http://localhost:8080/javaee-devops/metrics/read/java.lang:type=Memory/HeapMemoryUsage

{
    "request": {
        "mbean": "java.lang:type=Memory",
        "attribute": "HeapMemoryUsage",
        "type": "read"
    },
    "value": {
        "init": 67108864,
        "committed": 241696768,
        "max": 477626368,
        "used": 141716336
    },
    "timestamp": 1437392335,
    "status": 200
}

Overview over your server environment:
http://localhost:8080/javaee-devops/metrics/read/jboss.as:core-service=server-environment

You could not only read JMX attributes but also execute operations, like accessing the latest 10 lines of the server.log file:
http://localhost:8080/javaee-devops/metrics/exec/jboss.as.expr:subsystem=logging/readLogFile/server.log/UTF-8/10/0/true

Securing The Endpoint
As you would have expected, the AgentServlet is accessible like your application is. In order to prevent this, you will have to secure it. Good news is, that this is possible with basic authentication and the application realm in WildFly. Fist step is to add a user to the application realm. This can be done with the bin/add-user.sh|bat script. Make sure to add the role "SuperUser". Now add the following to your web.xml:

    <security-constraint>
        <display-name>Metrics Pages</display-name>
        <web-resource-collection>
            <web-resource-name>Protected Metrics Site</web-resource-name>
            <description>Protected Metrics Site</description>
            <url-pattern>/metrics/*</url-pattern>
        </web-resource-collection>
        <auth-constraint>
            <description/>
            <role-name>SuperUser</role-name>
        </auth-constraint>
        <user-data-constraint>
            <transport-guarantee>NONE</transport-guarantee>
        </user-data-constraint>
    </security-constraint>
     <login-config>
        <auth-method>BASIC</auth-method>
        <realm-name>ApplicationRealm</realm-name>
    </login-config>
    <security-role> 
        <role-name>SuperUser</role-name> 
    </security-role> 

One last thing to do here is to add a file to WEB-INF/ called jboss-web.xml. This will just contain three lines:

<jboss-web>
    <security-domain>other</security-domain>
</jboss-web>

Whenever you try to access the metrics endpoint the server now challenges you with a basic authentication request.

Looking For More?
This is just a simple example for now based on the standard JMX metrics, which WildFly exposes. You can for sure register your own MBeans or expand this by aggregating the individual calls into one single. Another option is, to use hawt.io as a ready to use, extensible UI which already provides all kinds of metrics for WildFly and many other subsystems. But this is a very straight forward way. Next major release of Jolokia might feature some more to make the DevOps ride a lot more convenient.