DataDog and JMX

External

• http://docs.datadoghq.com/integrations/java/

Internal

• Datadog

Concepts

JMX Attribute

JMX Attribute Alias

If an alias is specified, it will become the metric name when displayed by the DataDog UI. Camel case will be converted to snake case. Dots are acceptable.

JMX Attribute Type

The JMX attribute type is specified as "metric_type" in configuration.

Gauge

The actual value. The agent does not attempt to calculate the rate, as in the counter's case.

Counter

For a "counter", the rate per second will be computed for this metric.

Configuration

The JMX configuration is located in /etc/dd-agent/conf.d/jmx.yaml.example.

After modification and removal of the ".example" suffix, the agent has to be restarted as shown here.

JMX integration with JBoss

Server Preparation

JBoss instances have to be configured to allow JMX agent access. Details on how to configure JBoss various JBoss versions running in various modes are available here:

WildFly and JMX

Agent Classpath and Startup Options

Working jmx.yaml configuration:

init_config:
    custom_jar_paths:
      - /opt/jdg/bin/client/jboss-cli-client.jar

instances:
    - jmx_url: "service:jmx:remote://1.2.3.4:4447"
      user: jmx
      password: ***
      java_bin_path: /opt/jdk/bin/java
      name: jdg1
      tags:
        env: stage
        newTag: test
      conf:
        [...]

The specific value for "jmx_url" depends on the JBoss version and the mode it is running in. More details can be found here:

• For Standalone EAP 6
• For EAP 6 Host Controller
• For EAP 6 Domain Mode Server Node
• For Standalone EAP 7
• For EAP 7 Host Controller
• For EAP 7 Domain Mode Server Node

Make sure the JMX user has been configured on target JBoss instances, for symptoms see "JMX User Has not Been Configured on the Target JBoss Instance" section.

JMX Metric Specification

The MBeans are grouped per their JMX domains. Each attribute is specified by name, with an alias and its type.

Configuration example:

init_config:
    [...]
instances:
    - jmx_url: [...]
      [...]
      conf:
        - include:
            domain: jboss.as
            bean:
               - jboss.as:subsystem=datagrid-infinispan-endpoint,hotrod-connector=hotrod-connector
               - jboss.as:subsystem=datagrid-infinispan,cache-container=clustered
            attribute:
               bytesRead:
                 alias: hotrod_endpoint_read_bytes_per_second
                 metric_type: counter
               bytesWritten:
                 alias: hotrod_endpoint_written_bytes_per_second
                 metric_type: counter
               hits:
                 alias: node_read_hits
                 metric_type: counter
               misses:
                 alias: node_read_misses
                 metric_type: counter
               stores:
                 alias: node_writes
                 metric_type: counter
               averageReadTime:
                 alias: node_average_read_time
                 metric_type: gauge
               averageWriteTime:
                 alias: node_average_write_time
                 metric_type: gauge
               averageRemoveTime:
                 alias: node_average_remove_time
                 metric_type: gauge
               numberOfEntries:
                 alias: node_cache_entries
                 metric_type: gauge

Troubleshooting

Agent Process Signature

ps -ef | grep java | grep datadog

dd-agent 14889 14841  0 Oct15 ? 00:05:47 \
  /opt/jdk/bin/java -Xms50m -Xmx200m \
  -classpath /opt/jdk/lib/tools.jar:/opt/datadog-agent/agent/checks/libs/jmxfetch-0.11.0-jar-with-dependencies.jar org.datadog.jmxfetch.App \
  --check jmx.yaml --check_period 15000 --conf_directory /etc/dd-agent/conf.d --log_level INFO \
  --log_location /var/log/datadog/jmxfetch.log --reporter statsd:localhost:8125 
  --status_location /opt/datadog-agent/run/jmx_status.yaml collect

Agent Reinitialization

service datadog-agent restart

Metric List

sudo /etc/init.d/datadog-agent jmx list_not_matching_attributes

Errors

JMX User Has Not Been Configured on the target JBoss Instance

2016-10-19 15:34:53,433 | ERROR| App | Cannot connect to instance service:jmx:remoting-jmx://1.2.3.4:4447 javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed:
   DIGEST-MD5: Server rejected authentication