DataDog and JMX

External

• http://docs.datadoghq.com/integrations/java/

Internal

• DataDog

Configuration

The JMX configuration is located in /etc/dd-agent/conf.d/jmx.yaml.example.

After modification and removal of the ".example" suffix, restart datadog as root:

service datadog-agent restart

JMX integration with JBoss

Server Preparation

JBoss instances have to be configured to allow JMX agent access. Details on how to configure JBoss various JBoss versions running in various modes are available here:

WildFly and JMX

Agent Classpath and Startup Options

Working jmx.yaml configuration:

init_config:
    custom_jar_paths:
      - /opt/jdg/bin/client/jboss-cli-client.jar

instances:
    - jmx_url: "service:jmx:remote://1.2.3.4:4447"
      user: jmx
      password: ***
      java_bin_path: /opt/jdk/bin/java
      name: jdg1
      tags:
        env: stage
        newTag: test
      conf:
        - include:
            domain: jboss.datagrid-infinispan
            bean:
              - type=Cache,name=gameStateCache(dist_sync),manager=clustered,component=Statistics
            attribute:
              attribute1:
                metric_type: counter
                alias: numberOfEntries

Make sure the JMX user has been configured on target JBoss instances, for symptoms see "JMX User Has not Been Configured on the Target JBoss Instance" section.

Connect to the JBoss Instance

Use the following URLs:

• For Standalone EAP 6
• For EAP 6 Host Controller
• For EAP 6 Domain Mode Server Node
• For Standalone EAP 7
• For EAP 7 Host Controller
• For EAP 7 Domain Mode Server Node

Troubleshooting

Agent Process Signature

ps -ef | grep java | grep datadog

dd-agent 14889 14841  0 Oct15 ? 00:05:47 \
  /opt/jdk/bin/java -Xms50m -Xmx200m \
  -classpath /opt/jdk/lib/tools.jar:/opt/datadog-agent/agent/checks/libs/jmxfetch-0.11.0-jar-with-dependencies.jar org.datadog.jmxfetch.App \
  --check jmx.yaml --check_period 15000 --conf_directory /etc/dd-agent/conf.d --log_level INFO \
  --log_location /var/log/datadog/jmxfetch.log --reporter statsd:localhost:8125 
  --status_location /opt/datadog-agent/run/jmx_status.yaml collect

Agent Reinitialization

service datadog-agent restart

Metric List

sudo /etc/init.d/datadog-agent jmx list_not_matching_attributes

Errors

JMX User Has Not Been Configured on the target JBoss Instance

2016-10-19 15:34:53,433 | ERROR| App | Cannot connect to instance service:jmx:remoting-jmx://1.2.3.4:4447 javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed:
   DIGEST-MD5: Server rejected authentication

TODO

2016-10-19 16:20:43,683 | INFO | Connection | Connecting to: service:jmx:remote://1.2.3.4:4447
2016-10-19 16:20:43,731 | INFO | xnio | XNIO version 3.3.6.Final-redhat-1
2016-10-19 16:20:43,777 | INFO | nio | XNIO NIO Implementation Version 3.3.6.Final-redhat-1
2016-10-19 16:20:43,970 | INFO | remoting | JBoss Remoting version 4.0.18.Final-redhat-1
2016-10-19 16:20:44,663 | INFO | Instance | Connected to JMX Server at service:jmx:remote://1.2.3.4:4447
2016-10-19 16:20:44,664 | ERROR| Instance | Unable to compute a common bean scope, querying all beans as a fallback
java.lang.ArrayIndexOutOfBoundsException: 1
	at org.datadog.jmxfetch.Configuration.getIncludeFiltersByDomain(Configuration.java:89)
	at org.datadog.jmxfetch.Configuration.getGreatestCommonScopes(Configuration.java:227)
	at org.datadog.jmxfetch.Instance.getBeansScopes(Instance.java:284)
	at org.datadog.jmxfetch.Instance.refreshBeansList(Instance.java:301)
	at org.datadog.jmxfetch.Instance.init(Instance.java:132)
	at org.datadog.jmxfetch.App.init(App.java:354)
	at org.datadog.jmxfetch.App.main(App.java:92)