DataDog and JMX: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
No edit summary
 
(28 intermediate revisions by the same user not shown)
Line 5: Line 5:
=Internal=
=Internal=


* [[DataDog]]
* [[Datadog]]
 
=Concepts=
 
==JMX Attribute==
 
===JMX Attribute Alias===
 
If an alias is specified, it will become the metric name when displayed by the DataDog UI. Camel case will be converted to snake case. Dots are acceptable.
 
===JMX Attribute Type===
 
The JMX attribute type is specified as "metric_type" in configuration.
 
====Gauge====
 
The actual value. The agent does not attempt to calculate the rate, as in the [[#Counter|counter's]] case.
 
====Counter====
 
For a "counter", the rate per second will be computed for this metric.


=Configuration=
=Configuration=
Line 11: Line 31:
The JMX configuration is located in /etc/dd-agent/conf.d/jmx.yaml.example.  
The JMX configuration is located in /etc/dd-agent/conf.d/jmx.yaml.example.  


After modification and removal of the ".example" suffix, restart datadog as root:
After modification and removal of the ".example" suffix, the agent has to be restarted [[#Agent_Reinitialization|as shown here]].
 
<pre>
service datadog-agent restart
</pre>


=JMX integration with JBoss=
=JMX integration with JBoss=
Line 29: Line 45:
==Agent Classpath and Startup Options==
==Agent Classpath and Startup Options==


The agent must include the following classpath:
Working jmx.yaml configuration:


<pre>
<pre>
-Djava.class.path=${JAVA_HOME}/lib/jconsole.jar:${JAVA_HOME}/lib/tools.jar:${JBOSS_HOME}/bin/client/jboss-cli-client.jar
init_config:
    custom_jar_paths:
      - /opt/jdg/bin/client/jboss-cli-client.jar
 
instances:
    - jmx_url: "service:jmx:remote://1.2.3.4:4447"
      user: jmx
      password: ***
      java_bin_path: /opt/jdk/bin/java
      name: jdg1
      tags:
        env: stage
        newTag: test
      conf:
        [...]
</pre>
</pre>


The agent must also receive the following system property on command line:
The specific value for "jmx_url" depends on the JBoss version and the mode it is running in. More details can be found here:
* [[JMX_Access_to_Standalone_EAP_6#URL|For Standalone EAP 6]]
* [[JMX_Access_to_EAP_6_Host_Controller#URL|For EAP 6 Host Controller]]
* [[JMX_Access_to_Domain_Mode_EAP_6_Server_Node#URL|For EAP 6 Domain Mode Server Node]]
* [[JMX_Access_To_Standalone_EAP_7#URL|For Standalone EAP 7]]
* [[JMX_Access_to_EAP_7_Host_Controller#URL|For EAP 7 Host Controller]]
* [[JMX_Access_to_Domain_Mode_EAP_7_Server_Node#URL|For EAP 7 Domain Mode Server Node]]
 
<blockquote style="background-color: Gold; border: solid thin Goldenrod;">
:<br>Make sure the JMX user has been configured on target JBoss instances, for symptoms see "[[#JMX_User_Has_Not_Been_Configured_on_the_target_JBoss_Instance|JMX User Has not Been Configured on the Target JBoss Instance]]" section.<br><br>
</blockquote>
 
==JMX Metric Specification==
 
The MBeans are grouped per their JMX domains. Each attribute is specified by name, with an [[#JMX_Attribute_Alias|alias]] and its [[#JMX_Attribute_Type|type]].
 
Configuration example:
 
<pre>
init_config:
    [...]
instances:
    - jmx_url: [...]
      [...]
      conf:
        - include:
            domain: jboss.as
            bean:
              - jboss.as:subsystem=datagrid-infinispan-endpoint,hotrod-connector=hotrod-connector
              - jboss.as:subsystem=datagrid-infinispan,cache-container=clustered
            attribute:
              bytesRead:
                alias: hotrod_endpoint_read_bytes_per_second
                metric_type: counter
              bytesWritten:
                alias: hotrod_endpoint_written_bytes_per_second
                metric_type: counter
              hits:
                alias: node_read_hits
                metric_type: counter
              misses:
                alias: node_read_misses
                metric_type: counter
              stores:
                alias: node_writes
                metric_type: counter
              averageReadTime:
                alias: node_average_read_time
                metric_type: gauge
              averageWriteTime:
                alias: node_average_write_time
                metric_type: gauge
              averageRemoveTime:
                alias: node_average_remove_time
                metric_type: gauge
              numberOfEntries:
                alias: node_cache_entries
                metric_type: gauge
</pre>
 
=Troubleshooting=
 
==Agent Process Signature==
 
<pre>
ps -ef | grep java | grep datadog
 
dd-agent 14889 14841  0 Oct15 ? 00:05:47 \
  /opt/jdk/bin/java -Xms50m -Xmx200m \
  -classpath /opt/jdk/lib/tools.jar:/opt/datadog-agent/agent/checks/libs/jmxfetch-0.11.0-jar-with-dependencies.jar org.datadog.jmxfetch.App \
  --check jmx.yaml --check_period 15000 --conf_directory /etc/dd-agent/conf.d --log_level INFO \
  --log_location /var/log/datadog/jmxfetch.log --reporter statsd:localhost:8125
  --status_location /opt/datadog-agent/run/jmx_status.yaml collect
</pre>
 
==Agent Reinitialization==


<pre>
<pre>
-Dmodule.path=${JBOSS_HOME}/modules
service datadog-agent restart
</pre>
</pre>


The full command line is:
==Metric List==


<pre>
<pre>
jconsole \
sudo /etc/init.d/datadog-agent jmx list_not_matching_attributes
-J-Djava.class.path=${JAVA_HOME}/lib/jconsole.jar:${JAVA_HOME}/lib/tools.jar:${JBOSS_HOME}/bin/client/jboss-cli-client.jar \
-J-Dmodule.path=${JBOSS_HOME}/modules
</pre>
</pre>


==Connect to the JBoss Instance==
==Errors==


Use the following URLs:
===JMX User Has Not Been Configured on the target JBoss Instance===


* [[JMX_Access_to_Standalone_EAP_6#URL|For Standalone EAP 6]]
<pre>
* [[JMX_Access_to_EAP_6_Host_Controller#URL|For EAP 6 Host Controller]]
2016-10-19 15:34:53,433 | ERROR| App | Cannot connect to instance service:jmx:remoting-jmx://1.2.3.4:4447 javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed:
* [[JMX_Access_to_Domain_Mode_EAP_6_Server_Node#URL|For EAP 6 Domain Mode Server Node]]
  DIGEST-MD5: Server rejected authentication
* [[JMX_Access_To_Standalone_EAP_7#URL|For Standalone EAP 7]]
</pre>
* [[JMX_Access_to_EAP_7_Host_Controller#URL|For EAP 7 Host Controller]]
* [[JMX_Access_to_Domain_Mode_EAP_7_Server_Node#URL|For EAP 7 Domain Mode Server Node]]

Latest revision as of 05:30, 14 January 2022

External

Internal

Concepts

JMX Attribute

JMX Attribute Alias

If an alias is specified, it will become the metric name when displayed by the DataDog UI. Camel case will be converted to snake case. Dots are acceptable.

JMX Attribute Type

The JMX attribute type is specified as "metric_type" in configuration.

Gauge

The actual value. The agent does not attempt to calculate the rate, as in the counter's case.

Counter

For a "counter", the rate per second will be computed for this metric.

Configuration

The JMX configuration is located in /etc/dd-agent/conf.d/jmx.yaml.example.

After modification and removal of the ".example" suffix, the agent has to be restarted as shown here.

JMX integration with JBoss

Server Preparation

JBoss instances have to be configured to allow JMX agent access. Details on how to configure JBoss various JBoss versions running in various modes are available here:

WildFly and JMX

Agent Classpath and Startup Options

Working jmx.yaml configuration: 

init_config:
    custom_jar_paths:
      - /opt/jdg/bin/client/jboss-cli-client.jar

instances:
    - jmx_url: "service:jmx:remote://1.2.3.4:4447"
      user: jmx
      password: ***
      java_bin_path: /opt/jdk/bin/java
      name: jdg1
      tags:
        env: stage
        newTag: test
      conf:
        [...]

The specific value for "jmx_url" depends on the JBoss version and the mode it is running in. More details can be found here:


Make sure the JMX user has been configured on target JBoss instances, for symptoms see "JMX User Has not Been Configured on the Target JBoss Instance" section.

JMX Metric Specification

The MBeans are grouped per their JMX domains. Each attribute is specified by name, with an alias and its type.

Configuration example: 

init_config:
    [...]
instances:
    - jmx_url: [...]
      [...]
      conf:
        - include:
            domain: jboss.as
            bean:
               - jboss.as:subsystem=datagrid-infinispan-endpoint,hotrod-connector=hotrod-connector
               - jboss.as:subsystem=datagrid-infinispan,cache-container=clustered
            attribute:
               bytesRead:
                 alias: hotrod_endpoint_read_bytes_per_second
                 metric_type: counter
               bytesWritten:
                 alias: hotrod_endpoint_written_bytes_per_second
                 metric_type: counter
               hits:
                 alias: node_read_hits
                 metric_type: counter
               misses:
                 alias: node_read_misses
                 metric_type: counter
               stores:
                 alias: node_writes
                 metric_type: counter
               averageReadTime:
                 alias: node_average_read_time
                 metric_type: gauge
               averageWriteTime:
                 alias: node_average_write_time
                 metric_type: gauge
               averageRemoveTime:
                 alias: node_average_remove_time
                 metric_type: gauge
               numberOfEntries:
                 alias: node_cache_entries
                 metric_type: gauge

Troubleshooting

Agent Process Signature

ps -ef | grep java | grep datadog

dd-agent 14889 14841  0 Oct15 ? 00:05:47 \
  /opt/jdk/bin/java -Xms50m -Xmx200m \
  -classpath /opt/jdk/lib/tools.jar:/opt/datadog-agent/agent/checks/libs/jmxfetch-0.11.0-jar-with-dependencies.jar org.datadog.jmxfetch.App \
  --check jmx.yaml --check_period 15000 --conf_directory /etc/dd-agent/conf.d --log_level INFO \
  --log_location /var/log/datadog/jmxfetch.log --reporter statsd:localhost:8125 
  --status_location /opt/datadog-agent/run/jmx_status.yaml collect

Agent Reinitialization

service datadog-agent restart

Metric List

sudo /etc/init.d/datadog-agent jmx list_not_matching_attributes

Errors

JMX User Has Not Been Configured on the target JBoss Instance

2016-10-19 15:34:53,433 | ERROR| App | Cannot connect to instance service:jmx:remoting-jmx://1.2.3.4:4447 javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed:
   DIGEST-MD5: Server rejected authentication
Retrieved from "https://kb.novaordis.com/index.php?title=DataDog_and_JMX&oldid=89143"