DataDog and JMX: Difference between revisions

From NovaOrdis Knowledge Base
Jump to navigation Jump to search
No edit summary
 
(17 intermediate revisions by the same user not shown)
Line 5: Line 5:
=Internal=
=Internal=


* [[DataDog]]
* [[Datadog]]
 
=Concepts=
 
==JMX Attribute==
 
===JMX Attribute Alias===
 
If an alias is specified, it will become the metric name when displayed by the DataDog UI. Camel case will be converted to snake case. Dots are acceptable.
 
===JMX Attribute Type===
 
The JMX attribute type is specified as "metric_type" in configuration.
 
====Gauge====
 
The actual value. The agent does not attempt to calculate the rate, as in the [[#Counter|counter's]] case.
 
====Counter====
 
For a "counter", the rate per second will be computed for this metric.


=Configuration=
=Configuration=
Line 11: Line 31:
The JMX configuration is located in /etc/dd-agent/conf.d/jmx.yaml.example.  
The JMX configuration is located in /etc/dd-agent/conf.d/jmx.yaml.example.  


After modification and removal of the ".example" suffix, restart datadog as root:
After modification and removal of the ".example" suffix, the agent has to be restarted [[#Agent_Reinitialization|as shown here]].
 
<pre>
service datadog-agent restart
</pre>


=JMX integration with JBoss=
=JMX integration with JBoss=
Line 46: Line 62:
         newTag: test
         newTag: test
       conf:
       conf:
         - include:
         [...]
            domain: jboss.datagrid-infinispan
            bean:
              - type=Cache,name=gameStateCache(dist_sync),manager=clustered,component=Statistics
            attribute:
              attribute1:
                metric_type: counter
                alias: numberOfEntries
</pre>
</pre>
The specific value for "jmx_url" depends on the JBoss version and the mode it is running in. More details can be found here:
* [[JMX_Access_to_Standalone_EAP_6#URL|For Standalone EAP 6]]
* [[JMX_Access_to_EAP_6_Host_Controller#URL|For EAP 6 Host Controller]]
* [[JMX_Access_to_Domain_Mode_EAP_6_Server_Node#URL|For EAP 6 Domain Mode Server Node]]
* [[JMX_Access_To_Standalone_EAP_7#URL|For Standalone EAP 7]]
* [[JMX_Access_to_EAP_7_Host_Controller#URL|For EAP 7 Host Controller]]
* [[JMX_Access_to_Domain_Mode_EAP_7_Server_Node#URL|For EAP 7 Domain Mode Server Node]]


<blockquote style="background-color: Gold; border: solid thin Goldenrod;">
<blockquote style="background-color: Gold; border: solid thin Goldenrod;">
Line 60: Line 77:
</blockquote>
</blockquote>


==Connect to the JBoss Instance==
==JMX Metric Specification==


Use the following URLs:
The MBeans are grouped per their JMX domains. Each attribute is specified by name, with an [[#JMX_Attribute_Alias|alias]] and its [[#JMX_Attribute_Type|type]].


* [[JMX_Access_to_Standalone_EAP_6#URL|For Standalone EAP 6]]
Configuration example:
* [[JMX_Access_to_EAP_6_Host_Controller#URL|For EAP 6 Host Controller]]
 
* [[JMX_Access_to_Domain_Mode_EAP_6_Server_Node#URL|For EAP 6 Domain Mode Server Node]]
<pre>
* [[JMX_Access_To_Standalone_EAP_7#URL|For Standalone EAP 7]]
init_config:
* [[JMX_Access_to_EAP_7_Host_Controller#URL|For EAP 7 Host Controller]]
    [...]
* [[JMX_Access_to_Domain_Mode_EAP_7_Server_Node#URL|For EAP 7 Domain Mode Server Node]]
instances:
    - jmx_url: [...]
      [...]
      conf:
        - include:
            domain: jboss.as
            bean:
              - jboss.as:subsystem=datagrid-infinispan-endpoint,hotrod-connector=hotrod-connector
              - jboss.as:subsystem=datagrid-infinispan,cache-container=clustered
            attribute:
              bytesRead:
                alias: hotrod_endpoint_read_bytes_per_second
                metric_type: counter
              bytesWritten:
                alias: hotrod_endpoint_written_bytes_per_second
                metric_type: counter
              hits:
                alias: node_read_hits
                metric_type: counter
              misses:
                alias: node_read_misses
                metric_type: counter
              stores:
                alias: node_writes
                metric_type: counter
              averageReadTime:
                alias: node_average_read_time
                metric_type: gauge
              averageWriteTime:
                alias: node_average_write_time
                metric_type: gauge
              averageRemoveTime:
                alias: node_average_remove_time
                metric_type: gauge
              numberOfEntries:
                alias: node_cache_entries
                metric_type: gauge
</pre>


=Troubleshooting=
=Troubleshooting=
Line 90: Line 144:
<pre>
<pre>
service datadog-agent restart
service datadog-agent restart
</pre>
==Metric List==
<pre>
sudo /etc/init.d/datadog-agent jmx list_not_matching_attributes
</pre>
</pre>


Line 99: Line 159:
2016-10-19 15:34:53,433 | ERROR| App | Cannot connect to instance service:jmx:remoting-jmx://1.2.3.4:4447 javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed:
2016-10-19 15:34:53,433 | ERROR| App | Cannot connect to instance service:jmx:remoting-jmx://1.2.3.4:4447 javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed:
   DIGEST-MD5: Server rejected authentication
   DIGEST-MD5: Server rejected authentication
</pre>
===TODO===
<pre>
2016-10-19 16:20:43,683 | INFO | Connection | Connecting to: service:jmx:remote://1.2.3.4:4447
2016-10-19 16:20:43,731 | INFO | xnio | XNIO version 3.3.6.Final-redhat-1
2016-10-19 16:20:43,777 | INFO | nio | XNIO NIO Implementation Version 3.3.6.Final-redhat-1
2016-10-19 16:20:43,970 | INFO | remoting | JBoss Remoting version 4.0.18.Final-redhat-1
2016-10-19 16:20:44,663 | INFO | Instance | Connected to JMX Server at service:jmx:remote://1.2.3.4:4447
2016-10-19 16:20:44,664 | ERROR| Instance | Unable to compute a common bean scope, querying all beans as a fallback
java.lang.ArrayIndexOutOfBoundsException: 1
at org.datadog.jmxfetch.Configuration.getIncludeFiltersByDomain(Configuration.java:89)
at org.datadog.jmxfetch.Configuration.getGreatestCommonScopes(Configuration.java:227)
at org.datadog.jmxfetch.Instance.getBeansScopes(Instance.java:284)
at org.datadog.jmxfetch.Instance.refreshBeansList(Instance.java:301)
at org.datadog.jmxfetch.Instance.init(Instance.java:132)
at org.datadog.jmxfetch.App.init(App.java:354)
at org.datadog.jmxfetch.App.main(App.java:92)
</pre>
</pre>

Latest revision as of 05:30, 14 January 2022

External

Internal

Concepts

JMX Attribute

JMX Attribute Alias

If an alias is specified, it will become the metric name when displayed by the DataDog UI. Camel case will be converted to snake case. Dots are acceptable.

JMX Attribute Type

The JMX attribute type is specified as "metric_type" in configuration.

Gauge

The actual value. The agent does not attempt to calculate the rate, as in the counter's case.

Counter

For a "counter", the rate per second will be computed for this metric.

Configuration

The JMX configuration is located in /etc/dd-agent/conf.d/jmx.yaml.example.

After modification and removal of the ".example" suffix, the agent has to be restarted as shown here.

JMX integration with JBoss

Server Preparation

JBoss instances have to be configured to allow JMX agent access. Details on how to configure JBoss various JBoss versions running in various modes are available here:

WildFly and JMX

Agent Classpath and Startup Options

Working jmx.yaml configuration: 

init_config:
    custom_jar_paths:
      - /opt/jdg/bin/client/jboss-cli-client.jar

instances:
    - jmx_url: "service:jmx:remote://1.2.3.4:4447"
      user: jmx
      password: ***
      java_bin_path: /opt/jdk/bin/java
      name: jdg1
      tags:
        env: stage
        newTag: test
      conf:
        [...]

The specific value for "jmx_url" depends on the JBoss version and the mode it is running in. More details can be found here:


Make sure the JMX user has been configured on target JBoss instances, for symptoms see "JMX User Has not Been Configured on the Target JBoss Instance" section.

JMX Metric Specification

The MBeans are grouped per their JMX domains. Each attribute is specified by name, with an alias and its type.

Configuration example: 

init_config:
    [...]
instances:
    - jmx_url: [...]
      [...]
      conf:
        - include:
            domain: jboss.as
            bean:
               - jboss.as:subsystem=datagrid-infinispan-endpoint,hotrod-connector=hotrod-connector
               - jboss.as:subsystem=datagrid-infinispan,cache-container=clustered
            attribute:
               bytesRead:
                 alias: hotrod_endpoint_read_bytes_per_second
                 metric_type: counter
               bytesWritten:
                 alias: hotrod_endpoint_written_bytes_per_second
                 metric_type: counter
               hits:
                 alias: node_read_hits
                 metric_type: counter
               misses:
                 alias: node_read_misses
                 metric_type: counter
               stores:
                 alias: node_writes
                 metric_type: counter
               averageReadTime:
                 alias: node_average_read_time
                 metric_type: gauge
               averageWriteTime:
                 alias: node_average_write_time
                 metric_type: gauge
               averageRemoveTime:
                 alias: node_average_remove_time
                 metric_type: gauge
               numberOfEntries:
                 alias: node_cache_entries
                 metric_type: gauge

Troubleshooting

Agent Process Signature

ps -ef | grep java | grep datadog

dd-agent 14889 14841  0 Oct15 ? 00:05:47 \
  /opt/jdk/bin/java -Xms50m -Xmx200m \
  -classpath /opt/jdk/lib/tools.jar:/opt/datadog-agent/agent/checks/libs/jmxfetch-0.11.0-jar-with-dependencies.jar org.datadog.jmxfetch.App \
  --check jmx.yaml --check_period 15000 --conf_directory /etc/dd-agent/conf.d --log_level INFO \
  --log_location /var/log/datadog/jmxfetch.log --reporter statsd:localhost:8125 
  --status_location /opt/datadog-agent/run/jmx_status.yaml collect

Agent Reinitialization

service datadog-agent restart

Metric List

sudo /etc/init.d/datadog-agent jmx list_not_matching_attributes

Errors

JMX User Has Not Been Configured on the target JBoss Instance

2016-10-19 15:34:53,433 | ERROR| App | Cannot connect to instance service:jmx:remoting-jmx://1.2.3.4:4447 javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed:
   DIGEST-MD5: Server rejected authentication
Retrieved from "https://kb.novaordis.com/index.php?title=DataDog_and_JMX&oldid=89143"