DataDog and JMX: Difference between revisions
No edit summary |
|||
(8 intermediate revisions by the same user not shown) | |||
Line 5: | Line 5: | ||
=Internal= | =Internal= | ||
* [[ | * [[Datadog]] | ||
=Concepts= | =Concepts= | ||
Line 12: | Line 12: | ||
===JMX Attribute Alias=== | ===JMX Attribute Alias=== | ||
If an alias is specified, it will become the metric name when displayed by the DataDog UI. Camel case will be converted to snake case. Dots are acceptable. | |||
===JMX Attribute Type=== | ===JMX Attribute Type=== | ||
The JMX attribute type is specified as "metric_type" in configuration. | |||
====Gauge==== | ====Gauge==== | ||
The actual value. The agent does not attempt to calculate the rate, as in the [[#Counter|counter's]] case. | |||
====Counter==== | ====Counter==== | ||
For a "counter", the rate per second will be computed for this metric. | |||
=Configuration= | =Configuration= | ||
Line 23: | Line 31: | ||
The JMX configuration is located in /etc/dd-agent/conf.d/jmx.yaml.example. | The JMX configuration is located in /etc/dd-agent/conf.d/jmx.yaml.example. | ||
After modification and removal of the ".example" suffix, | After modification and removal of the ".example" suffix, the agent has to be restarted [[#Agent_Reinitialization|as shown here]]. | ||
=JMX integration with JBoss= | =JMX integration with JBoss= | ||
Line 75: | Line 79: | ||
==JMX Metric Specification== | ==JMX Metric Specification== | ||
The MBeans are grouped per their JMX domains. Each attribute is specified by name, with an alias and its [[#JMX_Attribute_Type|type]]. | The MBeans are grouped per their JMX domains. Each attribute is specified by name, with an [[#JMX_Attribute_Alias|alias]] and its [[#JMX_Attribute_Type|type]]. | ||
Configuration example: | |||
<pre> | |||
init_config: | |||
[...] | |||
instances: | |||
- jmx_url: [...] | |||
[...] | |||
conf: | |||
- include: | |||
domain: jboss.as | |||
bean: | |||
- jboss.as:subsystem=datagrid-infinispan-endpoint,hotrod-connector=hotrod-connector | |||
- jboss.as:subsystem=datagrid-infinispan,cache-container=clustered | |||
attribute: | |||
bytesRead: | |||
alias: hotrod_endpoint_read_bytes_per_second | |||
metric_type: counter | |||
bytesWritten: | |||
alias: hotrod_endpoint_written_bytes_per_second | |||
metric_type: counter | |||
hits: | |||
alias: node_read_hits | |||
metric_type: counter | |||
misses: | |||
alias: node_read_misses | |||
metric_type: counter | |||
stores: | |||
alias: node_writes | |||
metric_type: counter | |||
averageReadTime: | |||
alias: node_average_read_time | |||
metric_type: gauge | |||
averageWriteTime: | |||
alias: node_average_write_time | |||
metric_type: gauge | |||
averageRemoveTime: | |||
alias: node_average_remove_time | |||
metric_type: gauge | |||
numberOfEntries: | |||
alias: node_cache_entries | |||
metric_type: gauge | |||
</pre> | |||
=Troubleshooting= | =Troubleshooting= |
Latest revision as of 05:30, 14 January 2022
External
Internal
Concepts
JMX Attribute
JMX Attribute Alias
If an alias is specified, it will become the metric name when displayed by the DataDog UI. Camel case will be converted to snake case. Dots are acceptable.
JMX Attribute Type
The JMX attribute type is specified as "metric_type" in configuration.
Gauge
The actual value. The agent does not attempt to calculate the rate, as in the counter's case.
Counter
For a "counter", the rate per second will be computed for this metric.
Configuration
The JMX configuration is located in /etc/dd-agent/conf.d/jmx.yaml.example.
After modification and removal of the ".example" suffix, the agent has to be restarted as shown here.
JMX integration with JBoss
Server Preparation
JBoss instances have to be configured to allow JMX agent access. Details on how to configure JBoss various JBoss versions running in various modes are available here:
Agent Classpath and Startup Options
Working jmx.yaml configuration:
init_config: custom_jar_paths: - /opt/jdg/bin/client/jboss-cli-client.jar instances: - jmx_url: "service:jmx:remote://1.2.3.4:4447" user: jmx password: *** java_bin_path: /opt/jdk/bin/java name: jdg1 tags: env: stage newTag: test conf: [...]
The specific value for "jmx_url" depends on the JBoss version and the mode it is running in. More details can be found here:
- For Standalone EAP 6
- For EAP 6 Host Controller
- For EAP 6 Domain Mode Server Node
- For Standalone EAP 7
- For EAP 7 Host Controller
- For EAP 7 Domain Mode Server Node
Make sure the JMX user has been configured on target JBoss instances, for symptoms see "JMX User Has not Been Configured on the Target JBoss Instance" section.
JMX Metric Specification
The MBeans are grouped per their JMX domains. Each attribute is specified by name, with an alias and its type.
Configuration example:
init_config: [...] instances: - jmx_url: [...] [...] conf: - include: domain: jboss.as bean: - jboss.as:subsystem=datagrid-infinispan-endpoint,hotrod-connector=hotrod-connector - jboss.as:subsystem=datagrid-infinispan,cache-container=clustered attribute: bytesRead: alias: hotrod_endpoint_read_bytes_per_second metric_type: counter bytesWritten: alias: hotrod_endpoint_written_bytes_per_second metric_type: counter hits: alias: node_read_hits metric_type: counter misses: alias: node_read_misses metric_type: counter stores: alias: node_writes metric_type: counter averageReadTime: alias: node_average_read_time metric_type: gauge averageWriteTime: alias: node_average_write_time metric_type: gauge averageRemoveTime: alias: node_average_remove_time metric_type: gauge numberOfEntries: alias: node_cache_entries metric_type: gauge
Troubleshooting
Agent Process Signature
ps -ef | grep java | grep datadog dd-agent 14889 14841 0 Oct15 ? 00:05:47 \ /opt/jdk/bin/java -Xms50m -Xmx200m \ -classpath /opt/jdk/lib/tools.jar:/opt/datadog-agent/agent/checks/libs/jmxfetch-0.11.0-jar-with-dependencies.jar org.datadog.jmxfetch.App \ --check jmx.yaml --check_period 15000 --conf_directory /etc/dd-agent/conf.d --log_level INFO \ --log_location /var/log/datadog/jmxfetch.log --reporter statsd:localhost:8125 --status_location /opt/datadog-agent/run/jmx_status.yaml collect
Agent Reinitialization
service datadog-agent restart
Metric List
sudo /etc/init.d/datadog-agent jmx list_not_matching_attributes
Errors
JMX User Has Not Been Configured on the target JBoss Instance
2016-10-19 15:34:53,433 | ERROR| App | Cannot connect to instance service:jmx:remoting-jmx://1.2.3.4:4447 javax.security.sasl.SaslException: Authentication failed: all available authentication mechanisms failed: DIGEST-MD5: Server rejected authentication