Datadog Concepts Monitors and Alerting

External

• https://docs.datadoghq.com/monitors/

Internal

• Datadog Concepts

Overview

When something goes wrong, a computer tells you about it. This is what a monitor is: a Datadog feature that actively checks metrics, integration availability, network endpoints, etc. and communicates when an alerting condition occurs. The monitor has a query and alert conditions. There are different monitor types.

Structure of a Monitor

{
  "id": 99999999,
  "name": "MyApp: Latency Too High",
  "type": "query alert",
  "query": "min(last_30m):avg:myapp.health_check.latency{env:prod} > 60000",
  "message": "something\n\n something else\n\n @some-group@group.mycompany.com",
  "priority": 2,
  "restricted_roles": null,
  "tags": [
  	"env:prod"
  ],
  "options": {
  	"notify_audit": false,
  	"timeout_h": 0,
  	"silenced": {},
  	"include_tags": true,
  	"thresholds": {
  		"critical": 60000
  	},
  	"new_host_delay": 300,
  	"require_full_window": false,
  	"notify_no_data": false,
  	"renotify_interval": 0,
  	"escalation_message": "",
  	"no_data_timeframe": null
  }
  "created_at": 1629504639000,
  "created": "2021-08-21 00:10:39.507701+00:00",
  "creator": {
  	"email": "me@mycompany.com",
  	"handle": "me@mycompany.com",
       "id": 9999999,
	"name": null
  },
  "modified": "2022-04-18 23:52:11.280131+00:00",
  "overall_state_modified": "2021-08-21T21:43:23+00:00"
  "overall_state": "OK",
  "multi": false,
  "org_id": 999999,
  "deleted": null
}

Monitor Types

Metric Monitor

https://docs.datadoghq.com/monitors/create/types/metric/

Metric monitors watch a continuous stream of data. The metrics are collected via the Datadog Agent or the API and can be alerted upon if they cross a threshold (for example) over a given period of time. Other alert detection methods are available.

Any metric currently reporting to Datadog is available for monitors.

Alert Detection Method

Threshold

A threshold alert compares metric values to a static threshold. This is the standard alert case. On each alert evaluation, Datadog calculates average/min/max/sum over the selected period and checks if it is above or below the threshold. The distribution metric type offers additional threshold options of calculating percentiles over the selected period.

Change

A change alert compares the absolute or relative (%) change in value between N minutes ago and now, and against a given threshold. The compared data points are not single points but are computed using the parameters in the alert conditions section. On each alert evaluation, Datadog calculates the raw difference (a positive or negative value) between the series now and N minutes ago, then computes the average/minimum/maximum/sum over the selected period. An alert is triggered when this computed series crosses the threshold. This type of alert is useful to track spikes, drops, or slow changes in a metric when there is not an unexpected threshold.

Anomaly

An anomaly detection alert uses past behavior to detect when a metric is behaving abnormally. For more details see Anomaly Monitor

Outliers

An outlier alert notifies when a member of a group (host, availability zone, partition, etc) is behaving unusually compared to the rest. For more details see outlier monitors.

Forecast

A forecast alert predicts the future behavior of a metric and compares it to a static threshold. It is well-suited for metrics with strong trends or recurring patterns. On each alert evaluation, a forecast alert predicts the future values of the metric along with the expected deviation bounds. An alert is triggered when any part of the bounds crosses the configured threshold. For more details see forecast monitors.

Host Monitor

https://docs.datadoghq.com/monitors/create/types/host

A host monitor listens to the Datadog Agent heartbeats and notifies on the status of the heartbeat. This could give an indication whether the hosts the Agents run on are responsive. Every Datadog Agent reports a service check called datadog.agent.up with the status OK. The Host monitor has two kind of alert conditions: Check Alert and Cluster Alert.

Anomaly Monitor

https://docs.datadoghq.com/monitors/create/types/anomaly/

Outlier Monitor

https://docs.datadoghq.com/monitors/create/types/outlier/

Forecast Monitor

https://docs.datadoghq.com/monitors/create/types/forecasts/

APM Monitor

https://docs.datadoghq.com/monitors/create/types/apm/

APM Application Performance Monitoring.

Audit Logs Monitor

https://docs.datadoghq.com/monitors/create/types/audit_logs/

CI Pipelines Monitor

https://docs.datadoghq.com/monitors/create/types/ci_pipelines/

Composite Monitor

https://docs.datadoghq.com/monitors/create/types/composite/

Custom Check Monitor

https://docs.datadoghq.com/monitors/create/types/custom_check/

Error Tracking Monitor

https://docs.datadoghq.com/monitors/create/types/error_tracking/

Event Monitor

https://docs.datadoghq.com/monitors/create/types/event/

Integration Monitor

https://docs.datadoghq.com/monitors/create/types/integration/

Live Process Monitor

https://docs.datadoghq.com/monitors/create/types/process/

Logs Monitor

https://docs.datadoghq.com/monitors/create/types/log/

Network Monitor

https://docs.datadoghq.com/monitors/create/types/network/

Process Check Monitor

https://docs.datadoghq.com/monitors/create/types/process_check/

Real User Monitoring

https://docs.datadoghq.com/monitors/create/types/real_user_monitoring/

Watchdog Monitor

https://docs.datadoghq.com/monitors/

Notification

https://docs.datadoghq.com/monitors/notify/

Notification are a key component of monitors. They keep the team informed of issues and support troubleshooting. The notification are configured when the monitors are created.

Notify your Team

Integrations

Jira, PagerDuty, Slack, Webhooks

Triggered Monitor

https://docs.datadoghq.com/monitors/manage/#triggered-monitors

Incident

https://docs.datadoghq.com/monitors/incident_management

SLO

https://docs.datadoghq.com/monitors/service_level_objectives/

Also see:

Service Level Objectives (SLO)

Operations

• Create monitors
• Mute monitors
• Unmute monitors
• Resolve monitors
• Delete monitors