Originally reported by devclass.com. Published January 2020.
What Changed in Datadog Watchdog
Datadog announced an infrastructure monitoring expansion for its Watchdog anomaly-detection engine. The update extended Watchdog beyond application performance monitoring (APM) to cover a range of common infrastructure components, giving engineering teams a unified view of both application and infrastructure health inside the Datadog platform.
Background: What Is Datadog Watchdog?
Watchdog was first introduced in July 2018. Its original purpose was to surface anomalies in APM metrics — specifically spikes in request rates, error rates, and latency — without requiring engineers to write custom alert rules for every service.
Under the hood, Watchdog applies machine learning techniques including:
- Anomaly detection — identifying metric values that deviate significantly from established baselines.
- Outlier detection — flagging hosts or services behaving differently from their peers.
- Forecasting algorithms — predicting future metric trends to catch problems before they become critical.
Rather than firing raw alerts, Watchdog packages its findings into stories: structured summaries that include the timeframe of the anomaly, the affected resources, and the duration of the irregularity. Each story links through to a detail page with additional context, and Watchdog may surface related signals to help teams gauge severity.
The Infrastructure Monitoring Addition
The January 2020 update added a new infrastructure layer to Watchdog, enabling it to detect patterns and anomalies across system-level metrics and popular infrastructure integrations, including:
- System metrics (CPU, memory, disk, network)
- Redis
- PostgreSQL
- NGINX
- Amazon Web Services (AWS)
According to Datadog’s announcement, the engine automatically discovers monitored infrastructure — no special configuration is required beyond having the relevant Datadog integrations already enabled.
How to Enable Infrastructure Monitoring in Watchdog
There are two primary ways to set up infrastructure monitors through Watchdog:
1. Via the New Monitor Page
A new Watchdog category was added to the “New Monitor” section in the Datadog UI. From there, teams can add infrastructure sources to Watchdog and configure alerting thresholds and notification channels in one place.
2. Via Watchdog Stories
When Watchdog surfaces a story — for example, an unusual spike in Redis connection errors — it offers the option to create a custom monitor directly from that story. This is particularly useful for teams that want to ensure they are notified if a similar pattern recurs in the future, without having to recreate the detection logic manually.
Why This Matters for DevOps Teams
Before this update, Watchdog’s automated anomaly detection was primarily focused on application-layer signals. Infrastructure problems — a PostgreSQL instance running unusually slow queries, an NGINX worker pool hitting its connection limit, or an AWS service experiencing elevated error rates — required teams to write and maintain their own alert rules.
Extending Watchdog to the infrastructure tier reduces the manual overhead of defining thresholds for every metric across every integration. The automatic service discovery means that newly added hosts or services can be brought under Watchdog coverage without additional configuration steps.
The stories format also helps with cross-team communication: instead of sharing a raw graph link, an on-call engineer can share a Watchdog story that already includes relevant context, related signals, and a timeline of the anomaly.
Summary
| Feature | Detail |
|---|---|
| Engine | Datadog Watchdog |
| Originally launched | July 2018 (APM anomaly detection) |
| Infrastructure expansion announced | January 2020 |
| Covered integrations | System metrics, Redis, PostgreSQL, NGINX, AWS |
| Setup required | None beyond existing Datadog integrations |
| Monitor entry points | New Monitor page, Watchdog stories |
For teams already using Datadog, the infrastructure Watchdog capability is worth enabling — particularly for high-cardinality environments where manually writing alert rules for every Redis cluster or NGINX farm is impractical.