Originally announced August 2020. Source: aithority.com.
Overview
Datadog, the monitoring and security platform for cloud applications, announced native support for ingesting log data via Amazon Kinesis Data Firehose. The integration allows customers to stream logs from AWS services directly into Datadog without deploying or maintaining standalone log forwarder infrastructure in their environment.
What Is Amazon Kinesis Data Firehose?
Amazon Kinesis Data Firehose is a fully managed AWS service that reliably captures, transforms, and loads streaming data into data lakes, data stores, and analytics tools. It acts as a centralized collection point for log and event streams produced by multiple AWS services, then routes that data to configured destinations — including third-party monitoring platforms such as Datadog.
AWS services that can deliver logs through Kinesis Data Firehose include:
- Amazon CloudWatch Logs — application and infrastructure log groups
- Amazon API Gateway — access and execution logs
- AWS Lambda — function invocation and error logs
- Amazon EC2 — system and application logs forwarded via CloudWatch agents
How the Datadog Integration Works
With Kinesis Data Firehose configured as the delivery stream and Datadog set as the destination, log data flows from AWS services to Datadog entirely through the AWS Management Console and APIs. Teams no longer need to deploy and manage a separate Lambda-based log forwarder or custom pipeline to get AWS log data into Datadog.
Once logs arrive in Datadog, they are available through the platform’s full suite of Log Management capabilities.
Key Log Management Features Unlocked
Real-Time Visibility with Live Tail
Engineers can observe streaming log events as they arrive using Datadog’s Live Tail, making it possible to troubleshoot issues in real time without waiting for indexing cycles.
Logging Without Limits
Datadog’s Logging without Limits feature allows teams to ingest all logs and then apply dynamic indexing filters to control which logs are stored and searchable long-term. This decouples ingestion volume from cost, letting teams keep high-value logs while reducing spend on verbose but low-priority data.
Automatic Parsing and Metadata Enrichment
Datadog automatically parses structured and semi-structured log formats from AWS services and enriches each event with additional contextual metadata, reducing the manual pipeline work typically required before logs become queryable.
Log Analytics and Generated Metrics
Log Analytics enables high-cardinality queries across large log volumes, making it easier to surface patterns in error rates, latency distributions, or security events. Teams can also generate metrics directly from log data to track long-term trends without retaining the raw log events.
Correlation with Metrics and Traces
Logs ingested via Kinesis Data Firehose can be correlated with infrastructure metrics and distributed traces already flowing into Datadog, giving engineers a unified view of availability, performance, and security across their AWS environment.
What This Means for DevOps and Platform Teams
Before this integration, sending AWS service logs to Datadog typically required deploying a Lambda function (the Datadog Forwarder) and managing its configuration, permissions, and scaling. Kinesis Data Firehose as a native delivery mechanism reduces operational overhead significantly: the delivery stream is a managed AWS resource, and routing changes can be made from the AWS console without touching custom code.
For teams already using Kinesis Data Firehose for other data pipelines, this also means log routing to Datadog can be added as another destination within an existing stream configuration.
Official Statements
“With support for this new feature, customers can now send their AWS logs and telemetry to Datadog without setting up any additional infrastructure, which will make analysis and troubleshooting within Datadog’s cloud monitoring platform easier.”
— Ilan Rabinovitch, Vice President, Product and Community, Datadog
“Combined with Amazon Kinesis Data Firehose, Datadog’s log management offering can help customers gain a better understanding of the availability and security of their cloud infrastructure.”
— Doug Yeum, Head of Worldwide Channel and Alliances, Amazon Web Services
Getting Started
To configure this integration, you create a Kinesis Data Firehose delivery stream in the AWS Management Console, select Datadog as the HTTP endpoint destination, and provide your Datadog API key. CloudWatch Logs subscription filters or other AWS service log delivery settings can then point to that Firehose stream. Refer to the Datadog documentation on Kinesis Firehose log ingestion for current setup steps, as configuration details may have evolved since the original announcement.