usage_optimization

Datadog Cloud Integration: Setup and Cost Optimization

Nicolas Narbais

26 Feb 2025 — 7 min read

To achieve a comprehensive monitoring setup for your infrastructure, it’s essential to integrate cloud provider data into Datadog. Datadog supports integrations with major cloud providers, allowing you to streamline monitoring, optimize costs, and ensure a unified view of your system’s health.

This guide will take you through the key features of each cloud integration, focusing on the integration process, cost optimization, and methods to improve data freshness.

Global Integration Behavior

Crawling and Data Collection

Datadog utilizes crawlers that periodically fetch data from various cloud services, typically every 5 to 10 minutes. These crawlers gather metrics and other important data (e.g. tags) from the cloud provider’s monitoring APIs.

However, there might be slight delays in the data collection due to the inherent latency in cloud providers' APIs as well as the crawler frequency mentioned. The main two solutions are mentioned below but more specific solutions may be available, check the relevant section for it.

Increasing Fetch Frequency:
If you require more up-to-date data, you can request Datadog support to increase the fetch frequency. Keep in mind, increasing the frequency of API calls can result in higher costs from the cloud provider.
Native Data Collection:
Certain data, such as host-level metrics, can be collected directly through the Datadog Agent, which ensures more granular and real-time data collection compared to the crawlers.

Why setting up a cloud provider integration?

While the Datadog Agent provides a detailed view of your infrastructure, some cloud services cannot be monitored by the agent alone. These services require integration with Datadog to pull data via cloud provider APIs, allowing you to consolidate all monitoring data (e.g., logs, APM, metrics) into one unified dashboard.

In addition, cloud integrations enable Datadog to enrich your monitored assets with additional metadata (tags), providing a deeper level of insight into your data.

Cost Optimization

Reducing Cloud and Datadog Costs

By integrating cloud providers with Datadog, you can optimize your costs in several ways:

Asset Deduplication:
Datadog ensures that if a cloud-hosted asset (e.g., an EC2 instance or a serverless lambda) is monitored both through the cloud integration and the Datadog Agent, you will only be billed once for that asset.
Excluding Non-Essential Assets:
For cost optimization, Datadog provides options to exclude specific assets, such as unused instances, from being monitored, thus reducing the number of monitored assets and, consequently, your costs. You can create automated allowlists and denylists for more granular control. More details will be provided below.

Cloud Provider Billing

Cloud providers may charge for API calls, and Datadog integrates with cloud APIs to collect data. To minimize cloud provider billing, you can limit the scope of crawlers, excluding non-essential services or regions from being monitored. Again, more details for each provider will be given below.

AWS

Initial setup

Setting up the AWS integration is simple with Datadog’s CloudFormation template, which handles the configuration of roles, permissions to enable the crawler to fetch the data, and the installation of the Datadog Forwarder Lambda. For those who prefer Terraform, an alternative setup guide is also available.

AWS Integration Get Started: https://docs.datadoghq.com/getting_started/integrations/aws/
AWS Setup with Terraform: https://docs.datadoghq.com/integrations/guide/aws-terraform-setup/

To go beyond, if you create and delete AWS accounts on a regular basis, a multi-account setup is also available https://docs.datadoghq.com/integrations/guide/aws-organizations-setup/

Reduce Egress cost with Private Link

Using AWS PrivateLink can help reduce egress costs when connecting Datadog to AWS. This feature is currently available in specific regions like US1 and API1.

Increase metrics freshness

To enhance the freshness of metrics, a few options are available.

Use the Datadog Agent and some native integrations. For instance, the host level metrics are also captured by the agent almost instantly and it is similar with other integrations such as Redis.
Ask support to increase crawling frequency. Note that is may impact your AWS bill regarding those additional API calls.
Stream metrics. Use Metric Streams and Firehose to reduce the latency to 2 to 3 minutes. https://docs.datadoghq.com/integrations/guide/aws-cloudwatch-metric-streams-with-kinesis-data-firehose

Control costs

The Datadog Billing is related to the number of monitored assets such as hosts or lambdas. For large AWS accounts, it may be relevant to exclude some assets from being monitored. https://docs.datadoghq.com/account_management/billing/aws/#aws-resource-exclusion For this reason, it is possible to use tags to limit the resource collection.

Metric collection setup on an AWS integration tile

The AWS billing related to monitoring is often related to the number of API calls. The solution is to disable metrics from specific services and from specific regions.

AWS integration service selection to monitor

What is this Datadog Forwarder?

The Datadog Forwarder is an AWS Lambda function designed to send logs from AWS to Datadog, including:

CloudWatch and S3 logs
Logs from SNS and Kinesis events (note: Kinesis data stream events support only CloudWatch logs)
Metrics, traces, and logs from AWS Lambda functions

While the Datadog Forwarder can be used to forward metrics, traces, and logs from Lambda functions, Datadog recommends using the Datadog Lambda Extension for direct telemetry collection from Lambda execution environments. The Forwarder remains available for serverless monitoring, but it will no longer receive updates to support new features.