Datadog Event Platform by example

The Datadog event platform offers extensive insights to users regarding specific system changes. Before going further, compared to logs, events have a different pricing model, their retention is 15 months and the set of features is also slightly less extensive. Most often, events are used to signify alterations within a system, such as version updates.

To illustrate, let’s explore how to leverage the event platform for sending notifications on new Datadog agent updates. Getting some events in the platform is the initial step. While native integrations like Kubernetes provide many events, custom events can also be submitted through scripts or emails.

In this example, we’ll utilize the RSS integration and configure it to receive notifications from the Datadog agent repository from this url https://github.com/DataDog/datadog-agent/releases.atom.

After configuring the setup, it may take some time to witness the appearance of events, but you can trust the process for the subsequent steps.

With the configuration in place, expect a stream of events in various formats within the platform. The objective now is to parse these events and pinpoint those belonging to official new releases.

To achieve this, we’ll enhance the logs using a pipeline.

• Step 1: Establish a pipeline to analyze all events originating from the source:feed.
• Step 2: Implement a grok processor to effectively parse the message content and extract the new version number. Below is the grok parsing rule utilized. These rules will extract both the version numbers and their extensions.

my_rule %{regex("[0-9]+"):version.major}\.%{regex("[0-9]+"):version.minor}\.%{regex("[0-9]+"):version.fix}(\-%{regex("[^ ]+"):version.extension})? 
 
another_rule pkg\/%{regex("[^ ]+"):package}

• Step 3 (optional): Employ a category processor to discern whether the version is minor, major, or an extension. Refer below for each category. (Side note: category processors are often under utilized but are very powerful to organize the incoming data.)

The 3 categories:

• extension → @version.major:* @version.extension:*
• fix → @version.major:* -@version.extension:* -@version.fix:0
• main → @version.major:* -@version.extension:* @version.fix:0

The pipeline should utlimately looks like the one below.

To complete the process and receive alerts, all that’s left is to create a monitor based on events and configure it to send notifications when the conditions are satisfied.

For enhanced insights, it’s also recommended to utilize events by overlaying them on graphs within dashboards. This practice provides users with additional context regarding significant changes, aiding in troubleshooting efforts.

In the example below, we simply overlay on top of the number of agent running a bar when a new release is available. In other cases, it is often handy to overlay a app version change on top of latency and error rate graphs.