Effective Datadog Archiving to Maximize Log Value

In the quest to strike the perfect balance between log value and cost efficiency, archiving emerges as a pivotal component, instilling users with the assurance that their logs remain secure and retrievable. Particularly in scenarios characterized by high log volumes and sporadic analytics demands, archives play a critical role.

In this discourse, we endeavor to outline a recommended archiving strategy, tailored to the prevailing features and constraints.

Before delving into the specifics, if archiving within the Datadog log platform is unfamiliar territory, we encourage perusing our documentation on archives or participating in our enlightening enablement sessions.

Key Constraints

First and foremost, it’s imperative to acknowledge the constraint posed by the limit on the number of archives. Strategizing around this limitation necessitates thoughtful planning.

Secondly, access restrictions pose another significant constraint. In instances where users are compartmentalized and cannot access each other’s logs, archives must be configured accordingly, ensuring logs are directed to their designated secured storage space.

The third constraint pertains to billing for archives. While archiving itself does not incur additional costs, rehydrating archives is subject to variables such as the duration of log retention, the quantity of logs indexed, the volume of data to be scanned, and the frequency of reindexing.

Lastly, akin to the indexing system, the order of archives is crucial. Logs adhere to a top-to-bottom approach when selecting the appropriate archive based on filters.

Proposed Strategy

Commencing with a foundational archive featuring a wildcard filter (`*`) ensures comprehensive backup coverage, facilitating the implementation of more aggressive strategies such as Logs without Limits (LwL).

Subsequently, configuring specialized archives is recommended:

1. Team-specific Archives: Logs governed by specific teams with stringent access protocols warrant dedicated archives. Employing a unified tag (e.g., `team:security`) simplifies identification and management.

2. Frequently Accessed Archives: For teams requiring frequent rehydration, setting up compact archives is prudent. This not only optimizes costs but also expedites the rehydration process, ideal for scenarios where analysis is periodic and non-urgent. If urgency is also a requirement, check the new flex logs solution.

3. High-Volume Consumer Archives: Archives catering to high-volume log consumers isolate their rehydration costs, ensuring affordability and efficiency for other teams.

In Conclusion

Crafting the perfect archiving strategy may entail iterative refinement. Initiating with a catch-all archive serves as a solid foundation, allowing for subsequent additions tailored to specific use cases. Crucially, users must swiftly discern the location of their logs to optimize rehydration efficiency, leveraging features like pattern recognition and analytics for enhanced clarity.

For those grappling with identifying high-volume logs or uncertain where to commence, utilizing Datadog’s pattern feature or analytics tools provides invaluable insights and guidance.