Log Maturity: Score Assessment

Observability, like any other discipline, varies in implementation quality and maturity. This guide outlines key success criteria at each stage of the journey—from Pro to Legend to Icon. As a bonus, a scoring app link is provided at the end to assess your current maturity level.

📈 What Does It Take to Go Pro?

At the Pro level, logging practices are structured, efficient, and cost-aware. In Datadog, becoming a Pro means implementing the following key practices:

1. Multiple Retention Periods

Different logs serve different purposes. Not all logs need to be stored for the same amount of time.

For example:

• Development logs may only be relevant for a few days.
• Production logs might be needed for long-term audits or analysis.

Use multiple retention periods to reflect the true value of each log type.

2. Exclusion Filters

Exclusion filters are essential to the Logging Without Limits strategy. They help remove logs that are:

• Irrelevant
• Too verbose
• Poorly formatted

Best Practices:

• Aim for at least one exclusion filter per index.
• Use filters to reject logs that don’t follow your schema.
• Ensure debug logs are not indexed in production environments.
  • During troubleshooting, you can temporarily disable these filters.

Check your Logging Without Limits dashboard:

You should see an exclusion rate of at least 20%. This is typical among well-structured customer setups.

🧠 How to Become a Legend

Legends go beyond filtering—they prioritize log quality and cost-efficiency at scale.

1. Log Sampling

Instead of excluding all logs of a certain type, keep a small, representative sample.

Examples:

• Keep 5% of NGINX logs to monitor trends.
• Retain 3% of verbose application logs to capture patterns.

Sampling is ideal for high-volume environments where full indexing would be too expensive.

2. Index Quotas

Index quotas prevent runaway logging from impacting your bill.

Tips:

• Set quotas on each index.
• Do not set the quota too close to your daily average—this can result in losing critical logs during spikes.
• A good rule of thumb is to set quotas 2–3× higher than your typical daily volume.

3. Increased Exclusion Rates

Legends should aim for an exclusion rate of 30% or higher. Achieving this may require more aggressive filtering.

4. Archiving

Use archives to store filtered-out logs for retrieval in exceptional cases.

Benefits:

• Lets you filter more aggressively without data loss.
• Enables shorter retention periods for indexed logs, reducing costs.
• Provides a fallback for compliance or incident investigations.

5. Log Pipelines Validation

Ensure all logs are routed through log pipelines for parsing and enrichment. Check pipeline pass-through rates:

• Aim for less than 1–2% of logs bypassing pipelines.

🧬 How to Become an Icon

The Icon stage represents advanced observability—few teams achieve this level of control, flexibility, and optimization.

1. Dynamic Exclusion Filters

Icons make exclusion filters adaptive. For example:

• During normal operations, filters exclude verbose logs.
• During incidents, filters can be relaxed or disabled to capture more data for debugging.

2. Rehydration

Rehydration allows you to temporarily re-index archived logs for analysis.

Considerations:

• Useful during investigations.
• Requires good archiving practices.
• Should be balanced with cost controls.

3. Standardized Attributes

Different teams often log the same concepts using different keys (e.g., user.id vs user_id).

Use standard attributes to:

• Normalize log structure across services.
• Enable unified queries across teams.
• Avoid writing custom logic for each format.

4. Team Ownership Tags

Include a team tag on all logs to clarify ownership.

Benefits:

• Enables efficient collaboration during incidents.
• Helps route issues to the right team.
• Supports alerting and accountability.

5. Advanced Controls and Governance

Icons implement strong governance practices such as:

• RBAC (Role-Based Access Control) for pipelines and configurations.
• Terraform for managing pipeline changes with version control and auditability.

✅ Final Thoughts

Your logging setup evolves as your organization grows. Whether you’re aiming to be a Pro, a Legend, or an Icon, the key is to balance visibility, cost, and quality. Start with structured retention and exclusion, layer in sampling and quotas, and eventually build toward dynamic filtering, standardization, and governance.

Ready to find out where you stand? Click here to get your observability score.