Observability Through Datadog

Scattered metrics, logs, and traces are joined by a single tag, which is what lets you narrow an incident down to its cause on one screen.

Observability versus monitoring

Observability is the ability to infer the internal state of a system from external outputs alone, such as logs, metrics, and traces. Traditional monitoring watches for problems defined in advance; observability is closer to exploring an unanticipated problem after the fact.

Microservices are what make the distinction matter. A single user click passes through dozens of backend services and external APIs (application programming interfaces). Once the path is fragmented like that, there is no way to define in advance where the slowdown will occur. That calls for a structure where data is collected first and questioned later.

Observability rests on three kinds of data. Each answers a different question, so looking at them separately makes causes hard to narrow down.

Datadog combines these three data types with real user monitoring (RUM), security, and cost management. It provides them through one SaaS (software as a service) platform. Rather than running your own backend, agents ship data to the Datadog cloud and the analysis capability is rented.

Three collection paths

Datadog does not force a single collection path. The collection route depends on the monitored target: host resources, services outside your cloud, or application code.

flowchart LR
  subgraph collect[Collection layer]
    A[Datadog Agent<br/>host metrics and system logs]
    L[APM libraries<br/>language runtime traces]
    C[Crawler integrations<br/>cloud API polling]
  end
  subgraph backend[Datadog cloud backend]
    B[Aggregation, parsing, indexing<br/>trace sampling]
  end
  subgraph use[Consumption layer]
    D[Dashboards and monitors<br/>correlation analysis]
  end
  A --> B
  L --> B
  C --> B
  B --> D

The first route is agent-based integration. A lightweight agent installed on the host collects system metrics and logs. The second is crawler-based integration, which periodically calls APIs with stored credentials. It pulls data from managed cloud services and SaaS tools such as Slack and PagerDuty where no agent can be installed.

The third is library integration. A library embedded directly in a Node.js, Python, or Java runtime produces code-level APM traces.

What the three paths share is that none of the heavy computation happens on the host. Agents only collect and ship, while aggregation, parsing, indexing, and trace sampling all run in the Datadog cloud. That off-loading structure conserves host resources and keeps the agent light. Based on the integration catalog Datadog publishes, official plugins number more than 750 (as of 2024-10, Datadog Integrations page).

Tagging is what connects the data

More important than the collection structure is how the collected data gets joined. In Datadog, the join key is the tag. Before an agent ships anything, metadata such as host, env, service, version, and team is attached as tags to every metric, log, and trace.

The backend uses those tags the way a relational database uses foreign keys. Data from different sources ends up in one context whenever the tags match. For example, CPU may spike on service:payment in env:production. Matching tags connect that spike to trace bottlenecks and error logs from the same moment.

For tags to do their job, service names have to be consistent everywhere. Datadog calls this unified service tagging and standardizes it through three environment variables. Set these correctly and the language-level log processor injects the trace ID into every log automatically.

# Unified service tagging - three variables bind traces, logs, and metrics to one service
export DD_ENV=production
export DD_SERVICE=payment
export DD_VERSION=2.4.1

Here DD_ENV names the environment (production, staging), DD_SERVICE names the service, and DD_VERSION names the deployed version. Attach all three values to metrics, logs, and traces. The version tag then identifies error-rate increases immediately after a particular deployment.

APM and distributed tracing

Application performance monitoring (APM) tracks the path a request takes between services. Distributed tracing visualizes that path. A trace ID is attached to a single request, and each service segment becomes a span, which reveals which segment produced the latency.

When traces are joined with tags and logs, the debugging path gets short. Spotting a CPU bottleneck on an infrastructure dashboard leads directly to the execution plan of the database query behind it and to the exception logs from that moment. The flow from metrics down into logs and traces never breaks across screens.

Datadog adds a continuous profiler on top. It analyzes memory heap and CPU usage in a running application at the method and code-line level, without code changes or restarts. Beyond incident resolution, it is used to find wasted resources and cut costs.

Monitors and SLOs

Collection and correlation support investigation after an event. Monitors watch for predefined conditions. A monitor puts a threshold on a metric, log, or trace and sends an alert when the condition is crossed. A metric monitor query narrows its target with tags.

avg(last_5m):avg:system.cpu.user{env:production,service:payment} > 80

This query fires when the average CPU usage of the payment service in production over the last 5 minutes exceeds 80. The {env:production,service:payment} clause reuses exactly the tags defined earlier. With a consistent tag scheme, monitors, dashboards, and correlation analysis all speak the same language.

Where to put the threshold is decided by service level targets. Distinguishing three acronyms makes the criteria clear.

Once an SLO is set, its complement becomes the error budget, the amount of failure that is acceptable. The calculation is plain arithmetic.

SLO = 99.95% (monthly)
Error budget = 100% - 99.95% = 0.05%
Allowed downtime = 30 days x 24 hours x 60 minutes x 0.05% = about 21.6 minutes

While budget remains, new features ship; once it is spent, deployments stop and the team turns to stability work. Adding a burn rate alert that watches how fast the budget is being consumed makes it possible to react before it runs out.

Cost as a structural trap

Integration, Datadog's strength, also complicates the cost structure. Each product family bills on a different unit, and turning on more modules stacks cost onto the same host.

These figures come from Datadog's official pricing page. Prices vary with contract, plan, and date, so confirm them when adopting the service. What matters in the table is not the exact amount but the structure: APM attaches per host separately from infrastructure, and log costs grow through indexing.

The most common reason costs run away is cardinality. A unique tag value such as version:2.5.7234-alpha sharply increases the number of combinations. The backend must then handle many more time series. That calls for governance that enforces a finite tag pool at the major version level.

Reducing volume at ingestion is equally clear-cut. DEBUG logs and health check (2xx) logs in production get dropped at the agent or in the pipeline. Normal transactions are sampled partially, while error and latency-exceeding traces are kept at 100%.

Filtering and sampling alone can cut data volume substantially (estimate). Pair that with short retention windows and archiving long-term logs into a cheaper bucket.

Summary

What holds Datadog together is the connection model rather than the collection paths. Metrics, logs, and traces gathered along three routes are joined by a single tag, and that structure is the center of the design. With unified service tagging applying env, service, and version consistently, monitor queries, APM traces, and correlation analysis all share one tag scheme.

SLOs and error budgets sit on top of that and decide where alerts go and when deployments stop. But deeper integration grows cardinality and indexing costs alongside it. Define tag governance, sampling, and retention during the design phase.