Observability Without the Invoice: An SRE's Open Source APM Experiment

I spent years as a Dynatrace power user — building Golden Signal dashboards, SLOs, and alerts at scale via Terraform for a Vehicle Commerce platform, and before that, running Dynatrace for a high-traffic live commerce business across five international markets. I know what “good APM” feels like: one-click root cause, automatic baselining, a dependency graph that just works.

I also know what it costs.

That gap — between what Dynatrace/Datadog deliver and what they charge — is what pushed me to spend the last few months building an open-source alternative: Grafana Alloy, Beyla, Tempo, Mimir, Loki, Pyroscope, and Faro, tied together with Cilium/Hubble for network-layer visibility. This post is about what I built, what it actually replaces, and — just as important — where it doesn’t.

The cost problem, concretely

Datadog and Dynatrace price on host count, ingested GB, custom metrics, and “DEM units” that multiply fast once you have more than a handful of services. A mid-size platform team easily lands in six figures a year before anyone’s added a single new microservice. None of that spend buys you more insight — it buys you the same insight, packaged.

The OSS stack flips the cost model: you pay for compute and storage you control, not per-signal licensing. That’s the FinOps case in one sentence.

What I actually built

Grafana Alloy is the collector — a single OTel-compatible agent replacing Dynatrace’s OneAgent and Datadog’s Agent. It’s the ingestion spine for everything below.

Beyla gives eBPF-based auto-instrumentation at the application layer (L7) — HTTP/gRPC latency, errors, and traces without touching application code. This is the closest OSS equivalent to Dynatrace’s “just install OneAgent and see everything” magic.

Cilium + Hubble cover the network layer (L4) — service-to-service flow visibility, DNS, connection-level drops. Beyla and Hubble aren’t redundant; they answer different questions (app behavior vs. network behavior), and together they cover ground that a single commercial agent normally charges you extra for.

Tempo (traces), Mimir (metrics, Prometheus-compatible at scale), and Loki (logs) are the three storage backends — deliberately kept separate rather than in one proprietary blob, so each can scale and be queried on its own terms.

Pyroscope adds continuous profiling — CPU/memory flame graphs over time, a feature Datadog and Dynatrace both gate behind their higher pricing tiers.

Faro closes the loop with real user monitoring (RUM) on the frontend, correlating browser errors and Core Web Vitals back to backend traces.

K6 covers synthetics — scripted API and browser checks as code, with results flowing straight into Mimir alongside everything else.

Where it genuinely competes

• Golden signals and SLOs. Mimir + Grafana dashboards + Grafana Alertmanager reproduce Dynatrace’s Golden Signal dashboards and SLO burn-rate alerts almost 1:1 — I’ve rebuilt the same alerting logic I used to write in Dynatrace Terraform, now against Mimir.

• Distributed tracing. Tempo plus Beyla’s auto-instrumentation gets you request-level traces without code changes, matching OneAgent’s zero-touch tracing for most common frameworks.

• Cost predictability. You know your bill because it’s your infrastructure. No surprise true-ups when someone ships a new service and DEM units spike.

• No vendor lock-in on data. OTel is the lingua franca in and out, so switching a backend later (Mimir → something else) doesn’t mean re-instrumenting everything.

Where it doesn’t (yet)

Being honest here matters more than the pitch:

• Automatic root cause analysis. Dynatrace’s Davis AI correlating a deployment, a dependency failure, and a customer-facing error into one causal chain — with no query written — has no true OSS equivalent yet. Grafana’s correlation features are improving but still require a human to connect the dots.

• Operational overhead. This stack is you running Kubernetes operators, managing retention and compaction on Mimir/Loki/Tempo, and keeping eBPF probes healthy across kernel versions. Dynatrace’s cost includes not having to do any of that. That operational tax is real and should be weighed against the license fee it replaces.

• Anomaly detection out of the box. Dynatrace’s baselining is trained per-entity automatically. In the OSS stack, you’re writing PromQL-based alerting rules or bringing your own anomaly detection — there’s no default “this is weird for this service at this time of day” model.

• Single pane of glass, single vendor support line. When something breaks in the observability stack itself, you’re debugging your own platform, not opening a support ticket.

The honest framing

This isn’t “OSS observability beats commercial APM.” It’s “OSS observability, run by a team with the SRE muscle to operate it, gets you 80–90% of the signal at a fraction of the cost, in exchange for owning the operational burden Dynatrace and Datadog otherwise absorb for you.” For a platform team that already runs Kubernetes at scale and has the eBPF/observability chops, that trade is increasingly worth making — especially once you’re paying for hosts and ingested GB you can’t easily forecast.

For a smaller team without dedicated platform engineering capacity, the commercial tools’ operational simplicity is still buying something real. The right call depends less on the technology and more on whether your organization has people who’d rather run Mimir than write a check.