Prometheus & OpenTelemetry

DriftWatch instruments every agent decision as OpenTelemetry — traces, metrics, and logs. It exports over standard OTLP/HTTP, so any OTel-compatible backend can receive it. The drift detector then reads metrics back out through a Prometheus-compatible query API (/api/v1/query) to compute drift, via the SDK's built-in PrometheusMetricsSource — so a Prometheus-compatible backend is both a source of dashboards and part of the control loop.

This guide brings up the reference OTel Collector + Prometheus stack, points DriftWatch at it, and shows what to look at once data flows.

Two endpoints, two jobs

This is the one thing to get right. DriftWatch talks to your observability stack in two directions, and they're different hosts:

Metrics don't reach Prometheus directly — they go OTLP → Collector → the collector's Prometheus exporter → Prometheus scrapes that exporter. PROMETHEUS_URL points at Prometheus (or Mimir/Cortex/Thanos), never at the collector.

Run the reference stack

The repo ships a complete local stack — docker-compose.observability.yml — with DriftWatch, Redis, an OTel Collector, and Prometheus wired together on one network:

docker compose -f docker-compose.observability.yml up -d --build

• The collector (observability/otel-collector/otel-collector-config.yaml) receives OTLP on :4318 and re-exposes metrics on :8889 for scraping. Traces and logs go to its debug exporter (stdout) in this reference config — swap in Tempo/Loki/Jaeger/etc. there if you want them stored rather than just printed.
• Prometheus (observability/prometheus/prometheus.yml) scrapes the collector's :8889 target. This is what PROMETHEUS_URL points at inside the compose network (http://prometheus:9090).
• DriftWatch exports OTLP to the collector and queries Prometheus for drift detection — both wired automatically by the compose file.

Deploying on Coolify? docker-compose.coolify.yml is the same topology, adapted for Coolify's networking model — see deployment.md.

Already run a Prometheus/Mimir/Cortex/Thanos and an OTel Collector elsewhere? Point OTEL_EXPORTER_OTLP_ENDPOINT at your collector and PROMETHEUS_URL at your existing query API instead of running this stack — neither file is DriftWatch-specific beyond the metric names the collector happens to receive.

Connecting to a managed backend

PrometheusMetricsSource (packages/sdk/src/drift/prometheus-source.ts) works against anything speaking the Prometheus HTTP query API — including managed services like Grafana Cloud (Mimir):

PROMETHEUS_URL=https://prometheus-prod-…grafana.net/api/prom
PROMETHEUS_BEARER_TOKEN=<your-token>

PROMETHEUS_BEARER_TOKEN is sent as a standard Authorization: Bearer <token> header on every query. Leave it empty for a local/self-hosted Prometheus with no auth in front of it.

Verify it's flowing

Send some traffic (quickstart step 3), generate ~15–20 requests, wait a scrape interval or two, then query Prometheus directly:

curl 'http://localhost:9090/api/v1/query?query=agent_tool_calls_total' | jq

A non-empty result array means metrics are flowing end to end — OTLP export → collector → Prometheus scrape. From there:

• curl localhost:3000/drift — the live (non-dry-run) drift detector reads exactly this data; a real currentWindowStats object back means the whole pull path works.
• Prometheus's own UI (http://localhost:9090/graph) — graph agent_tool_calls_total, agent_tool_duration_bucket, or agent_tokens_total directly.

If agent_tool_calls_total never appears, it's almost always the push side: confirm OTEL_EXPORTER_OTLP_ENDPOINT points at the collector (not Prometheus itself), and that the collector's own logs show incoming OTLP requests.

What DriftWatch emits

Traces — the full decision chain

Every /run opens one root span, agent.run, with child spans for each model step and tool call:

agent.run              task id, skills used, token spend, provider, model
├─ gen_ai.step         one per LLM call — model, tokens, finish reason
├─ tool.get_weather    tool latency + outcome (ok/error)
└─ gen_ai.step

Open any agent.run span and read agent.task_id to pull that exact task's whole trace — every tool it invoked and every token it spent. This is your answer to "why did this one request behave this way?"

Metrics — aggregate behavior over time

Three custom metrics drive drift detection and make good dashboard panels. They are low-cardinality by design (labels are bounded sets — tool name, outcome, provider, model — never a request or task id), so they chart cleanly over time.

Metrics export with cumulative temporality (the OTel default) — this is what PromQL's increase()/rate() expect: an ever-growing counter between scrapes, not a per-export delta. PrometheusMetricsSource builds exactly those queries under the hood.

Building dashboard panels

In Prometheus's own UI, Grafana, or any PromQL-speaking dashboard tool:

• Calls per tool — sum by (tool) (increase(agent_tool_calls_total[5m])). Add outcome to split success vs error.
• Token spend by model — sum by (model) (increase(agent_tokens_total[1h])) (or group by function_id to separate agent work from drift-judge cost).
• Tool latency p95 — histogram_quantile(0.95, sum by (le, tool) (rate(agent_tool_duration_bucket[5m]))).

These are the same signals detectBehavioralDrift reads — the dashboard shows you what changed; the drift verdict tells you whether it matters.

Bring your own metrics backend

detectBehavioralDrift doesn't know about Prometheus at all — it depends on the MetricsQuerySource interface (packages/sdk/src/drift/metrics-source.ts), the same "bring your own concrete implementation" shape as StateStore/Notifier. PrometheusMetricsSource is the SDK's one built-in implementation. Targeting Datadog, Honeycomb, ClickHouse directly, or anything else instead? Implement MetricsQuerySource yourself and pass it as metricsQuerySource — traces and logs keep working over plain OTLP regardless of what you choose for the query side.

Troubleshooting