feat(telemetry): add runtime-selectable Grafana Cloud export

Add an opt-in collector config + compose override that forward traces, metrics, and logs to a Grafana Cloud OTLP gateway alongside the local backends. Enabling it is a runtime choice (layer the override compose file) — no xrpld rebuild, base stack unchanged. Credentials come from a gitignored .env file. Documented in the telemetry runbook and TESTING.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-31 02:50:24 +00:00 · 2026-06-26 17:34:29 +01:00
parent 620396497f
commit 6e71ee1888
6 changed files with 304 additions and 0 deletions
--- a/docker/telemetry/.env.grafanacloud.example
+++ b/docker/telemetry/.env.grafanacloud.example
@@ -0,0 +1,16 @@
+# Grafana Cloud OTLP credentials — copy to `.env.grafanacloud` and fill in.
+# Find all three under Grafana Cloud -> Connections -> "OpenTelemetry (OTLP)".
+#
+# `.env.grafanacloud` is gitignored; never commit real tokens.
+
+# OTLP/HTTP gateway URL for your stack, including the /otlp path.
+# Example: https://otlp-gateway-prod-us-east-0.grafana.net/otlp
+GRAFANA_CLOUD_OTLP_ENDPOINT=
+
+# Numeric instance/stack id shown on the OTLP connection page
+# (used as the Basic-auth username).
+GRAFANA_CLOUD_INSTANCE_ID=
+
+# A Cloud Access Policy token with metrics:write, traces:write, logs:write
+# (used as the Basic-auth password).
+GRAFANA_CLOUD_API_TOKEN=
--- a/docker/telemetry/.gitignore
+++ b/docker/telemetry/.gitignore
@@ -1,2 +1,6 @@
 # Runtime data generated by xrpld and telemetry stack
 data/
+
+# Grafana Cloud OTLP secrets (never commit). The .example template IS tracked.
+.env.grafanacloud
+!.env.grafanacloud.example
--- a/docker/telemetry/TESTING.md
+++ b/docker/telemetry/TESTING.md
@@ -466,6 +466,65 @@ Pre-configured datasources:

 ---

+## Exporting to Grafana Cloud
+
+Instead of (or alongside) the local backends, the collector can forward
+traces, metrics, and logs to a hosted **Grafana Cloud** stack. This is a
+runtime choice layered on top of the base stack — xrpld and the base
+`docker-compose.yml` are unchanged.
+
+### Step 1: Get Grafana Cloud OTLP credentials
+
+From **Grafana Cloud → Connections → OpenTelemetry (OTLP)**, note the OTLP
+gateway endpoint (ends in `/otlp`), the numeric instance id, and an
+access-policy token with `metrics:write`, `traces:write`, and `logs:write`.
+
+### Step 2: Fill in the env file
+
+```bash
+cp docker/telemetry/.env.grafanacloud.example docker/telemetry/.env.grafanacloud
+# edit .env.grafanacloud:
+#   GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-<zone>.grafana.net/otlp
+#   GRAFANA_CLOUD_INSTANCE_ID=<instance id>
+#   GRAFANA_CLOUD_API_TOKEN=<token>
+```
+
+`.env.grafanacloud` is gitignored — never commit real tokens.
+
+### Step 3: Start the stack with cloud export enabled
+
+```bash
+docker compose -f docker/telemetry/docker-compose.yml \
+    -f docker/telemetry/docker-compose.grafanacloud.yaml up -d
+```
+
+The override swaps the collector onto `otel-collector-config.grafanacloud.yaml`,
+which keeps the local Tempo/Prometheus/Loki exporters and adds one
+OTLP/HTTP exporter to Grafana Cloud on all three pipelines. Bring the stack
+up with just the base file to return to local-only.
+
+### Step 4: Verify data reaches Grafana Cloud
+
+After exercising RPC/transaction workflows (Tests 1 or 2), open your Grafana
+Cloud instance and confirm:
+
+- **Traces**: Explore → hosted Tempo datasource → search `{resource.service.name="xrpld"}`
+- **Metrics**: Explore → hosted Prometheus/Mimir → query `traces_span_metrics_calls_total`
+- **Logs**: Explore → hosted Loki → query `{job="xrpld"}` (requires `warning`+ file logging)
+
+If nothing appears, check the collector logs for auth/export errors:
+
+```bash
+docker compose -f docker/telemetry/docker-compose.yml \
+    -f docker/telemetry/docker-compose.grafanacloud.yaml \
+    logs otel-collector | grep -iE 'grafanacloud|401|403|export'
+```
+
+A `401`/`403` means the instance id or token is wrong; a connection error
+means the endpoint URL is wrong or missing the `/otlp` path.
+
+---
+
 ## Test 3: Log-Trace Correlation (Phase 8)

 Phase 8 injects `trace_id` and `span_id` into xrpld's log output when
--- a/docker/telemetry/docker-compose.grafanacloud.yaml
+++ b/docker/telemetry/docker-compose.grafanacloud.yaml
@@ -0,0 +1,26 @@
+# Compose override — enable Grafana Cloud export at runtime.
+#
+# Layer this on top of the base stack to swap the collector onto the
+# Grafana Cloud dual-export config and inject the OTLP credentials. The
+# base docker-compose.yml is unchanged, so plain `up` stays local-only.
+#
+# Usage:
+#   1. Put your three Grafana Cloud values in docker/telemetry/.env.grafanacloud
+#      (template: .env.grafanacloud.example). That file is gitignored.
+#   2. Bring the stack up with BOTH compose files:
+#        docker compose -f docker/telemetry/docker-compose.yml \
+#                       -f docker/telemetry/docker-compose.grafanacloud.yaml up -d
+#   3. To go back to local-only, bring the stack up with just the base file.
+
+services:
+  otel-collector:
+    # Mount the Grafana Cloud variant over the default config path. This
+    # replaces the local-only config for this run only — the file on disk
+    # is untouched.
+    volumes:
+      - ./otel-collector-config.grafanacloud.yaml:/etc/otel-collector-config.yaml:ro
+    # Secrets are read from the gitignored env file at compose time and
+    # passed into the container environment, where the collector resolves
+    # them via ${env:...}. Nothing is baked into the image or committed.
+    env_file:
+      - .env.grafanacloud
--- a/docker/telemetry/otel-collector-config.grafanacloud.yaml
+++ b/docker/telemetry/otel-collector-config.grafanacloud.yaml
@@ -0,0 +1,139 @@
+# OpenTelemetry Collector configuration — Grafana Cloud dual-export variant.
+#
+# Identical to otel-collector-config.yaml (local Tempo/Prometheus/Loki) but
+# ALSO ships all three signals to Grafana Cloud over a single OTLP/HTTP
+# endpoint. Local backends are kept so the on-box stack still works as a
+# fallback; remove the local exporters from the pipelines below if you want
+# cloud-only.
+#
+# Selecting this config is a RUNTIME choice — no code or image changes:
+#   Local only (default):
+#     docker compose -f docker/telemetry/docker-compose.yml up -d
+#   Local + Grafana Cloud:
+#     docker compose -f docker/telemetry/docker-compose.yml \
+#                    -f docker/telemetry/docker-compose.grafanacloud.yaml up -d
+#
+# The override file mounts THIS config over the collector's config path and
+# injects the three required secrets as environment variables:
+#   GRAFANA_CLOUD_OTLP_ENDPOINT  e.g. https://otlp-gateway-<zone>.grafana.net/otlp
+#   GRAFANA_CLOUD_INSTANCE_ID    numeric Grafana Cloud stack/instance id
+#   GRAFANA_CLOUD_API_TOKEN      a Cloud Access Policy token with metrics/traces/logs:write
+# Find all three under Grafana Cloud -> Connections -> "OpenTelemetry (OTLP)".
+
+extensions:
+  health_check:
+    endpoint: 0.0.0.0:13133
+  # Basic-auth for the Grafana Cloud OTLP gateway: username = instance id,
+  # password = API token. Values come from the container environment so no
+  # secret is committed to the repo.
+  basicauth/grafanacloud:
+    client_auth:
+      username: ${env:GRAFANA_CLOUD_INSTANCE_ID}
+      password: ${env:GRAFANA_CLOUD_API_TOKEN}
+
+receivers:
+  otlp:
+    protocols:
+      grpc:
+        endpoint: 0.0.0.0:4317
+      http:
+        endpoint: 0.0.0.0:4318
+  filelog:
+    include:
+      - /var/log/rippled/*/debug.log
+      - /var/log/rippled-integration/*/debug.log
+    operators:
+      - type: regex_parser
+        regex: '^(?P<timestamp>\S+\s+\S+)\s+\S+\s+(?:(?P<partition>\S+):)?(?P<severity>\S+)\s+(?:trace_id=(?P<trace_id>[a-f0-9]+)\s+span_id=(?P<span_id>[a-f0-9]+)\s+)?(?P<message>.*)$'
+        timestamp:
+          parse_from: attributes.timestamp
+          layout: "%Y-%b-%d %H:%M:%S.%f"
+          location: UTC
+
+processors:
+  batch:
+    timeout: 1s
+    send_batch_size: 100
+  resource/logs:
+    attributes:
+      - key: service.name
+        value: xrpld
+        action: upsert
+      - key: job
+        value: xrpld
+        action: upsert
+
+connectors:
+  spanmetrics:
+    resource_metrics_key_attributes:
+      - service.instance.id
+    histogram:
+      explicit:
+        buckets: [1ms, 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 5s]
+    dimensions:
+      - name: command
+      - name: rpc_status
+      - name: consensus_mode
+      - name: close_time_correct
+      - name: consensus_state
+      - name: local
+      - name: suppressed
+      - name: proposal_trusted
+      - name: validation_trusted
+      - name: tx_type
+      - name: ter_result
+      - name: stage
+      - name: txq_status
+      - name: load_type
+      - name: is_batch
+      - name: mode_new
+      - name: consensus_stalled
+      - name: consensus_phase
+      - name: consensus_result
+      - name: method
+      - name: grpc_role
+      - name: grpc_status
+      - name: outcome
+      - name: acquire_reason
+
+exporters:
+  debug:
+    verbosity: detailed
+  otlp/tempo:
+    endpoint: tempo:4317
+    tls:
+      insecure: true
+  otlphttp/loki:
+    endpoint: http://loki:3100/otlp
+  prometheus:
+    endpoint: 0.0.0.0:8889
+  # Single OTLP/HTTP exporter to Grafana Cloud. The gateway fans the three
+  # signals out to hosted Tempo (traces), Mimir/Prometheus (metrics), and
+  # Loki (logs). Retry + queue guard against transient gateway errors.
+  otlphttp/grafanacloud:
+    endpoint: ${env:GRAFANA_CLOUD_OTLP_ENDPOINT}
+    auth:
+      authenticator: basicauth/grafanacloud
+    retry_on_failure:
+      enabled: true
+    sending_queue:
+      enabled: true
+
+service:
+  extensions: [health_check, basicauth/grafanacloud]
+  pipelines:
+    # Each pipeline keeps its local exporter(s) AND adds Grafana Cloud.
+    # For cloud-only, drop debug/otlp/tempo, prometheus, and otlphttp/loki
+    # from the respective exporter lists.
+    traces:
+      receivers: [otlp]
+      processors: [batch]
+      exporters: [debug, otlp/tempo, spanmetrics, otlphttp/grafanacloud]
+    metrics:
+      receivers: [otlp, spanmetrics]
+      processors: [batch]
+      exporters: [prometheus, otlphttp/grafanacloud]
+    logs:
+      receivers: [filelog]
+      processors: [resource/logs, batch]
+      exporters: [otlphttp/loki, otlphttp/grafanacloud]
--- a/docs/telemetry-runbook.md
+++ b/docs/telemetry-runbook.md
@@ -64,6 +64,66 @@ cmake --build --preset default
 | `tls_client_cert`          | (empty)                           | Client cert (PEM) for mutual TLS; empty = one-way TLS     |
 | `tls_client_key`           | (empty)                           | Private key (PEM) for `tls_client_cert`                   |

+## Exporting to Grafana Cloud
+
+The collector can ship traces, metrics, and logs to a hosted **Grafana
+Cloud** stack instead of (or alongside) the local Tempo/Prometheus/Loki
+backends. This is a runtime choice — no xrpld rebuild and no change to the
+base stack. xrpld still exports to the local collector exactly as before;
+the collector adds one OTLP/HTTP exporter that forwards all three signals to
+the Grafana Cloud OTLP gateway, which fans them out to hosted Tempo, Mimir,
+and Loki.
+
+### Credentials
+
+Find these under **Grafana Cloud → Connections → OpenTelemetry (OTLP)**:
+
+| Value                         | Used as           | Notes                                              |
+| ----------------------------- | ----------------- | -------------------------------------------------- |
+| `GRAFANA_CLOUD_OTLP_ENDPOINT` | exporter endpoint | Full gateway URL incl. `/otlp` path                |
+| `GRAFANA_CLOUD_INSTANCE_ID`   | Basic-auth user   | Numeric stack/instance id                          |
+| `GRAFANA_CLOUD_API_TOKEN`     | Basic-auth pass   | Access-policy token with `*:write` for all signals |
+
+### Enable
+
+1. Copy the template and fill in the three values:
+
+   ```bash
+   cp docker/telemetry/.env.grafanacloud.example docker/telemetry/.env.grafanacloud
+   # edit .env.grafanacloud — this file is gitignored, never commit tokens
+   ```
+
+2. Bring the stack up with the base file **and** the Grafana Cloud override:
+
+   ```bash
+   docker compose -f docker/telemetry/docker-compose.yml \
+       -f docker/telemetry/docker-compose.grafanacloud.yaml up -d
+   ```
+
+To return to local-only export, bring the stack up with just the base
+`docker-compose.yml`.
+
+### Files
+
+| File                                      | Role                                                                                           |
+| ----------------------------------------- | ---------------------------------------------------------------------------------------------- |
+| `otel-collector-config.grafanacloud.yaml` | Collector config: local backends **plus** a Grafana Cloud OTLP exporter on all three pipelines |
+| `docker-compose.grafanacloud.yaml`        | Override that mounts that config and injects the credentials                                   |
+| `.env.grafanacloud.example`               | Credential template (copy to `.env.grafanacloud`)                                              |
+
+### Local + cloud vs cloud-only
+
+The prepared config **dual-exports**: data goes to both the local stack and
+Grafana Cloud, so the on-box backends remain a fallback. For cloud-only,
+remove the local exporters (`debug`, `otlp/tempo`, `prometheus`,
+`otlphttp/loki`) from the respective pipelines in
+`otel-collector-config.grafanacloud.yaml`, leaving only
+`otlphttp/grafanacloud`.
+
+> **Note**: shipping logs to Grafana Cloud requires keeping xrpld file
+> logging on (at least `warning` level) so the collector's filelog receiver
+> has a `debug.log` to tail. Traces and metrics are unaffected by log level.
+
 ## Span Reference

 All spans instrumented in xrpld, grouped by subsystem: