diff --git a/docker/telemetry/.env.grafanacloud.example b/docker/telemetry/.env.grafanacloud.example new file mode 100644 index 0000000000..53a1a2a7f3 --- /dev/null +++ b/docker/telemetry/.env.grafanacloud.example @@ -0,0 +1,16 @@ +# Grafana Cloud OTLP credentials — copy to `.env.grafanacloud` and fill in. +# Find all three under Grafana Cloud -> Connections -> "OpenTelemetry (OTLP)". +# +# `.env.grafanacloud` is gitignored; never commit real tokens. + +# OTLP/HTTP gateway URL for your stack, including the /otlp path. +# Example: https://otlp-gateway-prod-us-east-0.grafana.net/otlp +GRAFANA_CLOUD_OTLP_ENDPOINT= + +# Numeric instance/stack id shown on the OTLP connection page +# (used as the Basic-auth username). +GRAFANA_CLOUD_INSTANCE_ID= + +# A Cloud Access Policy token with metrics:write, traces:write, logs:write +# (used as the Basic-auth password). +GRAFANA_CLOUD_API_TOKEN= diff --git a/docker/telemetry/.gitignore b/docker/telemetry/.gitignore index bba4819d66..f8c8f42eea 100644 --- a/docker/telemetry/.gitignore +++ b/docker/telemetry/.gitignore @@ -1,2 +1,6 @@ # Runtime data generated by xrpld and telemetry stack data/ + +# Grafana Cloud OTLP secrets (never commit). The .example template IS tracked. +.env.grafanacloud +!.env.grafanacloud.example diff --git a/docker/telemetry/TESTING.md b/docker/telemetry/TESTING.md index 1c943ccf65..192345e141 100644 --- a/docker/telemetry/TESTING.md +++ b/docker/telemetry/TESTING.md @@ -466,6 +466,65 @@ Pre-configured datasources: --- +## Exporting to Grafana Cloud + +Instead of (or alongside) the local backends, the collector can forward +traces, metrics, and logs to a hosted **Grafana Cloud** stack. This is a +runtime choice layered on top of the base stack — xrpld and the base +`docker-compose.yml` are unchanged. + +### Step 1: Get Grafana Cloud OTLP credentials + +From **Grafana Cloud → Connections → OpenTelemetry (OTLP)**, note the OTLP +gateway endpoint (ends in `/otlp`), the numeric instance id, and an +access-policy token with `metrics:write`, `traces:write`, and `logs:write`. + +### Step 2: Fill in the env file + +```bash +cp docker/telemetry/.env.grafanacloud.example docker/telemetry/.env.grafanacloud +# edit .env.grafanacloud: +# GRAFANA_CLOUD_OTLP_ENDPOINT=https://otlp-gateway-.grafana.net/otlp +# GRAFANA_CLOUD_INSTANCE_ID= +# GRAFANA_CLOUD_API_TOKEN= +``` + +`.env.grafanacloud` is gitignored — never commit real tokens. + +### Step 3: Start the stack with cloud export enabled + +```bash +docker compose -f docker/telemetry/docker-compose.yml \ + -f docker/telemetry/docker-compose.grafanacloud.yaml up -d +``` + +The override swaps the collector onto `otel-collector-config.grafanacloud.yaml`, +which keeps the local Tempo/Prometheus/Loki exporters and adds one +OTLP/HTTP exporter to Grafana Cloud on all three pipelines. Bring the stack +up with just the base file to return to local-only. + +### Step 4: Verify data reaches Grafana Cloud + +After exercising RPC/transaction workflows (Tests 1 or 2), open your Grafana +Cloud instance and confirm: + +- **Traces**: Explore → hosted Tempo datasource → search `{resource.service.name="xrpld"}` +- **Metrics**: Explore → hosted Prometheus/Mimir → query `traces_span_metrics_calls_total` +- **Logs**: Explore → hosted Loki → query `{job="xrpld"}` (requires `warning`+ file logging) + +If nothing appears, check the collector logs for auth/export errors: + +```bash +docker compose -f docker/telemetry/docker-compose.yml \ + -f docker/telemetry/docker-compose.grafanacloud.yaml \ + logs otel-collector | grep -iE 'grafanacloud|401|403|export' +``` + +A `401`/`403` means the instance id or token is wrong; a connection error +means the endpoint URL is wrong or missing the `/otlp` path. + +--- + ## Test 3: Log-Trace Correlation (Phase 8) Phase 8 injects `trace_id` and `span_id` into xrpld's log output when diff --git a/docker/telemetry/docker-compose.grafanacloud.yaml b/docker/telemetry/docker-compose.grafanacloud.yaml new file mode 100644 index 0000000000..c2c2fde318 --- /dev/null +++ b/docker/telemetry/docker-compose.grafanacloud.yaml @@ -0,0 +1,26 @@ +# Compose override — enable Grafana Cloud export at runtime. +# +# Layer this on top of the base stack to swap the collector onto the +# Grafana Cloud dual-export config and inject the OTLP credentials. The +# base docker-compose.yml is unchanged, so plain `up` stays local-only. +# +# Usage: +# 1. Put your three Grafana Cloud values in docker/telemetry/.env.grafanacloud +# (template: .env.grafanacloud.example). That file is gitignored. +# 2. Bring the stack up with BOTH compose files: +# docker compose -f docker/telemetry/docker-compose.yml \ +# -f docker/telemetry/docker-compose.grafanacloud.yaml up -d +# 3. To go back to local-only, bring the stack up with just the base file. + +services: + otel-collector: + # Mount the Grafana Cloud variant over the default config path. This + # replaces the local-only config for this run only — the file on disk + # is untouched. + volumes: + - ./otel-collector-config.grafanacloud.yaml:/etc/otel-collector-config.yaml:ro + # Secrets are read from the gitignored env file at compose time and + # passed into the container environment, where the collector resolves + # them via ${env:...}. Nothing is baked into the image or committed. + env_file: + - .env.grafanacloud diff --git a/docker/telemetry/otel-collector-config.grafanacloud.yaml b/docker/telemetry/otel-collector-config.grafanacloud.yaml new file mode 100644 index 0000000000..93672b4368 --- /dev/null +++ b/docker/telemetry/otel-collector-config.grafanacloud.yaml @@ -0,0 +1,139 @@ +# OpenTelemetry Collector configuration — Grafana Cloud dual-export variant. +# +# Identical to otel-collector-config.yaml (local Tempo/Prometheus/Loki) but +# ALSO ships all three signals to Grafana Cloud over a single OTLP/HTTP +# endpoint. Local backends are kept so the on-box stack still works as a +# fallback; remove the local exporters from the pipelines below if you want +# cloud-only. +# +# Selecting this config is a RUNTIME choice — no code or image changes: +# Local only (default): +# docker compose -f docker/telemetry/docker-compose.yml up -d +# Local + Grafana Cloud: +# docker compose -f docker/telemetry/docker-compose.yml \ +# -f docker/telemetry/docker-compose.grafanacloud.yaml up -d +# +# The override file mounts THIS config over the collector's config path and +# injects the three required secrets as environment variables: +# GRAFANA_CLOUD_OTLP_ENDPOINT e.g. https://otlp-gateway-.grafana.net/otlp +# GRAFANA_CLOUD_INSTANCE_ID numeric Grafana Cloud stack/instance id +# GRAFANA_CLOUD_API_TOKEN a Cloud Access Policy token with metrics/traces/logs:write +# Find all three under Grafana Cloud -> Connections -> "OpenTelemetry (OTLP)". + +extensions: + health_check: + endpoint: 0.0.0.0:13133 + # Basic-auth for the Grafana Cloud OTLP gateway: username = instance id, + # password = API token. Values come from the container environment so no + # secret is committed to the repo. + basicauth/grafanacloud: + client_auth: + username: ${env:GRAFANA_CLOUD_INSTANCE_ID} + password: ${env:GRAFANA_CLOUD_API_TOKEN} + +receivers: + otlp: + protocols: + grpc: + endpoint: 0.0.0.0:4317 + http: + endpoint: 0.0.0.0:4318 + filelog: + include: + - /var/log/rippled/*/debug.log + - /var/log/rippled-integration/*/debug.log + operators: + - type: regex_parser + regex: '^(?P\S+\s+\S+)\s+\S+\s+(?:(?P\S+):)?(?P\S+)\s+(?:trace_id=(?P[a-f0-9]+)\s+span_id=(?P[a-f0-9]+)\s+)?(?P.*)$' + timestamp: + parse_from: attributes.timestamp + layout: "%Y-%b-%d %H:%M:%S.%f" + location: UTC + +processors: + batch: + timeout: 1s + send_batch_size: 100 + resource/logs: + attributes: + - key: service.name + value: xrpld + action: upsert + - key: job + value: xrpld + action: upsert + +connectors: + spanmetrics: + resource_metrics_key_attributes: + - service.instance.id + histogram: + explicit: + buckets: [1ms, 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 5s] + dimensions: + - name: command + - name: rpc_status + - name: consensus_mode + - name: close_time_correct + - name: consensus_state + - name: local + - name: suppressed + - name: proposal_trusted + - name: validation_trusted + - name: tx_type + - name: ter_result + - name: stage + - name: txq_status + - name: load_type + - name: is_batch + - name: mode_new + - name: consensus_stalled + - name: consensus_phase + - name: consensus_result + - name: method + - name: grpc_role + - name: grpc_status + - name: outcome + - name: acquire_reason + +exporters: + debug: + verbosity: detailed + otlp/tempo: + endpoint: tempo:4317 + tls: + insecure: true + otlphttp/loki: + endpoint: http://loki:3100/otlp + prometheus: + endpoint: 0.0.0.0:8889 + # Single OTLP/HTTP exporter to Grafana Cloud. The gateway fans the three + # signals out to hosted Tempo (traces), Mimir/Prometheus (metrics), and + # Loki (logs). Retry + queue guard against transient gateway errors. + otlphttp/grafanacloud: + endpoint: ${env:GRAFANA_CLOUD_OTLP_ENDPOINT} + auth: + authenticator: basicauth/grafanacloud + retry_on_failure: + enabled: true + sending_queue: + enabled: true + +service: + extensions: [health_check, basicauth/grafanacloud] + pipelines: + # Each pipeline keeps its local exporter(s) AND adds Grafana Cloud. + # For cloud-only, drop debug/otlp/tempo, prometheus, and otlphttp/loki + # from the respective exporter lists. + traces: + receivers: [otlp] + processors: [batch] + exporters: [debug, otlp/tempo, spanmetrics, otlphttp/grafanacloud] + metrics: + receivers: [otlp, spanmetrics] + processors: [batch] + exporters: [prometheus, otlphttp/grafanacloud] + logs: + receivers: [filelog] + processors: [resource/logs, batch] + exporters: [otlphttp/loki, otlphttp/grafanacloud] diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index 078f65692a..69dc99fdad 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -64,6 +64,66 @@ cmake --build --preset default | `tls_client_cert` | (empty) | Client cert (PEM) for mutual TLS; empty = one-way TLS | | `tls_client_key` | (empty) | Private key (PEM) for `tls_client_cert` | +## Exporting to Grafana Cloud + +The collector can ship traces, metrics, and logs to a hosted **Grafana +Cloud** stack instead of (or alongside) the local Tempo/Prometheus/Loki +backends. This is a runtime choice — no xrpld rebuild and no change to the +base stack. xrpld still exports to the local collector exactly as before; +the collector adds one OTLP/HTTP exporter that forwards all three signals to +the Grafana Cloud OTLP gateway, which fans them out to hosted Tempo, Mimir, +and Loki. + +### Credentials + +Find these under **Grafana Cloud → Connections → OpenTelemetry (OTLP)**: + +| Value | Used as | Notes | +| ----------------------------- | ----------------- | -------------------------------------------------- | +| `GRAFANA_CLOUD_OTLP_ENDPOINT` | exporter endpoint | Full gateway URL incl. `/otlp` path | +| `GRAFANA_CLOUD_INSTANCE_ID` | Basic-auth user | Numeric stack/instance id | +| `GRAFANA_CLOUD_API_TOKEN` | Basic-auth pass | Access-policy token with `*:write` for all signals | + +### Enable + +1. Copy the template and fill in the three values: + + ```bash + cp docker/telemetry/.env.grafanacloud.example docker/telemetry/.env.grafanacloud + # edit .env.grafanacloud — this file is gitignored, never commit tokens + ``` + +2. Bring the stack up with the base file **and** the Grafana Cloud override: + + ```bash + docker compose -f docker/telemetry/docker-compose.yml \ + -f docker/telemetry/docker-compose.grafanacloud.yaml up -d + ``` + +To return to local-only export, bring the stack up with just the base +`docker-compose.yml`. + +### Files + +| File | Role | +| ----------------------------------------- | ---------------------------------------------------------------------------------------------- | +| `otel-collector-config.grafanacloud.yaml` | Collector config: local backends **plus** a Grafana Cloud OTLP exporter on all three pipelines | +| `docker-compose.grafanacloud.yaml` | Override that mounts that config and injects the credentials | +| `.env.grafanacloud.example` | Credential template (copy to `.env.grafanacloud`) | + +### Local + cloud vs cloud-only + +The prepared config **dual-exports**: data goes to both the local stack and +Grafana Cloud, so the on-box backends remain a fallback. For cloud-only, +remove the local exporters (`debug`, `otlp/tempo`, `prometheus`, +`otlphttp/loki`) from the respective pipelines in +`otel-collector-config.grafanacloud.yaml`, leaving only +`otlphttp/grafanacloud`. + +> **Note**: shipping logs to Grafana Cloud requires keeping xrpld file +> logging on (at least `warning` level) so the collector's filelog receiver +> has a `debug.log` to tail. Traces and metrics are unaffected by log level. + ## Span Reference All spans instrumented in xrpld, grouped by subsystem: