diff --git a/OpenTelemetryPlan/05-configuration-reference.md b/OpenTelemetryPlan/05-configuration-reference.md index cdf8b798f8..a48e410b49 100644 --- a/OpenTelemetryPlan/05-configuration-reference.md +++ b/OpenTelemetryPlan/05-configuration-reference.md @@ -33,7 +33,7 @@ The authoritative `[telemetry]` example lives in `cfg/xrpld-example.cfg`. Teleme | `trace_ledger` | bool | `true` | Enable ledger tracing | | `tx_trace_strategy` | string | `"deterministic"` | TX trace ID strategy: `"deterministic"` (trace_id = txHash[0:16]) or `"attribute"` (random) | | `consensus_trace_strategy` | string | `"deterministic"` | Consensus trace ID strategy: `"deterministic"` (trace_id = prevLedgerHash[0:16]) or `"attribute"` (random) | -| `service_name` | string | `"xrpld"` | Service name for traces | +| `service_name` | string | `"xrpld"` | Service name (`service.name`) for traces and metrics | | `service_instance_id` | string | `` | Instance identifier | **Planned (not yet implemented)**: the following options appear in the design diff --git a/OpenTelemetryPlan/07-observability-backends.md b/OpenTelemetryPlan/07-observability-backends.md index d397fcb69d..cb17d2817a 100644 --- a/OpenTelemetryPlan/07-observability-backends.md +++ b/OpenTelemetryPlan/07-observability-backends.md @@ -141,7 +141,7 @@ flowchart TB **Reading the diagram:** - **Validator / Stock Nodes**: All xrpld nodes emit trace data via OTLP. Validators and stock nodes are grouped separately because they may reside in different network zones. -- **Collector Cluster (DC1, DC2)**: Regional collectors receive OTLP from nodes in their datacenter, apply processing (sampling, enrichment), and fan out to multiple backends. +- **Collector Cluster (DC1, DC2)**: Regional collectors receive OTLP from nodes in their datacenter, apply processing (sampling, enrichment), and fan out to multiple backends. Enrichment includes deployment-tier tagging: each collector stamps `deployment.environment` and (as a fallback) `xrpl.network.type` so one Grafana stack can filter data from many collectors by tier. - **Storage Backends**: Tempo and Elastic provide queryable trace storage; S3/GCS Archive provides long-term cold storage for compliance or post-incident analysis. - **Grafana Dashboards**: The single visualization layer that queries both Tempo and Elastic, giving operators a unified view of all traces. - **Data flow direction**: Nodes → Collectors → Storage → Grafana. Each arrow represents a network hop; minimizing collector-to-backend hops reduces latency. diff --git a/OpenTelemetryPlan/09-data-collection-reference.md b/OpenTelemetryPlan/09-data-collection-reference.md index eef2d92a98..ed52b58da6 100644 --- a/OpenTelemetryPlan/09-data-collection-reference.md +++ b/OpenTelemetryPlan/09-data-collection-reference.md @@ -611,7 +611,22 @@ For each of the 45+ overlay traffic categories (defined in `TrafficCount.h`), fo | Overlay Traffic Detail | `xrpld-system-overlay-detail` | Prometheus (OTLP) | Squelch, overhead, validator lists, set get/share, have/requested tx, proof paths | | Ledger Data & Sync | `xrpld-system-ledger-sync` | Prometheus (OTLP) | Ledger data exchange, legacy ledger share/get, getobject by type, traffic heatmap | -### 3.3 Accessing the Dashboards +### 3.3 Deployment-Tier Template Variables + +Every dashboard carries four filtering template variables (each variable name +matches its Prometheus label), letting one Grafana stack be sliced by tier: + +| Variable | Source label | Description | +| ------------------------- | ------------------------ | ------------------------------------------------------------ | +| `$node` | `exported_instance` | Filter by xrpld node instance | +| `$service_name` | `service_name` | Filter by service (`service.name`, e.g. `xrpld`) | +| `$deployment_environment` | `deployment_environment` | Filter by deployment tier (`local` / `test` / `ci` / `prod`) | +| `$xrpl_network_type` | `xrpl_network_type` | Filter by network (`mainnet` / `testnet` / `devnet`) | + +See [telemetry-runbook.md](../docs/telemetry-runbook.md) "Deployment Tiers" +for how the tier attributes are set and reach metrics. + +### 3.4 Accessing the Dashboards 1. Open Grafana at **http://localhost:3000** 2. Navigate to **Dashboards → xrpld** folder diff --git a/docker/telemetry/TESTING.md b/docker/telemetry/TESTING.md index ea610e43ad..b54e952e6e 100644 --- a/docker/telemetry/TESTING.md +++ b/docker/telemetry/TESTING.md @@ -444,6 +444,12 @@ curl -s "$PROM/api/v1/query?query=traces_span_metrics_duration_milliseconds_coun # RPC calls by command curl -s "$PROM/api/v1/query?query=traces_span_metrics_calls_total{span_name=~\"rpc.command.*\"}" | jq '.data.result[] | {command: .metric["xrpl.rpc.command"], count: .value[1]}' + +# Deployment-tier labels present on metrics (set by the collector's +# resource/tier processor and promoted via resource_to_telemetry_conversion). +# Expect deployment_environment and xrpl_network_type on each series. +curl -s "$PROM/api/v1/query?query=traces_span_metrics_calls_total" | + jq '.data.result[0].metric | {deployment_environment, xrpl_network_type, service_name}' ``` ### Grafana diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index de71740215..e613c6aeca 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -625,6 +625,90 @@ These gauges are exported via the OTel Metrics SDK `PeriodicMetricReader` (10s i | `xrpld_pathfind_fast` | PathRequests.h:23 | Fast pathfinding duration (ms) | | `xrpld_pathfind_full` | PathRequests.h:24 | Full pathfinding duration (ms) | +## Deployment Tiers + +Multiple xrpld instances can send telemetry to per-tier collectors that all +forward to one Grafana stack. Four resource attributes segregate the data so +one dashboard set serves every deployment: + +| Dimension | Attribute | Set by | Example values | +| ----------- | ------------------------ | ---------- | ------------------------------ | +| Node | `service.instance.id` | xrpld cfg | `alice-laptop`, `ci-runner-7` | +| Service | `service.name` | xrpld cfg | `xrpld`, `pratik-xrpld` | +| Network | `xrpl.network.type` | xrpld node | `mainnet`, `testnet`, `devnet` | +| Environment | `deployment.environment` | collector | `local`, `test`, `ci`, `prod` | + +Dashboards expose these as the template variables `$node`, `$service_name`, +`$xrpl_network_type`, and `$deployment_environment` (each variable name +matches its Prometheus label). Select them top-down — environment → network +→ service → node. Selecting **All** matches every value, including series +lacking the label, so mixed old/new data never disappears. + +### Who owns which attribute + +- **Node and service** come from xrpld config (`service_instance_id`, + `service_name`). Unique per process. +- **Network** is a property of the chain the node joined; the node derives it + from `[network_id]` and stamps `xrpl.network.type` on all three signals. +- **Environment** is a property of where the collector runs; each collector + serves one environment and stamps it. + +### The upsert vs insert rule + +The collector's `resource/tier` processor uses two actions on purpose: + +- `deployment.environment` → **`upsert`** (overwrite). The collector _is_ the + environment, so it is authoritative. +- `xrpl.network.type` → **`insert`** (fill only if absent). The node knows + its real network, so the collector must not overwrite it — `insert` only + supplies a value when the source did not (e.g. an older xrpld build). This + is what lets a local node connected to mainnet report `network=mainnet`, + not the collector's default. + +### Configuring a collector for a tier + +Each tier runs its own collector. Set the two values in the `resource/tier` +processor of the collector config (`otel-collector-config.yaml` for local +backends, `otel-collector-config.grafanacloud.yaml` for Grafana Cloud): + +```yaml +processors: + resource/tier: + attributes: + - key: deployment.environment + value: # local | test | ci | prod + action: upsert + - key: xrpl.network.type + value: # mainnet | testnet | devnet (fallback only) + action: insert +``` + +Suggested per-tier values: + +| Collector | `deployment.environment` | `xrpl.network.type` (fallback) | +| ------------------- | ------------------------ | ------------------------------ | +| Developer laptop | `local` | `devnet` | +| Test machines | `test` | `testnet` | +| CI runs | `ci` | `testnet` | +| Production observer | `prod` | `mainnet` | + +The `xrpl.network.type` value is only a fallback: when the node stamps its +own network (all current builds do), the node's value wins. Set it to the +network the collector most commonly serves. + +### How the tier labels reach metrics + +Resource attributes do not become Prometheus labels automatically. Two +collector settings make it work, both already enabled: + +- `prometheus.resource_to_telemetry_conversion: enabled: true` promotes + resource attributes to metric labels on the local scrape surface. +- `spanmetrics.resource_metrics_key_attributes` lists the tier attributes so + span-derived series stay grouped per node and tier. + +Traces and logs carry resource attributes natively; Grafana Cloud ingests all +three signals' attributes over OTLP directly. + ## Grafana Dashboards Ten dashboards are pre-provisioned in `docker/telemetry/grafana/dashboards/`: