From d2c7a0058450bcc4a569bf332e698d2867d39e2e Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Fri, 3 Jul 2026 19:53:54 +0100 Subject: [PATCH 1/3] docs(telemetry): note service.name on metrics and collector tier tagging - 05-configuration-reference: service_name now applies to metrics as well as traces (service.name resource attribute). - 07-observability-backends: document that collector enrichment includes deployment-tier tagging (deployment.environment + fallback xrpl.network.type) for filtering one Grafana stack by tier. Co-Authored-By: Claude Opus 4.8 --- .../05-configuration-reference.md | 32 +++++++++---------- .../07-observability-backends.md | 2 +- 2 files changed, 17 insertions(+), 17 deletions(-) diff --git a/OpenTelemetryPlan/05-configuration-reference.md b/OpenTelemetryPlan/05-configuration-reference.md index c08d0510c9..a81add88d7 100644 --- a/OpenTelemetryPlan/05-configuration-reference.md +++ b/OpenTelemetryPlan/05-configuration-reference.md @@ -15,22 +15,22 @@ The authoritative `[telemetry]` example lives in `cfg/xrpld-example.cfg`. Teleme ### 5.1.2 Configuration Options Summary -| Option | Type | Default | Description | -| --------------------- | ------ | --------------------------------- | ----------------------------------------- | -| `enabled` | bool | `false` | Enable/disable telemetry | -| `endpoint` | string | `http://localhost:4318/v1/traces` | OTLP/HTTP collector endpoint | -| `use_tls` | bool | `false` | Enable TLS for exporter connection | -| `tls_ca_cert` | string | `""` | Path to CA certificate file | -| `batch_size` | uint | `512` | Spans per export batch | -| `batch_delay_ms` | uint | `5000` | Max delay before sending batch (ms) | -| `max_queue_size` | uint | `2048` | Maximum queued spans | -| `trace_transactions` | bool | `true` | Enable transaction tracing | -| `trace_consensus` | bool | `true` | Enable consensus tracing | -| `trace_rpc` | bool | `true` | Enable RPC tracing | -| `trace_peer` | bool | `true` | Enable peer message tracing (high volume) | -| `trace_ledger` | bool | `true` | Enable ledger tracing | -| `service_name` | string | `"xrpld"` | Service name for traces | -| `service_instance_id` | string | `` | Instance identifier | +| Option | Type | Default | Description | +| --------------------- | ------ | --------------------------------- | ---------------------------------------------------- | +| `enabled` | bool | `false` | Enable/disable telemetry | +| `endpoint` | string | `http://localhost:4318/v1/traces` | OTLP/HTTP collector endpoint | +| `use_tls` | bool | `false` | Enable TLS for exporter connection | +| `tls_ca_cert` | string | `""` | Path to CA certificate file | +| `batch_size` | uint | `512` | Spans per export batch | +| `batch_delay_ms` | uint | `5000` | Max delay before sending batch (ms) | +| `max_queue_size` | uint | `2048` | Maximum queued spans | +| `trace_transactions` | bool | `true` | Enable transaction tracing | +| `trace_consensus` | bool | `true` | Enable consensus tracing | +| `trace_rpc` | bool | `true` | Enable RPC tracing | +| `trace_peer` | bool | `true` | Enable peer message tracing (high volume) | +| `trace_ledger` | bool | `true` | Enable ledger tracing | +| `service_name` | string | `"xrpld"` | Service name (`service.name`) for traces and metrics | +| `service_instance_id` | string | `` | Instance identifier | **Planned (not yet implemented)**: the following options appear in the design documents but are not parsed by `TelemetryConfig.cpp` in Phase 1b and later diff --git a/OpenTelemetryPlan/07-observability-backends.md b/OpenTelemetryPlan/07-observability-backends.md index d397fcb69d..cb17d2817a 100644 --- a/OpenTelemetryPlan/07-observability-backends.md +++ b/OpenTelemetryPlan/07-observability-backends.md @@ -141,7 +141,7 @@ flowchart TB **Reading the diagram:** - **Validator / Stock Nodes**: All xrpld nodes emit trace data via OTLP. Validators and stock nodes are grouped separately because they may reside in different network zones. -- **Collector Cluster (DC1, DC2)**: Regional collectors receive OTLP from nodes in their datacenter, apply processing (sampling, enrichment), and fan out to multiple backends. +- **Collector Cluster (DC1, DC2)**: Regional collectors receive OTLP from nodes in their datacenter, apply processing (sampling, enrichment), and fan out to multiple backends. Enrichment includes deployment-tier tagging: each collector stamps `deployment.environment` and (as a fallback) `xrpl.network.type` so one Grafana stack can filter data from many collectors by tier. - **Storage Backends**: Tempo and Elastic provide queryable trace storage; S3/GCS Archive provides long-term cold storage for compliance or post-incident analysis. - **Grafana Dashboards**: The single visualization layer that queries both Tempo and Elastic, giving operators a unified view of all traces. - **Data flow direction**: Nodes → Collectors → Storage → Grafana. Each arrow represents a network hop; minimizing collector-to-backend hops reduces latency. From 4c46d3995550eaeef3a7d57bf5325a497d418ccc Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Fri, 3 Jul 2026 19:54:42 +0100 Subject: [PATCH 2/3] docs(telemetry): document deployment-tier filtering in runbook Add a Deployment Tiers section covering the four filtering dimensions (node, service, network, environment), who owns each attribute, the collector's upsert-vs-insert rule, per-tier collector values, and how the tier labels reach metrics. This is the operator-facing home for the deployment-tier feature. Co-Authored-By: Claude Opus 4.8 --- docs/telemetry-runbook.md | 84 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 84 insertions(+) diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index ffc43d17fd..02fa709735 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -391,6 +391,90 @@ Configured in `otel-collector-config.yaml`: 1ms, 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 5s ``` +## Deployment Tiers + +Multiple xrpld instances can send telemetry to per-tier collectors that all +forward to one Grafana stack. Four resource attributes segregate the data so +one dashboard set serves every deployment: + +| Dimension | Attribute | Set by | Example values | +| ----------- | ------------------------ | ---------- | ------------------------------ | +| Node | `service.instance.id` | xrpld cfg | `alice-laptop`, `ci-runner-7` | +| Service | `service.name` | xrpld cfg | `xrpld`, `pratik-xrpld` | +| Network | `xrpl.network.type` | xrpld node | `mainnet`, `testnet`, `devnet` | +| Environment | `deployment.environment` | collector | `local`, `test`, `ci`, `prod` | + +Dashboards expose these as the template variables `$node`, `$service_name`, +`$xrpl_network_type`, and `$deployment_environment` (each variable name +matches its Prometheus label). Select them top-down — environment → network +→ service → node. Selecting **All** matches every value, including series +lacking the label, so mixed old/new data never disappears. + +### Who owns which attribute + +- **Node and service** come from xrpld config (`service_instance_id`, + `service_name`). Unique per process. +- **Network** is a property of the chain the node joined; the node derives it + from `[network_id]` and stamps `xrpl.network.type` on all three signals. +- **Environment** is a property of where the collector runs; each collector + serves one environment and stamps it. + +### The upsert vs insert rule + +The collector's `resource/tier` processor uses two actions on purpose: + +- `deployment.environment` → **`upsert`** (overwrite). The collector _is_ the + environment, so it is authoritative. +- `xrpl.network.type` → **`insert`** (fill only if absent). The node knows + its real network, so the collector must not overwrite it — `insert` only + supplies a value when the source did not (e.g. an older xrpld build). This + is what lets a local node connected to mainnet report `network=mainnet`, + not the collector's default. + +### Configuring a collector for a tier + +Each tier runs its own collector. Set the two values in the `resource/tier` +processor of the collector config (`otel-collector-config.yaml` for local +backends, `otel-collector-config.grafanacloud.yaml` for Grafana Cloud): + +```yaml +processors: + resource/tier: + attributes: + - key: deployment.environment + value: # local | test | ci | prod + action: upsert + - key: xrpl.network.type + value: # mainnet | testnet | devnet (fallback only) + action: insert +``` + +Suggested per-tier values: + +| Collector | `deployment.environment` | `xrpl.network.type` (fallback) | +| ------------------- | ------------------------ | ------------------------------ | +| Developer laptop | `local` | `devnet` | +| Test machines | `test` | `testnet` | +| CI runs | `ci` | `testnet` | +| Production observer | `prod` | `mainnet` | + +The `xrpl.network.type` value is only a fallback: when the node stamps its +own network (all current builds do), the node's value wins. Set it to the +network the collector most commonly serves. + +### How the tier labels reach metrics + +Resource attributes do not become Prometheus labels automatically. Two +collector settings make it work, both already enabled: + +- `prometheus.resource_to_telemetry_conversion: enabled: true` promotes + resource attributes to metric labels on the local scrape surface. +- `spanmetrics.resource_metrics_key_attributes` lists the tier attributes so + span-derived series stay grouped per node and tier. + +Traces and logs carry resource attributes natively; Grafana Cloud ingests all +three signals' attributes over OTLP directly. + ## Grafana Dashboards Three dashboards are pre-provisioned in `docker/telemetry/grafana/dashboards/`: From f81919af05171192e10097cdab1d5b4066cf3c62 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Fri, 3 Jul 2026 19:56:15 +0100 Subject: [PATCH 3/3] docs(telemetry): document tier template variables and label checks - 09-data-collection-reference: add $service_name, $deployment_environment, $xrpl_network_type to the template-variable table; note they appear on all dashboards and point to the runbook's Deployment Tiers section. - TESTING.md: add a Prometheus check verifying the tier labels (deployment_environment, xrpl_network_type, service_name) are present on metric series. Co-Authored-By: Claude Opus 4.8 --- .../09-data-collection-reference.md | 18 +++++++++++++----- docker/telemetry/TESTING.md | 6 ++++++ 2 files changed, 19 insertions(+), 5 deletions(-) diff --git a/OpenTelemetryPlan/09-data-collection-reference.md b/OpenTelemetryPlan/09-data-collection-reference.md index 7d4555d7d0..92bdc46751 100644 --- a/OpenTelemetryPlan/09-data-collection-reference.md +++ b/OpenTelemetryPlan/09-data-collection-reference.md @@ -580,11 +580,19 @@ The Consensus Health dashboard includes 5 close-time panels added in Phase 4: **Template variables** (Consensus Health dashboard): -| Variable | Source Attribute | Description | -| ----------------------- | ------------------------------------- | ------------------------------------------------------------------------ | -| `$node` | `exported_instance` | Filter by xrpld node instance | -| `$close_time_correct` | `xrpl_consensus_close_time_correct` | Filter by close time correctness (`true` / `false`) | -| `$resolution_direction` | `xrpl_consensus_resolution_direction` | Filter by resolution direction (`increased` / `decreased` / `unchanged`) | +| Variable | Source Attribute | Description | +| ------------------------- | ------------------------------------- | ------------------------------------------------------------------------ | +| `$node` | `exported_instance` | Filter by xrpld node instance | +| `$service_name` | `service_name` | Filter by service (`service.name`, e.g. `xrpld`) | +| `$deployment_environment` | `deployment_environment` | Filter by deployment tier (`local` / `test` / `ci` / `prod`) | +| `$xrpl_network_type` | `xrpl_network_type` | Filter by network (`mainnet` / `testnet` / `devnet`) | +| `$close_time_correct` | `xrpl_consensus_close_time_correct` | Filter by close time correctness (`true` / `false`) | +| `$resolution_direction` | `xrpl_consensus_resolution_direction` | Filter by resolution direction (`increased` / `decreased` / `unchanged`) | + +The `$service_name`, `$deployment_environment`, and `$xrpl_network_type` +variables are present on all dashboards; each variable name matches its +Prometheus label. See [telemetry-runbook.md](../docs/telemetry-runbook.md) +"Deployment Tiers" for how the tier attributes are set and reach metrics. ### 3.4 Accessing the Dashboards diff --git a/docker/telemetry/TESTING.md b/docker/telemetry/TESTING.md index 6fe7bdc732..2a6cfac9f2 100644 --- a/docker/telemetry/TESTING.md +++ b/docker/telemetry/TESTING.md @@ -444,6 +444,12 @@ curl -s "$PROM/api/v1/query?query=traces_span_metrics_duration_milliseconds_coun # RPC calls by command curl -s "$PROM/api/v1/query?query=traces_span_metrics_calls_total{span_name=~\"rpc.command.*\"}" | jq '.data.result[] | {command: .metric["xrpl.rpc.command"], count: .value[1]}' + +# Deployment-tier labels present on metrics (set by the collector's +# resource/tier processor and promoted via resource_to_telemetry_conversion). +# Expect deployment_environment and xrpl_network_type on each series. +curl -s "$PROM/api/v1/query?query=traces_span_metrics_calls_total" | + jq '.data.result[0].metric | {deployment_environment, xrpl_network_type, service_name}' ``` ### Grafana