Merge branch 'pratik/otel-phase8-log-correlation' into pratik/otel-phase9-metric-gap-fill

2026-07-06 23:00:20 +00:00 · 2026-07-03 20:01:54 +01:00
parent 55404b7011 93f776fda4
commit 84eb08eb8c
5 changed files with 108 additions and 3 deletions
--- a/OpenTelemetryPlan/05-configuration-reference.md
+++ b/OpenTelemetryPlan/05-configuration-reference.md
@@ -33,7 +33,7 @@ The authoritative `[telemetry]` example lives in `cfg/xrpld-example.cfg`. Teleme
 | `trace_ledger`             | bool   | `true`                            | Enable ledger tracing                                                                                      |
 | `tx_trace_strategy`        | string | `"deterministic"`                 | TX trace ID strategy: `"deterministic"` (trace_id = txHash[0:16]) or `"attribute"` (random)                |
 | `consensus_trace_strategy` | string | `"deterministic"`                 | Consensus trace ID strategy: `"deterministic"` (trace_id = prevLedgerHash[0:16]) or `"attribute"` (random) |
-| `service_name`             | string | `"xrpld"`                         | Service name for traces                                                                                    |
+| `service_name`             | string | `"xrpld"`                         | Service name (`service.name`) for traces and metrics                                                       |
 | `service_instance_id`      | string | `<node_pubkey>`                   | Instance identifier                                                                                        |

 **Planned (not yet implemented)**: the following options appear in the design
--- a/OpenTelemetryPlan/07-observability-backends.md
+++ b/OpenTelemetryPlan/07-observability-backends.md
@@ -141,7 +141,7 @@ flowchart TB
 **Reading the diagram:**

 - **Validator / Stock Nodes**: All xrpld nodes emit trace data via OTLP. Validators and stock nodes are grouped separately because they may reside in different network zones.
- **Collector Cluster (DC1, DC2)**: Regional collectors receive OTLP from nodes in their datacenter, apply processing (sampling, enrichment), and fan out to multiple backends.
+- **Collector Cluster (DC1, DC2)**: Regional collectors receive OTLP from nodes in their datacenter, apply processing (sampling, enrichment), and fan out to multiple backends. Enrichment includes deployment-tier tagging: each collector stamps `deployment.environment` and (as a fallback) `xrpl.network.type` so one Grafana stack can filter data from many collectors by tier.
 - **Storage Backends**: Tempo and Elastic provide queryable trace storage; S3/GCS Archive provides long-term cold storage for compliance or post-incident analysis.
 - **Grafana Dashboards**: The single visualization layer that queries both Tempo and Elastic, giving operators a unified view of all traces.
 - **Data flow direction**: Nodes → Collectors → Storage → Grafana. Each arrow represents a network hop; minimizing collector-to-backend hops reduces latency.
--- a/OpenTelemetryPlan/09-data-collection-reference.md
+++ b/OpenTelemetryPlan/09-data-collection-reference.md
@@ -611,7 +611,22 @@ For each of the 45+ overlay traffic categories (defined in `TrafficCount.h`), fo
 | Overlay Traffic Detail | `xrpld-system-overlay-detail` | Prometheus (OTLP) | Squelch, overhead, validator lists, set get/share, have/requested tx, proof paths |
 | Ledger Data & Sync     | `xrpld-system-ledger-sync`    | Prometheus (OTLP) | Ledger data exchange, legacy ledger share/get, getobject by type, traffic heatmap |

-### 3.3 Accessing the Dashboards
+### 3.3 Deployment-Tier Template Variables
+
+Every dashboard carries four filtering template variables (each variable name
+matches its Prometheus label), letting one Grafana stack be sliced by tier:
+
+| Variable                  | Source label             | Description                                                  |
+| ------------------------- | ------------------------ | ------------------------------------------------------------ |
+| `$node`                   | `exported_instance`      | Filter by xrpld node instance                                |
+| `$service_name`           | `service_name`           | Filter by service (`service.name`, e.g. `xrpld`)             |
+| `$deployment_environment` | `deployment_environment` | Filter by deployment tier (`local` / `test` / `ci` / `prod`) |
+| `$xrpl_network_type`      | `xrpl_network_type`      | Filter by network (`mainnet` / `testnet` / `devnet`)         |
+
+See [telemetry-runbook.md](../docs/telemetry-runbook.md) "Deployment Tiers"
+for how the tier attributes are set and reach metrics.
+
+### 3.4 Accessing the Dashboards

 1. Open Grafana at **http://localhost:3000**
 2. Navigate to **Dashboards → xrpld** folder
--- a/docker/telemetry/TESTING.md
+++ b/docker/telemetry/TESTING.md
@@ -444,6 +444,12 @@ curl -s "$PROM/api/v1/query?query=traces_span_metrics_duration_milliseconds_coun
 # RPC calls by command
 curl -s "$PROM/api/v1/query?query=traces_span_metrics_calls_total{span_name=~\"rpc.command.*\"}" |
    jq '.data.result[] | {command: .metric["xrpl.rpc.command"], count: .value[1]}'
+
+# Deployment-tier labels present on metrics (set by the collector's
+# resource/tier processor and promoted via resource_to_telemetry_conversion).
+# Expect deployment_environment and xrpl_network_type on each series.
+curl -s "$PROM/api/v1/query?query=traces_span_metrics_calls_total" |
+    jq '.data.result[0].metric | {deployment_environment, xrpl_network_type, service_name}'
 ```

 ### Grafana
--- a/docs/telemetry-runbook.md
+++ b/docs/telemetry-runbook.md
@@ -625,6 +625,90 @@ These gauges are exported via the OTel Metrics SDK `PeriodicMetricReader` (10s i
 | `xrpld_pathfind_fast` | PathRequests.h:23     | Fast pathfinding duration (ms) |
 | `xrpld_pathfind_full` | PathRequests.h:24     | Full pathfinding duration (ms) |

+## Deployment Tiers
+
+Multiple xrpld instances can send telemetry to per-tier collectors that all
+forward to one Grafana stack. Four resource attributes segregate the data so
+one dashboard set serves every deployment:
+
+| Dimension   | Attribute                | Set by     | Example values                 |
+| ----------- | ------------------------ | ---------- | ------------------------------ |
+| Node        | `service.instance.id`    | xrpld cfg  | `alice-laptop`, `ci-runner-7`  |
+| Service     | `service.name`           | xrpld cfg  | `xrpld`, `pratik-xrpld`        |
+| Network     | `xrpl.network.type`      | xrpld node | `mainnet`, `testnet`, `devnet` |
+| Environment | `deployment.environment` | collector  | `local`, `test`, `ci`, `prod`  |
+
+Dashboards expose these as the template variables `$node`, `$service_name`,
+`$xrpl_network_type`, and `$deployment_environment` (each variable name
+matches its Prometheus label). Select them top-down — environment → network
+→ service → node. Selecting **All** matches every value, including series
+lacking the label, so mixed old/new data never disappears.
+
+### Who owns which attribute
+
+- **Node and service** come from xrpld config (`service_instance_id`,
+  `service_name`). Unique per process.
+- **Network** is a property of the chain the node joined; the node derives it
+  from `[network_id]` and stamps `xrpl.network.type` on all three signals.
+- **Environment** is a property of where the collector runs; each collector
+  serves one environment and stamps it.
+
+### The upsert vs insert rule
+
+The collector's `resource/tier` processor uses two actions on purpose:
+
+- `deployment.environment` → **`upsert`** (overwrite). The collector _is_ the
+  environment, so it is authoritative.
+- `xrpl.network.type` → **`insert`** (fill only if absent). The node knows
+  its real network, so the collector must not overwrite it — `insert` only
+  supplies a value when the source did not (e.g. an older xrpld build). This
+  is what lets a local node connected to mainnet report `network=mainnet`,
+  not the collector's default.
+
+### Configuring a collector for a tier
+
+Each tier runs its own collector. Set the two values in the `resource/tier`
+processor of the collector config (`otel-collector-config.yaml` for local
+backends, `otel-collector-config.grafanacloud.yaml` for Grafana Cloud):
+
+```yaml
+processors:
+  resource/tier:
+    attributes:
+      - key: deployment.environment
+        value: <tier> # local | test | ci | prod
+        action: upsert
+      - key: xrpl.network.type
+        value: <network> # mainnet | testnet | devnet (fallback only)
+        action: insert
+```
+
+Suggested per-tier values:
+
+| Collector           | `deployment.environment` | `xrpl.network.type` (fallback) |
+| ------------------- | ------------------------ | ------------------------------ |
+| Developer laptop    | `local`                  | `devnet`                       |
+| Test machines       | `test`                   | `testnet`                      |
+| CI runs             | `ci`                     | `testnet`                      |
+| Production observer | `prod`                   | `mainnet`                      |
+
+The `xrpl.network.type` value is only a fallback: when the node stamps its
+own network (all current builds do), the node's value wins. Set it to the
+network the collector most commonly serves.
+
+### How the tier labels reach metrics
+
+Resource attributes do not become Prometheus labels automatically. Two
+collector settings make it work, both already enabled:
+
+- `prometheus.resource_to_telemetry_conversion: enabled: true` promotes
+  resource attributes to metric labels on the local scrape surface.
+- `spanmetrics.resource_metrics_key_attributes` lists the tier attributes so
+  span-derived series stay grouped per node and tier.
+
+Traces and logs carry resource attributes natively; Grafana Cloud ingests all
+three signals' attributes over OTLP directly.
+
 ## Grafana Dashboards

 Ten dashboards are pre-provisioned in `docker/telemetry/grafana/dashboards/`: