From 4c46d3995550eaeef3a7d57bf5325a497d418ccc Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Fri, 3 Jul 2026 19:54:42 +0100
Subject: [PATCH] docs(telemetry): document deployment-tier filtering in
 runbook

Add a Deployment Tiers section covering the four filtering dimensions
(node, service, network, environment), who owns each attribute, the
collector's upsert-vs-insert rule, per-tier collector values, and how the
tier labels reach metrics. This is the operator-facing home for the
deployment-tier feature.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---
 docs/telemetry-runbook.md | 84 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 84 insertions(+)

diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md
index ffc43d17fd..02fa709735 100644
--- a/docs/telemetry-runbook.md
+++ b/docs/telemetry-runbook.md
@@ -391,6 +391,90 @@ Configured in `otel-collector-config.yaml`:
 1ms, 5ms, 10ms, 25ms, 50ms, 100ms, 250ms, 500ms, 1s, 5s
 ```
 
+## Deployment Tiers
+
+Multiple xrpld instances can send telemetry to per-tier collectors that all
+forward to one Grafana stack. Four resource attributes segregate the data so
+one dashboard set serves every deployment:
+
+| Dimension   | Attribute                | Set by     | Example values                 |
+| ----------- | ------------------------ | ---------- | ------------------------------ |
+| Node        | `service.instance.id`    | xrpld cfg  | `alice-laptop`, `ci-runner-7`  |
+| Service     | `service.name`           | xrpld cfg  | `xrpld`, `pratik-xrpld`        |
+| Network     | `xrpl.network.type`      | xrpld node | `mainnet`, `testnet`, `devnet` |
+| Environment | `deployment.environment` | collector  | `local`, `test`, `ci`, `prod`  |
+
+Dashboards expose these as the template variables `$node`, `$service_name`,
+`$xrpl_network_type`, and `$deployment_environment` (each variable name
+matches its Prometheus label). Select them top-down — environment → network
+→ service → node. Selecting **All** matches every value, including series
+lacking the label, so mixed old/new data never disappears.
+
+### Who owns which attribute
+
+- **Node and service** come from xrpld config (`service_instance_id`,
+  `service_name`). Unique per process.
+- **Network** is a property of the chain the node joined; the node derives it
+  from `[network_id]` and stamps `xrpl.network.type` on all three signals.
+- **Environment** is a property of where the collector runs; each collector
+  serves one environment and stamps it.
+
+### The upsert vs insert rule
+
+The collector's `resource/tier` processor uses two actions on purpose:
+
+- `deployment.environment` → **`upsert`** (overwrite). The collector _is_ the
+  environment, so it is authoritative.
+- `xrpl.network.type` → **`insert`** (fill only if absent). The node knows
+  its real network, so the collector must not overwrite it — `insert` only
+  supplies a value when the source did not (e.g. an older xrpld build). This
+  is what lets a local node connected to mainnet report `network=mainnet`,
+  not the collector's default.
+
+### Configuring a collector for a tier
+
+Each tier runs its own collector. Set the two values in the `resource/tier`
+processor of the collector config (`otel-collector-config.yaml` for local
+backends, `otel-collector-config.grafanacloud.yaml` for Grafana Cloud):
+
+```yaml
+processors:
+  resource/tier:
+    attributes:
+      - key: deployment.environment
+        value: <tier> # local | test | ci | prod
+        action: upsert
+      - key: xrpl.network.type
+        value: <network> # mainnet | testnet | devnet (fallback only)
+        action: insert
+```
+
+Suggested per-tier values:
+
+| Collector           | `deployment.environment` | `xrpl.network.type` (fallback) |
+| ------------------- | ------------------------ | ------------------------------ |
+| Developer laptop    | `local`                  | `devnet`                       |
+| Test machines       | `test`                   | `testnet`                      |
+| CI runs             | `ci`                     | `testnet`                      |
+| Production observer | `prod`                   | `mainnet`                      |
+
+The `xrpl.network.type` value is only a fallback: when the node stamps its
+own network (all current builds do), the node's value wins. Set it to the
+network the collector most commonly serves.
+
+### How the tier labels reach metrics
+
+Resource attributes do not become Prometheus labels automatically. Two
+collector settings make it work, both already enabled:
+
+- `prometheus.resource_to_telemetry_conversion: enabled: true` promotes
+  resource attributes to metric labels on the local scrape surface.
+- `spanmetrics.resource_metrics_key_attributes` lists the tier attributes so
+  span-derived series stay grouped per node and tier.
+
+Traces and logs carry resource attributes natively; Grafana Cloud ingests all
+three signals' attributes over OTLP directly.
+
 ## Grafana Dashboards
 
 Three dashboards are pre-provisioned in `docker/telemetry/grafana/dashboards/`: