3153f3ef56
The reference docs had drifted from the code in ways that break the reader rather than merely misinform: PromQL examples that return no data, a rollback flag that is a no-op, a sampling knob that does not exist, and two span parents that moved. Code is treated as the truth throughout; where the code is the defective side, the doc now records it as a known issue instead of describing the bug as intent. Renames the docs missed: histogram names gain the exporter's unit suffix (ios_latency_milliseconds_bucket and four siblings), ledger_history_mismatch gains _total, the StatsD-era quantile label gives way to le buckets, rpc.request becomes rpc.http_request, traces_spanmetrics_calls_total becomes span_calls_total, and the nine dotted xrpl.* span attributes are recorded as renamed rather than left as live keys. Re-parenting: consensus.update_positions and consensus.check are children of consensus.establish, not of consensus.round. Units and labels: state_accounting_*_duration is microseconds, not seconds; cache_metrics label values are case-sensitive; object_count carries demangled C++ type names. Nodestore read and write latency stays microseconds -- the nanosecond accumulator change did not move the exported unit. Adds what shipped but was undocumented: the ledger.acquire span, seven consensus.round events, twelve span attributes, node_writes_duration_us, the 7-day validation-agreement window, the TxQ admission and reduce-relay metric families, metrics_endpoint, and the phase-10 validation workflow. Corrects claims that never held: 10% head sampling (it is fixed at 100%), configurable redaction (it is unconditional), -DXRPL_ENABLE_TELEMETRY=OFF (the flag is -Dtelemetry=OFF, default ON), FindOpenTelemetry.cmake and the xrpl_telemetry target (neither exists), Promtail and a StatsD exporter in the pipeline (neither exists), and Loki stream selection on job= (only service_name is a stream label). Phase 9 is marked complete, its provisioned alerting is attributed to the branch that shipped it, and Phase 11 stays at zero except the one prerequisite its code closes. Counts are reconciled repo-wide: 41 emitted span families, 15 dashboards on disk with 14 asserted, 13 alert rules in 5 groups. Hardens the gate that let this drift through: Rule E of the naming check now covers the reference docs, its allow-dotted marker is key-scoped and warns on stale or empty use, a missing checked file is reported instead of silently skipped, the test suite runs in CI, and doc paths trigger the check. C++ and CMake changes are comment-only: three MetricsRegistry instrument names, eight OTelCollector claims of a metric-name prefix that formatName never adds, and the telemetry option's inverted default.
16 KiB
Appendix
Parent Document: OpenTelemetryPlan.md Related: Observability Backends
8.1 Glossary
OTLP = OpenTelemetry Protocol | TxQ = Transaction Queue
| Term | Definition |
|---|---|
| Span | A unit of work with start/end time, name, and attributes |
| Trace | A collection of spans representing a complete request flow |
| Trace ID | 128-bit unique identifier for a trace |
| Span ID | 64-bit unique identifier for a span within a trace |
| Context | Carrier for trace/span IDs across boundaries |
| Propagator | Component that injects/extracts context |
| Sampler | Decides which traces to record |
| Exporter | Sends spans to backend |
| Collector | Receives, processes, and forwards telemetry |
| OTLP | OpenTelemetry Protocol (wire format) |
| W3C Trace Context | Standard HTTP headers for trace propagation |
| Baggage | Key-value pairs propagated across service boundaries |
| Resource | Entity producing telemetry (service, host, etc.) |
| Instrumentation | Code that creates telemetry data |
xrpld-Specific Terms
| Term | Definition |
|---|---|
| Overlay | P2P network layer managing peer connections |
| Consensus | XRP Ledger consensus algorithm (RCL) |
| Proposal | Validator's suggested transaction set for a ledger |
| Validation | Validator's signature on a closed ledger |
| HashRouter | Component for transaction deduplication |
| JobQueue | Thread pool for asynchronous task execution |
| PerfLog | Existing performance logging system in xrpld |
| Beast Insight | Existing metrics framework in xrpld |
| PathFinding | Payment path computation engine for cross-currency payments |
| TxQ | Transaction queue managing fee-based prioritization |
| LoadManager | Dynamic fee escalation based on network load |
| SHAMap | SHA-256 hash-based map (Merkle trie variant) for ledger state |
Phase 9–11 Terms
| Term | Definition |
|---|---|
| MetricsRegistry | Centralized class for OTel async gauge registrations (Phase 9) |
| ObservableGauge | OTel Metrics SDK async instrument polled via callback at fixed intervals |
| PeriodicMetricReader | OTel SDK component that invokes gauge callbacks at configurable intervals |
| CountedObject | xrpld template that tracks live instance counts via atomic counters |
| TxQ | Transaction queue managing fee escalation and ordering |
| Load Factor | Combined multiplier affecting transaction cost (local, cluster, network) |
| OTel Collector Receiver | Custom Go plugin that polls xrpld RPC and emits OTel metrics (Phase 11) |
8.2 Span Hierarchy Visualization
The authoritative span-flow diagrams — a master overview plus per-stage
flowcharts (ingress, the shared apply pipeline, the consensus round, ledger
finalize, and the pathfinding / ledger-acquire side flows) — live in the operator
runbook. They map every span onto the real xrpld control flow and XRPL protocol
order (verified against code and docs/consensus.md, with file:line evidence),
label every node and branch with the span that represents that state or
transition, and call out where the OpenTelemetry span parent links diverge from
that flow.
The full span inventory (names, attributes, parents as instrumented) is in 09-data-collection-reference.md §1.
8.3 References
OTLP = OpenTelemetry Protocol
OpenTelemetry Resources
- OpenTelemetry C++ SDK
- OpenTelemetry Specification
- OpenTelemetry Collector
- OTLP Protocol Specification
Standards
xrpld Resources
- xrpld Source Code
- XRP Ledger Documentation
- xrpld Overlay README
- xrpld RPC README
- xrpld Consensus README
8.4 Version History
| Version | Date | Author | Changes |
|---|---|---|---|
| 1.0 | 2026-02-12 | - | Initial implementation plan |
| 1.1 | 2026-02-13 | - | Refactored into modular documents |
| 1.2 | 2026-03-09 | - | Added Phases 9–11 (future enhancement plans) |
| 1.3 | 2026-03-24 | - | Review fixes: accuracy corrections, cross-document consistency |
8.5 Document Index
Plan Documents
| Document | Description |
|---|---|
| OpenTelemetryPlan.md | Master overview and executive summary |
| 00-tracing-fundamentals.md | Distributed tracing concepts and OTel primer |
| 01-architecture-analysis.md | xrpld architecture and trace points |
| 02-design-decisions.md | SDK selection, exporters, span conventions |
| 03-implementation-strategy.md | Directory structure, performance analysis |
| 05-configuration-reference.md | xrpld config, CMake, Collector configs |
| 06-implementation-phases.md | Timeline, tasks, risks, success metrics |
| 07-observability-backends.md | Backend selection and architecture |
| 08-appendix.md | Glossary, references, version history |
| secure-OTel.md | Threat model and hardening (mTLS, peer validation) |
| 09-data-collection-reference.md | Span/metric/dashboard inventory |
Task Lists
| Document | Description |
|---|---|
| Phase2_taskList.md | RPC layer trace instrumentation |
| Phase3_taskList.md | Peer overlay & consensus tracing |
| Phase4_taskList.md | Transaction lifecycle tracing |
| Phase5_taskList.md | Ledger processing & advanced tracing |
| Phase5_IntegrationTest_taskList.md | Observability stack integration tests |
| Phase7_taskList.md | Native OTel metrics migration |
| Phase8_taskList.md | Log-trace correlation |
| Phase9_taskList.md | Internal metric instrumentation gap fill |
| Phase10_taskList.md | Synthetic workload generation & validation |
| Phase11_taskList.md | Third-party data collection pipelines (future) |
Only Phase 11 is still "future". Phase 9 ships on
pratik/otel-phase9-metric-gap-fill(18 task entries, 9.1–9.17 plus 9.7a) and Phase 10 onpratik/otel-phase10-workload-validation(7 tasks). Their task lists are present on every branch from those points forward, so a reader on a later branch sees plans that are already implemented, not proposals. Phase 11 (13 tasks) has no implementation branch.
Note
: Phases 1 and 6 do not have separate task list files. Phase 1 tasks are documented in 06-implementation-phases.md §6.2. Phase 6 tasks are documented in 06-implementation-phases.md §6.7.
8.6 Phase 9–11 Cross-Reference Guide
This guide maps Phase 9–11 content to its location across the documentation.
Phase 9: Internal Metric Instrumentation Gap Fill
| Content | Location |
|---|---|
| Plan & architecture | 06-implementation-phases.md §6.8.2 |
| Task list (18 entries, 9.1–9.17) | Phase9_taskList.md |
| Metric definitions | 09-data-collection-reference.md §5b |
New class: MetricsRegistry |
src/xrpld/telemetry/MetricsRegistry.h/.cpp — shipped |
| New dashboards (4) | fee-market, job-queue, peer-quality, validator-health — shipped |
| Updated dashboards (2) | node-health, rpc-performance |
| Provisioned alert rules | docker/telemetry/grafana/provisioning/alerting/rules.yaml — 13 rules in 5 groups (07 §7.6.2) |
Task numbering:
Phase9_taskList.mdcarries 18## Task 9.xheadings — 9.1 through 9.17 plus the inserted 9.7a (push_metrics.pyparity). The "10 tasks" figure in earlier revisions predates 9.7a and 9.11–9.17. Tasks 9.8 and 9.11–9.13 together produce the four new dashboards; Task 9.17 (peer span coverage) is explicitly deferred to Phase 11.
Metric categories: NodeStore I/O, Cache Hit Rates, TxQ, PerfLog Per-RPC, PerfLog Per-Job, Counted Objects, Fee Escalation & Load Factors.
Phase 10: Synthetic Workload Generation & Telemetry Validation
| Content | Location |
|---|---|
| Plan & architecture | 06-implementation-phases.md §6.8.3 |
| Task list (7 tasks) | Phase10_taskList.md |
| Branch | pratik/otel-phase10-workload-validation |
| Validation inventory | 09-data-collection-reference.md §5c |
| Test harness | docker/telemetry/docker-compose.workload.yaml (phase-10 branch) |
| CI workflow | .github/workflows/telemetry-validation.yml (phase-10 branch) |
Validates (Phase-10 harness inventory): 40 span types, 67 unique required span attributes, 36 metric entries, 14 dashboards, log-trace correlation.
These are the harness manifests' counts, and two of them lag the code. The manifests (
docker/telemetry/workload/expected_spans.json,expected_metrics.json) live only on the phase-10 branch.expected_spans.jsonholds 40 span entries against the 41 span-name families the code emits (rpc.ws_upgradehas no entry), and its owntotal_unique_attributes: 58field is stale against the 67 attributes its per-spanrequired_attributeslists actually name.expected_metrics.jsonasserts 14 dashboard uids against the 15 dashboard JSONs indocker/telemetry/grafana/dashboards/;log-derived-insightsis the unasserted one. The full emitted inventory is in 09-data-collection-reference.md §1.1 and §5c.
Phase 11: Third-Party Data Collection Pipelines
| Content | Location |
|---|---|
| Plan & architecture | 06-implementation-phases.md §6.8.4 |
| Task list (13 tasks) | Phase11_taskList.md |
| External metric definitions (~30) | 09-data-collection-reference.md §5d |
| Custom OTel Collector receiver | docker/telemetry/otel-rippled-receiver/ (planned) |
| Prometheus alerting rules (11) | 09-data-collection-reference.md §5d |
| New dashboards (4) | Validator Health, Network Topology, Fee Market (External), DEX & AMM |
Two of those names now collide with shipped Phase-9 boards. Phase 9 already ships
validator-healthandfee-market, both built from the node's own telemetry. The Phase-11 entries are the third-party-data variants (network-wide validator agreement, external fee/DEX feeds via the custom receiver). They need distinct uids, or they will overwrite the Phase-9 boards on provisioning.
Consumer categories: Exchanges, Payment Processors, DeFi/AMM, NFT Marketplaces, Analytics Providers, Wallets, Compliance, Academic Researchers, Institutional Custody, CBDC Bridge Operators.
Previous: Observability Backends | Back to: Overview