mirror of
https://github.com/XRPLF/rippled.git
synced 2026-04-29 15:37:57 +00:00
feat(telemetry): add Phase 5 documentation, deployment configs, and integration tests
Add the observability stack deployment infrastructure and integration test framework for verifying end-to-end trace export. - Add Grafana dashboards: RPC performance, transaction overview, consensus health (pre-provisioned via dashboards.yaml) - Add Prometheus config for spanmetrics collection from OTel Collector - Update OTel Collector config with spanmetrics connector and prometheus exporter for RED metrics - Add docker-compose services: prometheus, dashboard provisioning - Add integration-test.sh with Tempo API-based span verification (replaces previous Jaeger-based approach) - Add TESTING.md with step-by-step deployment and verification guide - Add telemetry-runbook.md for production operations reference - Add xrpld-telemetry.cfg sample configuration - Add toDisplayString() for ConsensusMode (human-readable span values) - Update Phase 2/3 task lists with known issues sections - Add Phase 5 integration test task list - Add TraceContext protobuf fields for future relay propagation - Wire telemetry lifecycle (setServiceInstanceId/start/stop) in Application.cpp Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -204,3 +204,36 @@ This surfaces all RPCs served during a blocked period — critical for post-inci
|
||||
**Delivered in this branch**: Tasks 2.4, 2.7, 2.8, 2.9.
|
||||
**Deferred with rationale**: Tasks 2.1 (→Phase 3), 2.5 (low priority).
|
||||
**Superseded**: Task 2.2 (Phase 1c SpanGuard factory covers this).
|
||||
|
||||
---
|
||||
|
||||
## Known Issues / Future Work
|
||||
|
||||
### Thread safety of TelemetryImpl::stop() vs startSpan()
|
||||
|
||||
`TelemetryImpl::stop()` resets `sdkProvider_` (a `std::shared_ptr`) without
|
||||
synchronization. `getTracer()` reads the same member from RPC handler threads.
|
||||
This is a data race if any thread calls `startSpan()` concurrently with `stop()`.
|
||||
|
||||
**Current mitigation**: `Application::stop()` shuts down `serverHandler_`,
|
||||
`overlay_`, and `jobQueue_` before calling `telemetry_->stop()`, so no callers
|
||||
remain. See comments in `Telemetry.cpp:stop()` and `Application.cpp`.
|
||||
|
||||
**TODO**: Add an `std::atomic<bool> stopped_` flag checked in `getTracer()` to
|
||||
make this robust against future shutdown order changes.
|
||||
|
||||
### Macro incompatibility: XRPL_TRACE_SPAN vs XRPL_TRACE_SET_ATTR
|
||||
|
||||
`XRPL_TRACE_SPAN` and `XRPL_TRACE_SPAN_KIND` declare `_xrpl_guard_` as a bare
|
||||
`SpanGuard`, but `XRPL_TRACE_SET_ATTR` and `XRPL_TRACE_EXCEPTION` call
|
||||
`_xrpl_guard_.has_value()` which requires `std::optional<SpanGuard>`. Using
|
||||
`XRPL_TRACE_SPAN` followed by `XRPL_TRACE_SET_ATTR` in the same scope would
|
||||
fail to compile.
|
||||
|
||||
**Current mitigation**: No call site currently uses `XRPL_TRACE_SPAN` — all
|
||||
production code uses the conditional macros (`XRPL_TRACE_RPC`, `XRPL_TRACE_TX`,
|
||||
etc.) which correctly wrap the guard in `std::optional`.
|
||||
|
||||
**TODO**: Either make `XRPL_TRACE_SPAN`/`XRPL_TRACE_SPAN_KIND` also wrap in
|
||||
`std::optional`, or document that `XRPL_TRACE_SET_ATTR` is only compatible with
|
||||
the conditional macros.
|
||||
|
||||
Reference in New Issue
Block a user