ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-08-21 22:30:57 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor(telemetry): route dashboards, runbook and collector work to phase-9

Browse Source 
These changes were developed on the phase-10 branch but belong to content this
branch and its upstreams introduced. Carrying them on phase-10 made its PR diff
report churn in files phase-10 does not own, and left each PR claiming a scope
that did not match its contents.

Moved here from phase-10 (identical content, no functional change):

- Dashboards: all 14 existing boards plus the new log-derived-insights board.
- Docs: telemetry-runbook.md (minus the workload/benchmark sections, which
  describe phase-10 tooling) and the new telemetry-glossary.md.
- Grafana Cloud + Alloy export path: collector config, compose override, the
  two .env examples and alloy/config.alloy.
- Local stack: otel-collector-config.yaml gains sub-millisecond and
  second-scale spanmetrics buckets, pins unit=ms, and promotes
  close_time_correct; integration-test.sh and TESTING.md follow.
- Node configs: exported_instance -> service_instance_id in comments; the
  mainnet sample now logs at warning to bound log volume.
- Metrics code: Telemetry.cpp builds the metrics pipeline in the constructor
  via initMetrics() so the global MeterProvider is published before any
  subsystem creates a beast::insight instrument, and the histogram view keeps
  each instrument's own name instead of collapsing them under one series.
  MetricsRegistry gains a last_close_time gauge and skips negative job-queue
  durations. OTelCollector drops an unused accessor.
- Naming CI: xrpl_work_item joins EXTERNAL_INFRA_LABELS and Rule E accepts the
  dotted perf-iac resource-attribute form. This must travel with the
  dashboards and runbook that reference those labels, or the rules fail.
- Doxygen input glob no longer recurses dot-directories.

Sections describing phase-10 tooling stay on phase-10 and keep their
"Future Enhancement" / "Planned, not yet implemented" markers here; phase-10
removes those markers when it lands the tooling.
This commit is contained in:
Pratik Mankawde
2026-08-04 16:10:04 +01:00
parent feabcc5b89
commit 3860c93db2
48 changed files with 12682 additions and 4055 deletions
Download Patch File Download Diff File Expand all files Collapse all files

23
src/libxrpl/beast/insight/OTelCollector.cpp
View File

@@ -135,7 +135,7 @@ public:
/**
* @param name Export-ready metric name, already run through
* formatName() by the collector: prefix prepended and
* dots replaced with underscores (e.g. "xrpld_rpc_size").
* dots replaced with underscores (e.g. "rpc_size").
* @param meter OTel Meter used to create the counter instrument.
*/
OTelCounterImpl(
@@ -179,7 +179,7 @@ public:
/**
* @param name Export-ready metric name, already run through
* formatName() by the collector: prefix prepended and
* dots replaced with underscores (e.g. "xrpld_rpc_size").
* dots replaced with underscores (e.g. "rpc_size").
* @param meter OTel Meter used to create the histogram instrument.
*/
OTelEventImpl(
@@ -311,7 +311,7 @@ public:
/**
* @param name Export-ready metric name, already run through
* formatName() by the collector: prefix prepended and
* dots replaced with underscores (e.g. "xrpld_rpc_size").
* dots replaced with underscores (e.g. "rpc_size").
* @param meter OTel Meter used to create the counter instrument.
*/
OTelMeterImpl(
@@ -393,7 +393,7 @@ private:
* "node-1", "xrpld", "mainnet", journal);
* auto counter = collector->makeCounter("rpc.requests");
* counter.increment(1);
* // Metric "xrpld_rpc_requests" exported via OTLP every 1s.
* // Metric "rpc_requests" exported via OTLP every 1s.
* @endcode
*/
class OTelCollectorImp : public OTelCollector, public std::enable_shared_from_this<OTelCollectorImp>
@@ -497,19 +497,12 @@ public:
removeGauge(OTelGaugeImpl* gauge);
/** @} */
/**
* @brief Get the OTel Meter instance for creating instruments.
* @return Shared pointer to the OTel Meter.
*/
opentelemetry::nostd::shared_ptr<metrics_api::Meter> const&
otelMeter() const;
/**
* @brief Format a metric name with the configured prefix.
*
* Replaces dots with underscores to match StatsD->Prometheus naming.
* Example: prefix="xrpld", name="LedgerMaster.Validated_Ledger_Age"
* -> "xrpld_LedgerMaster_Validated_Ledger_Age"
* -> "ledgermaster_validated_ledger_age"
*
* @param name Raw metric name from beast::insight callers.
* @return Fully-qualified metric name.
@@ -850,12 +843,6 @@ OTelCollectorImp::removeGauge(OTelGaugeImpl* gauge)
std::erase(gauges_, gauge);
}
opentelemetry::nostd::shared_ptr<metrics_api::Meter> const&
OTelCollectorImp::otelMeter() const
{
return otelMeter_;
}
std::string
OTelCollectorImp::formatName(std::string const& name)
{

77
src/libxrpl/telemetry/Telemetry.cpp
View File

@@ -320,6 +320,21 @@ class TelemetryImpl : public Telemetry
public:
TelemetryImpl(Setup setup, beast::Journal journal) : setup_(std::move(setup)), journal_(journal)
{
// Build the metrics pipeline NOW, in the constructor, so the global
// MeterProvider is published before any subsystem is constructed.
// beast::insight instruments are created eagerly in subsystem
// constructors (e.g. LedgerMaster, NetworkOPs, ServerHandler), which
// run during ApplicationImp's member-init list — long before start().
// opentelemetry-cpp has no proxy MeterProvider, so an instrument
// created before SetMeterProvider() binds to the noop provider forever.
// Tracing does not have this problem because getTracer() is called
// fresh at each span creation (runtime, after start()).
//
// The metrics resource uses setup_.serviceInstanceId as provided by
// config. A later setServiceInstanceId() (node-key fallback) cannot
// change this immutable resource, so operators relying on the node-key
// identity should set [telemetry] service_instance_id explicitly.
initMetrics();
}
void
@@ -407,10 +422,30 @@ public:
trace_api::Provider::SetTracerProvider(
opentelemetry::nostd::shared_ptr<trace_api::TracerProvider>(sdkProvider_));
// Build the metrics pipeline, parallel to the tracer above and
// reusing the same resourceAttrs so metrics and traces share one
// resource identity.
// The metrics pipeline (meterProvider_) was already built and published
// in the constructor via initMetrics(), before any subsystem could
// create a beast::insight instrument. See initMetrics().
// Register as the global Telemetry instance so SpanGuard factory
// methods can access it without callers passing a reference.
Telemetry::setInstance(this);
JLOG(journal_.info()) << "Telemetry started successfully";
}
/**
* Build and publish the metrics pipeline (MeterProvider + periodic
* reader + OTLP exporter + histogram view).
*
* Called from the constructor, NOT start(), so the global MeterProvider
* exists before subsystems construct their beast::insight instruments
* during ApplicationImp's member-init list. The metrics resource uses
* setup_.serviceInstanceId from config; it is immutable once the provider
* is built, so a later node-key setServiceInstanceId() does not affect it.
*/
void
initMetrics()
{
// Derive the metrics endpoint from the trace endpoint by swapping
// the trailing "/v1/traces" path for "/v1/metrics". Any other URL
// shape is used as-is.
@@ -444,24 +479,44 @@ public:
auto reader = metrics_sdk::PeriodicExportingMetricReaderFactory::Create(
std::move(metricExporter), readerOpts);
// Metrics resource: same attributes as the tracer resource so metrics
// and traces share one identity. Built here (not shared with start())
// because start() runs later; serviceInstanceId comes from config.
auto resourceAttrs = resource::Resource::Create({
{opentelemetry::semconv::service::kServiceName, setup_.serviceName},
{opentelemetry::semconv::service::kServiceVersion, setup_.serviceVersion},
{opentelemetry::semconv::service::kServiceInstanceId, setup_.serviceInstanceId},
{std::string(attr::networkId), static_cast<int64_t>(setup_.networkId)},
{std::string(attr::networkType), setup_.networkType},
});
// Create MeterProvider with the shared resource, then attach reader.
meterProvider_ = metrics_sdk::MeterProviderFactory::Create(
std::make_unique<metrics_sdk::ViewRegistry>(), resourceAttrs);
meterProvider_->AddMetricReader(std::move(reader));
// Histogram view: SpanMetrics-compatible bucket boundaries (ms) so
// histogram instruments align with the collector's SpanMetrics.
// histogram instruments align with the collector's SpanMetrics. The
// view is created with an EMPTY name so it applies the buckets WITHOUT
// renaming instruments — a non-empty view name would collapse every
// matching histogram (ios_latency, rpc_size, rpc_time, pathfind_*)
// into a single series under that one name.
auto histogramSelector = metrics_sdk::InstrumentSelectorFactory::Create(
metrics_sdk::InstrumentType::kHistogram, "*", "ms");
auto meterSelector = metrics_sdk::MeterSelectorFactory::Create("xrpld_metrics", "", "");
// Meter selector MUST match the meter name used by getMeter() and the
// beast OTelCollector (kMeterName = "xrpld"); otherwise this histogram
// view never applies and duration histograms fall back to the SDK
// default boundaries instead of these SpanMetrics-aligned buckets.
auto meterSelector =
metrics_sdk::MeterSelectorFactory::Create(std::string(kMeterName), "", "");
auto histogramConfig = std::make_shared<metrics_sdk::HistogramAggregationConfig>();
histogramConfig->boundaries_ =
std::vector<double>{1.0, 5.0, 10.0, 25.0, 50.0, 100.0, 250.0, 500.0, 1000.0, 5000.0};
auto histogramView = metrics_sdk::ViewFactory::Create(
"default_histogram",
"Default histogram view with SpanMetrics-compatible buckets",
"", // empty name: keep each instrument's own name, only set buckets
"SpanMetrics-compatible histogram buckets",
metrics_sdk::AggregationType::kHistogram,
std::move(histogramConfig));
histogramConfig);
meterProvider_->AddView(
std::move(histogramSelector), std::move(meterSelector), std::move(histogramView));
@@ -470,12 +525,6 @@ public:
// OTelCollector shim) reach the same pipeline.
metrics_api::Provider::SetMeterProvider(
opentelemetry::nostd::shared_ptr<metrics_api::MeterProvider>(meterProvider_));
// Register as the global Telemetry instance so SpanGuard factory
// methods can access it without callers passing a reference.
Telemetry::setInstance(this);
JLOG(journal_.info()) << "Telemetry started successfully";
}
void

2
src/xrpld/app/main/Application.cpp
View File

@@ -1670,7 +1670,7 @@ ApplicationImp::startTelemetry()
set(endpoint, "metrics_endpoint", section);
// Pass the service_instance_id so the MeterProvider Resource
// carries it, giving Prometheus an exported_instance label.
// carries it, giving Prometheus an service_instance_id label.
std::string instanceId;
set(instanceId, "service_instance_id", section);
if (instanceId.empty() && nodeIdentity_)

2
src/xrpld/app/main/CollectorManager.cpp
View File

@@ -54,7 +54,7 @@ public:
// Read service_instance_id, same key as the [telemetry]
// section uses, so multi-node deployments can distinguish
// metric sources via the exported_instance Prometheus label.
// metric sources via the service_instance_id Prometheus label.
std::string const instanceId = get(params, "service_instance_id");
// service.name from [insight] (falls back to the value the

31
src/xrpld/telemetry/MetricsRegistry.cpp
View File

@@ -288,7 +288,7 @@ MetricsRegistry::initExporterAndProvider(std::string const& endpoint, std::strin
auto reader =
metric_sdk::PeriodicExportingMetricReaderFactory::Create(std::move(exporter), readerOpts);
// Configure resource attributes so Prometheus exported_instance labels
// Configure resource attributes so Prometheus service_instance_id labels
// distinguish metrics from different nodes (matches OTelCollector setup).
resource::ResourceAttributes attrs;
// Use std::string, not a string literal: ResourceAttributes stores an
@@ -377,9 +377,8 @@ MetricsRegistry::initSyncInstruments()
"txq_expired_total", "Total transactions expired out of the transaction queue");
txqDroppedCounter_ = meter_->CreateUInt64Counter(
"txq_dropped_total", "Total transactions refused admission to the queue by reason");
// Note: xrpld_validation_agreements_total / xrpld_validation_missed_total
// are monotonic ObservableCounters created in registerValidationTotalsCounters()
// (below), observed from ValidationTracker's gross lifetime tallies.
// Note: validation_agreements_total / validation_missed_total are monotonic
// ObservableCounters created in registerValidationTotalsCounters() (below).
}
#endif // XRPL_ENABLE_TELEMETRY
@@ -514,8 +513,12 @@ MetricsRegistry::recordJobStarted(
// must carry the identical label set or they cannot be joined.
std::string const handler(sanitiseHandler(jobName));
jobStartedCounter_->Add(1, {{kJobTypeLabel, std::string(jobType)}, {kHandlerLabel, handler}});
if (jobQueuedDurationHistogram_)
if (jobQueuedDurationHistogram_ && queuedDurUs >= 0)
{
// Guard against negative queued durations: the caller derives this
// from a steady-clock delta that can go slightly negative under clock
// skew or reordering. The OTel SDK rejects negative histogram values
// (logging a warning per call), so skip them rather than spam.
jobQueuedDurationHistogram_->Record(
static_cast<double>(queuedDurUs),
{{kJobTypeLabel, std::string(jobType)}, {kHandlerLabel, handler}},
@@ -1043,6 +1046,22 @@ MetricsRegistry::registerServerInfoGauge()
"last_close_converge_time_ms",
static_cast<int64_t>(consensusInfo["previous_mseconds"].asUInt()));
}
// Network close time of the last closed ledger, as NetClock
// seconds since the XRPL epoch (2000-01-01). Unlike a span
// timestamp, a gauge value survives as a queryable time series,
// so dashboards can show last-close age (staleness) via
// now - value. The close interval comes from the
// ledgers_closed_total counter, not a delta of this gauge
// (a timestamp gauge's delta aliases to the scrape period).
// Skip until a ledger has closed.
if (auto const closed = app.getLedgerMaster().getClosedLedger())
{
observe(
"last_close_time",
static_cast<int64_t>(
closed->header().closeTime.time_since_epoch().count()));
}
}
catch (...) // NOLINT(bugprone-empty-catch)
{
@@ -1605,7 +1624,7 @@ MetricsRegistry::registerValidationTotalsCounters()
// count). We therefore observe the tracker's GROSS lifetime tallies, which
// count each ledger once at first classification and are never adjusted on
// repair (initial-classification semantics — see ValidationTracker). The
// repaired/agreement view remains available from xrpld_validation_agreement.
// repaired/agreement view remains available from validation_agreement.
//
// reconcile() is called first so pending events are resolved before the
// tallies are read; the callback fires every ~10 s from the

2
src/xrpld/telemetry/MetricsRegistry.h
View File

@@ -283,7 +283,7 @@ public:
* (e.g. "http://localhost:4318/v1/metrics").
* @param instanceId Value for the service.instance.id resource
* attribute. When non-empty, Prometheus metrics
* carry an exported_instance label for per-node
* carry a service_instance_id label for per-node
* filtering.
*/
void