diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 31702e7c3c..d15548e239 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -414,6 +414,14 @@ python .github/scripts/otel-naming/check_otel_naming.py See [.github/scripts/otel-naming/README.md](.github/scripts/otel-naming/README.md) for the full rule list. +## Adding a new OTel metric + +See `src/xrpld/telemetry/MetricMacros.h` for the call-site macros covering every +OTel instrument kind (Counter, UpDownCounter, Histogram, Gauge, and their +Observable/async counterparts) and the "Adding a New Metric" section in +[docs/telemetry-runbook.md](docs/telemetry-runbook.md) for the walkthrough and a +need-to-macro lookup table. + ## Contracts and instrumentation We are using [Antithesis](https://antithesis.com/) for continuous fuzzing, diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index 615e8f148f..6897d8a18b 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -761,27 +761,28 @@ The `OTelCollector` implementation exports metrics via OTLP/HTTP to the same OTe These gauges are exported via the OTel Metrics SDK `PeriodicMetricReader` (10s interval), NOT through beast::insight. -| Prometheus Metric | Source | Description | -| --------------------------------------------------- | ------------------- | -------------------------------------------- | -| `server_info{metric="server_state"}` | MetricsRegistry.cpp | Operating mode (0=DISCONNECTED .. 4=FULL) | -| `server_info{metric="uptime"}` | MetricsRegistry.cpp | Seconds since server start | -| `server_info{metric="peers"}` | MetricsRegistry.cpp | Total connected peers | -| `server_info{metric="validated_ledger_seq"}` | MetricsRegistry.cpp | Validated ledger sequence number | -| `server_info{metric="ledger_current_index"}` | MetricsRegistry.cpp | Current open ledger sequence | -| `server_info{metric="peer_disconnects_resources"}` | MetricsRegistry.cpp | Cumulative resource-related peer disconnects | -| `server_info{metric="last_close_proposers"}` | MetricsRegistry.cpp | Proposers in last closed round | -| `server_info{metric="last_close_converge_time_ms"}` | MetricsRegistry.cpp | Last close convergence time (ms) | -| `build_info{version=""}` | MetricsRegistry.cpp | Info-style metric (always 1) | -| `complete_ledgers{bound="start\|end",index=""}` | MetricsRegistry.cpp | Complete ledger range start/end pairs | -| `db_metrics{metric="db_kb_total"}` | MetricsRegistry.cpp | Total database size (KB) | -| `db_metrics{metric="db_kb_ledger"}` | MetricsRegistry.cpp | Ledger database size (KB) | -| `db_metrics{metric="db_kb_transaction"}` | MetricsRegistry.cpp | Transaction database size (KB) | -| `db_metrics{metric="historical_perminute"}` | MetricsRegistry.cpp | Historical ledger fetches per minute | -| `cache_metrics{metric="AL_size"}` | MetricsRegistry.cpp | AcceptedLedger cache size | -| `nodestore_state{metric="node_reads_duration_us"}` | MetricsRegistry.cpp | Cumulative read time (microseconds) | -| `nodestore_state{metric="read_request_bundle"}` | MetricsRegistry.cpp | Read request bundle count | -| `nodestore_state{metric="read_threads_running"}` | MetricsRegistry.cpp | Active read threads | -| `nodestore_state{metric="read_threads_total"}` | MetricsRegistry.cpp | Total read threads configured | +| Prometheus Metric | Source | Description | +| --------------------------------------------------- | ------------------- | ------------------------------------------------ | +| `server_info{metric="server_state"}` | MetricsRegistry.cpp | Operating mode (0=DISCONNECTED .. 4=FULL) | +| `server_info{metric="uptime"}` | MetricsRegistry.cpp | Seconds since server start | +| `server_info{metric="peers"}` | MetricsRegistry.cpp | Total connected peers | +| `server_info{metric="validated_ledger_seq"}` | MetricsRegistry.cpp | Validated ledger sequence number | +| `server_info{metric="ledger_current_index"}` | MetricsRegistry.cpp | Current open ledger sequence | +| `server_info{metric="peer_disconnects_resources"}` | MetricsRegistry.cpp | Cumulative resource-related peer disconnects | +| `server_info{metric="last_close_proposers"}` | MetricsRegistry.cpp | Proposers in last closed round | +| `server_info{metric="last_close_converge_time_ms"}` | MetricsRegistry.cpp | Last close convergence time (ms) | +| `build_info{version=""}` | MetricsRegistry.cpp | Info-style metric (always 1) | +| `complete_ledgers{bound="start\|end",index=""}` | MetricsRegistry.cpp | Complete ledger range start/end pairs | +| `db_metrics{metric="db_kb_total"}` | MetricsRegistry.cpp | Total database size (KB) | +| `db_metrics{metric="db_kb_ledger"}` | MetricsRegistry.cpp | Ledger database size (KB) | +| `db_metrics{metric="db_kb_transaction"}` | MetricsRegistry.cpp | Transaction database size (KB) | +| `db_metrics{metric="historical_perminute"}` | MetricsRegistry.cpp | Historical ledger fetches per minute | +| `cache_metrics{metric="AL_size"}` | MetricsRegistry.cpp | AcceptedLedger cache size | +| `nodestore_state{metric="node_reads_duration_us"}` | MetricsRegistry.cpp | Cumulative read time (microseconds) | +| `nodestore_state{metric="read_request_bundle"}` | MetricsRegistry.cpp | Read request bundle count | +| `nodestore_state{metric="read_threads_running"}` | MetricsRegistry.cpp | Active read threads | +| `nodestore_state{metric="read_threads_total"}` | MetricsRegistry.cpp | Total read threads configured | +| `rpc_in_flight_requests` | PerfLogImp.cpp | RPC requests currently executing (UpDownCounter) | #### Counters @@ -803,6 +804,49 @@ These gauges are exported via the OTel Metrics SDK `PeriodicMetricReader` (10s i | `pathfind_fast` | PathRequests.h:23 | Fast pathfinding duration (ms) | | `pathfind_full` | PathRequests.h:24 | Full pathfinding duration (ms) | +#### Adding a New Metric + + + + +Use the call-site macros in `src/xrpld/telemetry/MetricMacros.h` -- no +`MetricsRegistry.h`/`.cpp` edit is needed for any of these: + +| Need | Macro | +| ------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Monotonic tally (never decreases) | `XRPL_METRIC_COUNTER_INC` / `_ADD` [+ `_LABELED`] | +| Running total that can decrease | `XRPL_METRIC_UPDOWN_ADD` [+ `_LABELED`] | +| Distribution of values (latency, size) | `XRPL_METRIC_HISTOGRAM_RECORD` [+ `_LABELED`] | +| Last-value snapshot (not a distribution) | `XRPL_METRIC_GAUGE_RECORD` [+ `_LABELED`] -- requires an ABI v2 opentelemetry-cpp build; this repo currently builds ABI v1, so use the observable-gauge row below instead | +| Value your own code already tracks, sampled on a timer | `XRPL_METRIC_OBSERVABLE_GAUGE_REGISTER` / `_COUNTER_REGISTER` / `_UPDOWN_REGISTER` | + +```cpp +#include + +// Monotonic counter: +XRPL_METRIC_COUNTER_INC(app_, "my_new_thing_total", "Description of what this counts"); + +// Value that can go up and down, e.g. in-flight work (no _total suffix -- that +// is reserved for monotonic counters; an UpDownCounter is a current value): +XRPL_METRIC_UPDOWN_ADD(app_, "my_in_flight_requests", "Currently executing", 1); // on start +XRPL_METRIC_UPDOWN_ADD(app_, "my_in_flight_requests", "Currently executing", -1); // on finish + +// Sampled from your own state, on the OTel export timer (register ONCE, in init code): +XRPL_METRIC_OBSERVABLE_GAUGE_REGISTER(app_, "my_thing_size", "Current size", + [this] { return static_cast(myThing_.size()); }); +``` + +Counters use a `_total` suffix by convention. A histogram whose values can +exceed ~10,000 units (e.g. a microsecond duration beyond 10ms) still needs one +line added to `addMicrosecondHistogramView()` in `MetricsRegistry.cpp` -- the +only case that still touches a central file. There is no way to read a metric's +current value back from application code -- OTel's API is write-only by design; +keep your own state if your logic needs to both record and read a running value +(see the Doxygen header in `MetricMacros.h` and "Use Case 4" in +`tasks/metric-macro-plan.md` for the full explanation and the `prometheus-cpp` +contrast rationale). + ## Deployment Tiers Multiple xrpld instances can send telemetry to per-tier collectors that all diff --git a/src/tests/libxrpl/telemetry/MetricMacros.cpp b/src/tests/libxrpl/telemetry/MetricMacros.cpp new file mode 100644 index 0000000000..ddcd2af33c --- /dev/null +++ b/src/tests/libxrpl/telemetry/MetricMacros.cpp @@ -0,0 +1,338 @@ +/** + * @file MetricMacros.cpp + * Unit tests for the XRPL_METRIC_* call-site macros (MetricMacros.h). + * + * Compiled only when XRPL_ENABLE_TELEMETRY is defined -- the macros expand + * to no-ops otherwise and have nothing to test beyond "it compiles", which + * MetricsRegistry.cpp's own no-op build already proves. + * + * These tests exercise the macros against a bare SDK MeterProvider (no + * OTLP exporter, no network, no background thread), the same technique + * GetMeter.cpp uses. The macros are duck-typed: they only ever call + * app.getMetricsRegistry(), then isEnabled() and meter() on the result. + * The tests therefore drive them through a tiny FakeApp/FakeMetricsRegistry + * pair instead of the real telemetry::MetricsRegistry, whose enabled-path + * .cpp drags xrpld link dependencies (LedgerMaster, TxQ, NetworkOPs, ...) + * that the standalone xrpl_tests binary cannot satisfy -- the same reason + * MetricsRegistry.cpp is only compiled into this binary on the no-op path. + */ + +// cspell:ignore ISTOGRAM +// The all-caps macro name XRPL_METRIC_HISTOGRAM_RECORD trips cspell's +// compound-word splitter, which emits the subword "ISTOGRAM"; ignore it here. + +#ifdef XRPL_ENABLE_TELEMETRY + +#include + +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +using namespace xrpl; + +namespace { + +/** + * Duck-typed stand-in for telemetry::MetricsRegistry. The macros only + * ever call isEnabled() and meter() on the registry pointer, so this + * minimal type is all they need -- and it links without pulling in the + * real MetricsRegistry.cpp's xrpld dependencies. + */ +class FakeMetricsRegistry +{ +public: + /** + * Sets the enable flag and meter the macros will observe. Tests call + * this once before exercising a macro, mimicking a started registry. + */ + void + configure(bool enabled, opentelemetry::nostd::shared_ptr meter) + { + enabled_ = enabled; + meter_ = std::move(meter); + } + + /** + * Number of times meter() has been consulted, so a test can assert the + * create-once (call_once) and disabled-gating behavior exactly. + */ + [[nodiscard]] int + meterCalls() const noexcept + { + return meterCalls_.load(std::memory_order_relaxed); + } + + [[nodiscard]] bool + isEnabled() const noexcept + { + return enabled_; + } + + [[nodiscard]] opentelemetry::nostd::shared_ptr + meter() const noexcept + { + meterCalls_.fetch_add(1, std::memory_order_relaxed); + return meter_; + } + +private: + /** + * Master enable flag the macro consults via isEnabled(). + */ + bool enabled_ = true; + + /** + * Meter handed to the macro; sourced from a bare SDK provider. + */ + opentelemetry::nostd::shared_ptr meter_; + + /** + * Counts meter() calls so tests can assert create-once / gating. + */ + mutable std::atomic meterCalls_{0}; +}; + +/** + * Duck-typed stand-in for ServiceRegistry. The macros only call + * getMetricsRegistry() on the app object, so this is all they need. + */ +class FakeApp +{ +public: + [[nodiscard]] FakeMetricsRegistry* + getMetricsRegistry() noexcept + { + return ®istry_; + } + + /** + * Direct handle to the underlying fake registry, for test setup and + * post-condition assertions. + */ + [[nodiscard]] FakeMetricsRegistry& + registry() noexcept + { + return registry_; + } + +private: + /** + * The macro-facing registry; tests configure it before use. + */ + FakeMetricsRegistry registry_; +}; + +/** + * Wraps a bare, exporter-less SDK MeterProvider so tests can build a + * real (non-noop) meter without touching the network. This is exactly + * the technique GetMeter.cpp's global_provider_meter_accepts_updown_counter + * test uses. + */ +class ScopedBareProvider +{ +public: + ScopedBareProvider() + { + // Hold the SDK provider in a std::shared_ptr first, then wrap it in a + // nostd::shared_ptr, exactly as GetMeter.cpp does. Constructing the + // nostd::shared_ptr directly from the factory's std::unique_ptr is + // ambiguous across its unique_ptr/shared_ptr overloads. + std::shared_ptr const sdkProvider = + opentelemetry::sdk::metrics::MeterProviderFactory::Create(); + previous_ = opentelemetry::metrics::Provider::GetMeterProvider(); + opentelemetry::metrics::Provider::SetMeterProvider( + opentelemetry::nostd::shared_ptr(sdkProvider)); + } + + ~ScopedBareProvider() + { + opentelemetry::metrics::Provider::SetMeterProvider(previous_); + } + +private: + /** + * Previous global provider, restored on scope exit. + */ + opentelemetry::nostd::shared_ptr previous_; +}; + +/** + * Fetches a real (non-noop) meter from whatever provider is globally + * installed -- inside a test this is the ScopedBareProvider's SDK provider. + */ +[[nodiscard]] opentelemetry::nostd::shared_ptr +bareMeter() +{ + return opentelemetry::metrics::Provider::GetMeterProvider()->GetMeter("xrpld_test", "1.0.0"); +} + +/** + * Builds a FakeApp wired to a real meter from the bare provider. + */ +void +wire(FakeApp& app, bool enabled) +{ + app.registry().configure(enabled, bareMeter()); +} + +} // namespace + +TEST(MetricMacros, counter_inc_creates_once_and_does_not_crash) +{ + ScopedBareProvider const bareProvider; + FakeApp app; + wire(app, /*enabled=*/true); + + // Call three times from the same call site; must not crash, and must + // not throw even though no real collector is listening. + for (int i = 0; i < 3; ++i) + { + XRPL_METRIC_COUNTER_INC( + app, "test_macro_counter_total", "Test counter for macro unit test"); + } + + // Create-once proof: std::call_once consults meter() exactly once across + // the three calls at this site, then reuses the cached instrument handle. + EXPECT_EQ(app.registry().meterCalls(), 1); +} + +TEST(MetricMacros, counter_inc_labeled_does_not_crash) +{ + ScopedBareProvider const bareProvider; + FakeApp app; + wire(app, /*enabled=*/true); + + XRPL_METRIC_COUNTER_INC_LABELED( + app, + "test_macro_labeled_counter_total", + "Test labeled counter for macro unit test", + {{"reason", std::string("unit_test")}}); + + // Instrument was created exactly once at this single call site. + EXPECT_EQ(app.registry().meterCalls(), 1); +} + +TEST(MetricMacros, histogram_record_does_not_crash) +{ + ScopedBareProvider const bareProvider; + FakeApp app; + wire(app, /*enabled=*/true); + + for (std::int64_t const v : {100, 250, 9999}) + { + XRPL_METRIC_HISTOGRAM_RECORD( + app, "test_macro_histogram_us", "Test histogram for macro unit test", v); + } + + // One call site, three records: histogram created once, then reused. + EXPECT_EQ(app.registry().meterCalls(), 1); +} + +TEST(MetricMacros, updown_add_accepts_positive_and_negative) +{ + ScopedBareProvider const bareProvider; + FakeApp app; + wire(app, /*enabled=*/true); + + // The whole point of UpDownCounter vs. Counter: negative Add() must not + // throw/crash/assert, unlike a plain Counter where a negative value + // violates the API contract. These are two distinct call sites (two + // source lines), so each lazily creates its own handle: meter() twice. + XRPL_METRIC_UPDOWN_ADD(app, "test_macro_updown_total", "Test updown for macro unit test", 1); + XRPL_METRIC_UPDOWN_ADD(app, "test_macro_updown_total", "Test updown for macro unit test", -1); + + EXPECT_EQ(app.registry().meterCalls(), 2); +} + +TEST(MetricMacros, updown_add_labeled_does_not_crash) +{ + ScopedBareProvider const bareProvider; + FakeApp app; + wire(app, /*enabled=*/true); + + XRPL_METRIC_UPDOWN_ADD_LABELED( + app, + "test_macro_updown_labeled_total", + "Test labeled updown for macro unit test", + -1, + {{"reason", std::string("unit_test")}}); + + EXPECT_EQ(app.registry().meterCalls(), 1); +} + +TEST(MetricMacros, observable_gauge_register_reports_current_value) +{ + ScopedBareProvider const bareProvider; + FakeApp app; + wire(app, /*enabled=*/true); + + // Own the state exactly as a real caller would (Use Case 5 in the + // design doc) -- the macro's callback reads through this atomic on + // every collection tick, it does not own the value itself. + std::atomic queueDepth{0}; + XRPL_METRIC_OBSERVABLE_GAUGE_REGISTER( + app, + "test_macro_observable_gauge", + "Test observable gauge for macro unit test", + [&queueDepth] { return queueDepth.load(); }); + + // There is no application-level read-back API (Use Case 4) -- this + // test can only prove registration doesn't crash and that meter() was + // consulted to create the observable instrument. It does NOT assert the + // observed value reaches Prometheus; that is Task 3b's docker-harness + // job, not this hermetic unit test. + queueDepth.store(42); + EXPECT_EQ(app.registry().meterCalls(), 1); +} + +TEST(MetricMacros, observable_counter_and_updown_register_do_not_crash) +{ + ScopedBareProvider const bareProvider; + FakeApp app; + wire(app, /*enabled=*/true); + + std::atomic total{0}; + XRPL_METRIC_OBSERVABLE_COUNTER_REGISTER( + app, + "test_macro_observable_counter_total", + "Test observable counter for macro unit test", + [&total] { return total.load(); }); + + std::atomic delta{0}; + XRPL_METRIC_OBSERVABLE_UPDOWN_REGISTER( + app, + "test_macro_observable_updown", + "Test observable updown for macro unit test", + [&delta] { return delta.load(); }); + + // Two observable registrations, each consulting meter() once. + EXPECT_EQ(app.registry().meterCalls(), 2); +} + +TEST(MetricMacros, disabled_registry_is_noop) +{ + ScopedBareProvider const bareProvider; + FakeApp app; + // enabled=false: the macro's isEnabled() gate short-circuits before it + // ever touches meter(), so nothing is created or recorded. + wire(app, /*enabled=*/false); + + XRPL_METRIC_COUNTER_INC( + app, "test_macro_disabled_counter_total", "Test counter for macro unit test (disabled)"); + + // Gating proof: meter() is never consulted when disabled. + EXPECT_EQ(app.registry().meterCalls(), 0); +} + +#endif // XRPL_ENABLE_TELEMETRY diff --git a/src/xrpld/app/consensus/RCLConsensus.cpp b/src/xrpld/app/consensus/RCLConsensus.cpp index 24a960cea9..e7a988faa5 100644 --- a/src/xrpld/app/consensus/RCLConsensus.cpp +++ b/src/xrpld/app/consensus/RCLConsensus.cpp @@ -22,6 +22,7 @@ #include #include #include +#include #include #include @@ -739,9 +740,10 @@ RCLConsensus::Adaptor::doAccept( // See if we can accept a ledger as fully-validated ledgerMaster_.consensusBuilt(built.ledger, result.txns.id(), std::move(consensusJson)); - // Record ledger close for OTel dashboard parity counter. - if (auto* mr = app_.getMetricsRegistry()) - mr->incrementLedgersClosed(); + // Record ledger close for OTel dashboard parity counter. Uses the + // call-site macro (see MetricMacros.h) rather than a MetricsRegistry + // member -- proof-of-concept for tasks/metric-macro-plan.md. + XRPL_METRIC_COUNTER_INC(app_, "ledgers_closed_total", "Total ledgers closed by consensus"); //------------------------------------------------------------------------- { diff --git a/src/xrpld/perflog/detail/PerfLogImp.cpp b/src/xrpld/perflog/detail/PerfLogImp.cpp index f06df3f3ff..f840ec2bf3 100644 --- a/src/xrpld/perflog/detail/PerfLogImp.cpp +++ b/src/xrpld/perflog/detail/PerfLogImp.cpp @@ -1,6 +1,7 @@ #include #include +#include #include #include @@ -336,6 +337,13 @@ PerfLogImp::rpcStart(std::string const& method, std::uint64_t const requestId) // Task 9.4: Record RPC start in OTel metrics pipeline. if (auto* mr = app_.getMetricsRegistry()) mr->recordRpcStarted(method); + + // Proof-of-concept for tasks/metric-macro-plan.md Use Case 2: a value + // that must be able to decrease (UpDownCounter), added at its call + // site with no MetricsRegistry member/init-line/method. Paired with the + // matching -1 in rpcEnd(). Runs on the same path as recordRpcStarted + // above, i.e. only after a methods-map entry exists for this request. + XRPL_METRIC_UPDOWN_ADD(app_, "rpc_in_flight_requests", "RPC requests currently executing", 1); } void @@ -398,6 +406,11 @@ PerfLogImp::rpcEnd(std::string const& method, std::uint64_t const requestId, boo mr->recordRpcErrored(method, durationUs.count()); } } + + // Matching -1 for the +1 recorded in rpcStart(). Placed after the early + // returns above so it runs only when this request's methods-map entry was + // found (i.e. a +1 was recorded for it), keeping the in-flight count balanced. + XRPL_METRIC_UPDOWN_ADD(app_, "rpc_in_flight_requests", "RPC requests currently executing", -1); } void diff --git a/src/xrpld/telemetry/MetricMacros.h b/src/xrpld/telemetry/MetricMacros.h new file mode 100644 index 0000000000..4a835233c2 --- /dev/null +++ b/src/xrpld/telemetry/MetricMacros.h @@ -0,0 +1,494 @@ +#pragma once + +// cspell:ignore ISTOGRAM +// The all-caps macro name XRPL_METRIC_HISTOGRAM_RECORD trips cspell's +// compound-word splitter, which emits the subword "ISTOGRAM"; ignore it here. + +/** + * Call-site OTel metric macros. + * + * Adds a new OTel metric instrument entirely at the call site -- no member + * field, no init line, no wrapper method in MetricsRegistry. Covers every + * instrument kind the OTel Metrics API defines: + * + * Synchronous (create-once via std::call_once, then record on every call): + * Counter XRPL_METRIC_COUNTER_INC / _ADD [+ _LABELED] + * UpDownCounter XRPL_METRIC_UPDOWN_ADD [+ _LABELED] + * Histogram XRPL_METRIC_HISTOGRAM_RECORD [+ _LABELED] + * Gauge XRPL_METRIC_GAUGE_RECORD [+ _LABELED] + * (requires OPENTELEMETRY_ABI_VERSION_NO >= 2; + * this repo currently builds ABI v1 -- see the + * static_assert branch below) + * + * Asynchronous/observable (register a callback ONCE, eagerly, during + * construction/init -- see the "Observable" note below): + * Observable Counter XRPL_METRIC_OBSERVABLE_COUNTER_REGISTER + * Observable UpDownCounter XRPL_METRIC_OBSERVABLE_UPDOWN_REGISTER + * Observable Gauge XRPL_METRIC_OBSERVABLE_GAUGE_REGISTER + * + * When XRPL_ENABLE_TELEMETRY is not defined, every macro expands to a + * no-op statement, so call sites never need their own #ifdef. + * + * Example usage -- plain counter: + * @code + * void RCLConsensus::Adaptor::doAccept() + * { + * // ... existing consensus-accept logic ... + * XRPL_METRIC_COUNTER_INC(app_, "ledgers_closed_total", + * "Total ledgers closed by consensus"); + * } + * @endcode + * + * Example usage -- labeled counter (edge case: per-reason tally): + * @code + * void TxQ::apply(...) + * { + * if (queueIsFull) + * XRPL_METRIC_COUNTER_INC_LABELED(app, "txq_dropped_total", + * "Transactions refused admission to the queue", + * ({{"reason", std::string("queue_full")}})); + * } + * @endcode + * + * Example usage -- UpDownCounter (edge case: value that can decrease): + * @code + * void ServerHandler::onRpcStart() + * { + * XRPL_METRIC_UPDOWN_ADD(app_, "rpc_in_flight_requests", + * "RPC requests currently executing", 1); + * } + * void ServerHandler::onRpcFinish() + * { + * XRPL_METRIC_UPDOWN_ADD(app_, "rpc_in_flight_requests", + * "RPC requests currently executing", -1); + * } + * @endcode + * + * Example usage -- observable gauge registered from a non-MetricsRegistry + * class (edge case: a subsystem exposing its own live state): + * @code + * SomeSubsystem::SomeSubsystem(ServiceRegistry& app) : app_(app) + * { + * XRPL_METRIC_OBSERVABLE_GAUGE_REGISTER( + * app_, "some_subsystem_queue_depth", "Current queue depth", + * [this] { return static_cast(queue_.size()); }); + * } + * @endcode + * + * @note A histogram whose values can exceed ~10,000 units (e.g. a + * microsecond duration beyond 10ms) needs an explicit-bucket View, which + * OTel can only register at MeterProvider construction time -- this + * cannot be done from a call site. See Limitation 2. Register such a + * view in MetricsRegistry::initExporterAndProvider() as today; the + * histogram-record call itself can still use the macro. + * + * @note Only call the SYNCHRONOUS macros (Counter/UpDownCounter/ + * Histogram/Gauge) from code that runs AFTER MetricsRegistry::start() has + * completed (RPC handlers, job callbacks, consensus rounds, tx apply, peer + * message handlers). See Limitation 1. + * + * @note The OBSERVABLE registration macros are the opposite: call them + * EAGERLY, exactly once, from constructor/init code -- never from a hot + * path. Repeated calls at the same call site register a NEW callback + * each time (no create-once caching, unlike the synchronous macros), + * which leaks callbacks. See Limitation 3. + * + * @note There is no way to read back a synchronous instrument's current + * accumulated value from application code -- the OTel API is + * write-only/push-based by design. If your logic needs both to record a + * metric AND read its running value, keep your own state (std::atomic or + * similar) and separately feed OTel via these macros. See "Use Case 4" in + * tasks/metric-macro-plan.md. + */ + +#include + +#include + +#ifdef XRPL_ENABLE_TELEMETRY + +#include +#include + +#define XRPL_METRIC_COUNTER_INC(app, name, description) \ + do \ + { \ + if (auto* xrpl_mr_ = (app).getMetricsRegistry(); xrpl_mr_ && xrpl_mr_->isEnabled()) \ + { \ + static opentelemetry::nostd::unique_ptr> \ + xrpl_counter_; \ + static std::once_flag xrpl_once_; \ + std::call_once(xrpl_once_, [&] { \ + if (auto xrpl_m_ = xrpl_mr_->meter()) \ + xrpl_counter_ = xrpl_m_->CreateUInt64Counter((name), (description)); \ + }); \ + if (xrpl_counter_) \ + xrpl_counter_->Add(1); \ + } \ + } while (false) + +// The label set is passed as trailing variadic arguments so a +// brace-enclosed initializer list (e.g. {{"reason", std::string("x")}}), +// which contains a top-level comma, survives preprocessing as a single +// logical argument. __VA_ARGS__ re-joins it verbatim into the Add() call. +#define XRPL_METRIC_COUNTER_INC_LABELED(app, name, description, ...) \ + do \ + { \ + if (auto* xrpl_mr_ = (app).getMetricsRegistry(); xrpl_mr_ && xrpl_mr_->isEnabled()) \ + { \ + static opentelemetry::nostd::unique_ptr> \ + xrpl_counter_; \ + static std::once_flag xrpl_once_; \ + std::call_once(xrpl_once_, [&] { \ + if (auto xrpl_m_ = xrpl_mr_->meter()) \ + xrpl_counter_ = xrpl_m_->CreateUInt64Counter((name), (description)); \ + }); \ + if (xrpl_counter_) \ + xrpl_counter_->Add(1, __VA_ARGS__); \ + } \ + } while (false) + +// Same as XRPL_METRIC_COUNTER_INC, but increments by a caller-supplied amount +// instead of a fixed 1 (e.g. bytes transferred, batch sizes). +#define XRPL_METRIC_COUNTER_ADD(app, name, description, amount) \ + do \ + { \ + if (auto* xrpl_mr_ = (app).getMetricsRegistry(); xrpl_mr_ && xrpl_mr_->isEnabled()) \ + { \ + static opentelemetry::nostd::unique_ptr> \ + xrpl_counter_; \ + static std::once_flag xrpl_once_; \ + std::call_once(xrpl_once_, [&] { \ + if (auto xrpl_m_ = xrpl_mr_->meter()) \ + xrpl_counter_ = xrpl_m_->CreateUInt64Counter((name), (description)); \ + }); \ + if (xrpl_counter_) \ + xrpl_counter_->Add(amount); \ + } \ + } while (false) + +// amount is fixed; the trailing variadic args carry the label set (see the +// note on XRPL_METRIC_COUNTER_INC_LABELED for why labels are variadic). +#define XRPL_METRIC_COUNTER_ADD_LABELED(app, name, description, amount, ...) \ + do \ + { \ + if (auto* xrpl_mr_ = (app).getMetricsRegistry(); xrpl_mr_ && xrpl_mr_->isEnabled()) \ + { \ + static opentelemetry::nostd::unique_ptr> \ + xrpl_counter_; \ + static std::once_flag xrpl_once_; \ + std::call_once(xrpl_once_, [&] { \ + if (auto xrpl_m_ = xrpl_mr_->meter()) \ + xrpl_counter_ = xrpl_m_->CreateUInt64Counter((name), (description)); \ + }); \ + if (xrpl_counter_) \ + xrpl_counter_->Add(amount, __VA_ARGS__); \ + } \ + } while (false) + +// UpDownCounter: like COUNTER_ADD, but the underlying instrument permits a +// negative amount (Use Case 2 -- e.g. in-flight request count, +1 on start +// / -1 on finish from two different points in the same or different call +// sites). A plain Counter's Add() must never see a negative value per the +// OTel API contract; use this macro, not COUNTER_ADD, whenever the value +// can decrease. +#define XRPL_METRIC_UPDOWN_ADD(app, name, description, amount) \ + do \ + { \ + if (auto* xrpl_mr_ = (app).getMetricsRegistry(); xrpl_mr_ && xrpl_mr_->isEnabled()) \ + { \ + static opentelemetry::nostd::unique_ptr< \ + opentelemetry::metrics::UpDownCounter> \ + xrpl_updown_; \ + static std::once_flag xrpl_once_; \ + std::call_once(xrpl_once_, [&] { \ + if (auto xrpl_m_ = xrpl_mr_->meter()) \ + xrpl_updown_ = xrpl_m_->CreateInt64UpDownCounter((name), (description)); \ + }); \ + if (xrpl_updown_) \ + xrpl_updown_->Add(amount); \ + } \ + } while (false) + +// amount may be negative; the trailing variadic args carry the label set +// (see the note on XRPL_METRIC_COUNTER_INC_LABELED for why labels are variadic). +#define XRPL_METRIC_UPDOWN_ADD_LABELED(app, name, description, amount, ...) \ + do \ + { \ + if (auto* xrpl_mr_ = (app).getMetricsRegistry(); xrpl_mr_ && xrpl_mr_->isEnabled()) \ + { \ + static opentelemetry::nostd::unique_ptr< \ + opentelemetry::metrics::UpDownCounter> \ + xrpl_updown_; \ + static std::once_flag xrpl_once_; \ + std::call_once(xrpl_once_, [&] { \ + if (auto xrpl_m_ = xrpl_mr_->meter()) \ + xrpl_updown_ = xrpl_m_->CreateInt64UpDownCounter((name), (description)); \ + }); \ + if (xrpl_updown_) \ + xrpl_updown_->Add(amount, __VA_ARGS__); \ + } \ + } while (false) + +#define XRPL_METRIC_HISTOGRAM_RECORD(app, name, description, value) \ + do \ + { \ + if (auto* xrpl_mr_ = (app).getMetricsRegistry(); xrpl_mr_ && xrpl_mr_->isEnabled()) \ + { \ + static opentelemetry::nostd::unique_ptr> \ + xrpl_hist_; \ + static std::once_flag xrpl_once_; \ + std::call_once(xrpl_once_, [&] { \ + if (auto xrpl_m_ = xrpl_mr_->meter()) \ + xrpl_hist_ = xrpl_m_->CreateDoubleHistogram((name), (description)); \ + }); \ + if (xrpl_hist_) \ + xrpl_hist_->Record(static_cast(value), opentelemetry::context::Context{}); \ + } \ + } while (false) + +// value is fixed; the trailing variadic args carry the label set (see the +// note on XRPL_METRIC_COUNTER_INC_LABELED for why labels are variadic). +#define XRPL_METRIC_HISTOGRAM_RECORD_LABELED(app, name, description, value, ...) \ + do \ + { \ + if (auto* xrpl_mr_ = (app).getMetricsRegistry(); xrpl_mr_ && xrpl_mr_->isEnabled()) \ + { \ + static opentelemetry::nostd::unique_ptr> \ + xrpl_hist_; \ + static std::once_flag xrpl_once_; \ + std::call_once(xrpl_once_, [&] { \ + if (auto xrpl_m_ = xrpl_mr_->meter()) \ + xrpl_hist_ = xrpl_m_->CreateDoubleHistogram((name), (description)); \ + }); \ + if (xrpl_hist_) \ + xrpl_hist_->Record( \ + static_cast(value), __VA_ARGS__, opentelemetry::context::Context{}); \ + } \ + } while (false) + +// Synchronous Gauge: last-value snapshot, not a distribution (contrast +// Histogram) and not a running total (contrast Counter/UpDownCounter). +// ABI-gated: opentelemetry-cpp only exposes CreateInt64Gauge/ +// CreateDoubleGauge when OPENTELEMETRY_ABI_VERSION_NO >= 2. This project's +// Conan build currently pins ABI v1 (verified: +// .build/build/generators/opentelemetry-cpp-release-x86_64-data.cmake sets +// OPENTELEMETRY_ABI_VERSION_NO=1), so the real path below is presently +// dead code on this codebase's build -- shipped anyway so it activates +// automatically the day the ABI version is bumped, and so a developer who +// reaches for "just the current value, not a distribution" sees an +// actionable compile error now instead of silently reaching for the wrong +// instrument kind (misusing Histogram or UpDownCounter as a gauge +// substitute is explicitly discouraged -- see Design/taxonomy section). +#if OPENTELEMETRY_ABI_VERSION_NO >= 2 +#define XRPL_METRIC_GAUGE_RECORD(app, name, description, value) \ + do \ + { \ + if (auto* xrpl_mr_ = (app).getMetricsRegistry(); xrpl_mr_ && xrpl_mr_->isEnabled()) \ + { \ + static opentelemetry::nostd::unique_ptr> \ + xrpl_gauge_; \ + static std::once_flag xrpl_once_; \ + std::call_once(xrpl_once_, [&] { \ + if (auto xrpl_m_ = xrpl_mr_->meter()) \ + xrpl_gauge_ = xrpl_m_->CreateDoubleGauge((name), (description)); \ + }); \ + if (xrpl_gauge_) \ + xrpl_gauge_->Record( \ + static_cast(value), opentelemetry::context::Context{}); \ + } \ + } while (false) +// value is fixed; the trailing variadic args carry the label set (see the +// note on XRPL_METRIC_COUNTER_INC_LABELED for why labels are variadic). +#define XRPL_METRIC_GAUGE_RECORD_LABELED(app, name, description, value, ...) \ + do \ + { \ + if (auto* xrpl_mr_ = (app).getMetricsRegistry(); xrpl_mr_ && xrpl_mr_->isEnabled()) \ + { \ + static opentelemetry::nostd::unique_ptr> \ + xrpl_gauge_; \ + static std::once_flag xrpl_once_; \ + std::call_once(xrpl_once_, [&] { \ + if (auto xrpl_m_ = xrpl_mr_->meter()) \ + xrpl_gauge_ = xrpl_m_->CreateDoubleGauge((name), (description)); \ + }); \ + if (xrpl_gauge_) \ + xrpl_gauge_->Record( \ + static_cast(value), __VA_ARGS__, opentelemetry::context::Context{}); \ + } \ + } while (false) +#else +#define XRPL_METRIC_GAUGE_RECORD(app, name, description, value) \ + static_assert( \ + false, \ + "XRPL_METRIC_GAUGE_RECORD requires OPENTELEMETRY_ABI_VERSION_NO >= 2 (this build " \ + "uses ABI v1). Use XRPL_METRIC_OBSERVABLE_GAUGE_REGISTER with your own state, or " \ + "bump OPENTELEMETRY_ABI_VERSION_NO as a separate, reviewed change.") +#define XRPL_METRIC_GAUGE_RECORD_LABELED(app, name, description, value, ...) \ + static_assert( \ + false, \ + "XRPL_METRIC_GAUGE_RECORD_LABELED requires OPENTELEMETRY_ABI_VERSION_NO >= 2 " \ + "(this build uses ABI v1). Use XRPL_METRIC_OBSERVABLE_GAUGE_REGISTER with your " \ + "own state, or bump OPENTELEMETRY_ABI_VERSION_NO as a separate, reviewed change.") +#endif // OPENTELEMETRY_ABI_VERSION_NO >= 2 + +// ----------------------------------------------------------------- +// Observable/async instrument registration (Use Case 5). Unlike the +// synchronous macros above, these do NOT lazily create-on-first-call -- +// they register a callback with the SDK immediately, at the call site, +// the moment the macro executes. Callers MUST invoke this during +// construction/init, before the server is fully live (same timing rule +// MetricsRegistry::registerAsyncGauges() already follows for its own +// gauges -- see Limitation 3). Calling it from a hot-path function +// instead of an init path re-registers a new callback on every call, +// which leaks callbacks and is NOT what this macro is for. +// +// The callable is captured in a heap-allocated std::function, and its +// address is passed as the `void* state` to AddCallback (whose signature, +// `void (*)(ObserverResult, void*)`, is a raw C function pointer -- it +// cannot bind a capturing lambda directly). A static trampoline +// function reinterprets `state` back to the std::function and invokes +// it inside the callback. The heap allocation is intentionally leaked +// for the process lifetime (matches every existing ObservableGauge in +// MetricsRegistry, which are member fields with the same lifetime as +// the registry itself) -- do not "fix" this with a smart pointer that +// frees before the reader thread's last collection tick. +// ----------------------------------------------------------------- +#define XRPL_METRIC_OBSERVABLE_GAUGE_REGISTER(app, name, description, valueFn) \ + do \ + { \ + if (auto* xrpl_mr_ = (app).getMetricsRegistry(); xrpl_mr_ && xrpl_mr_->isEnabled()) \ + { \ + if (auto xrpl_m_ = xrpl_mr_->meter()) \ + { \ + auto* xrpl_fn_ = new std::function(valueFn); \ + auto xrpl_inst_ = xrpl_m_->CreateInt64ObservableGauge((name), (description)); \ + xrpl_inst_->AddCallback( \ + [](opentelemetry::metrics::ObserverResult result, void* state) { \ + auto* fn = static_cast*>(state); \ + try \ + { \ + opentelemetry::nostd::get>>(result) \ + ->Observe((*fn)()); \ + } \ + catch (...) \ + { \ + } \ + }, \ + xrpl_fn_); \ + } \ + } \ + } while (false) + +#define XRPL_METRIC_OBSERVABLE_COUNTER_REGISTER(app, name, description, valueFn) \ + do \ + { \ + if (auto* xrpl_mr_ = (app).getMetricsRegistry(); xrpl_mr_ && xrpl_mr_->isEnabled()) \ + { \ + if (auto xrpl_m_ = xrpl_mr_->meter()) \ + { \ + auto* xrpl_fn_ = new std::function(valueFn); \ + auto xrpl_inst_ = xrpl_m_->CreateInt64ObservableCounter((name), (description)); \ + xrpl_inst_->AddCallback( \ + [](opentelemetry::metrics::ObserverResult result, void* state) { \ + auto* fn = static_cast*>(state); \ + try \ + { \ + opentelemetry::nostd::get>>(result) \ + ->Observe((*fn)()); \ + } \ + catch (...) \ + { \ + } \ + }, \ + xrpl_fn_); \ + } \ + } \ + } while (false) + +#define XRPL_METRIC_OBSERVABLE_UPDOWN_REGISTER(app, name, description, valueFn) \ + do \ + { \ + if (auto* xrpl_mr_ = (app).getMetricsRegistry(); xrpl_mr_ && xrpl_mr_->isEnabled()) \ + { \ + if (auto xrpl_m_ = xrpl_mr_->meter()) \ + { \ + auto* xrpl_fn_ = new std::function(valueFn); \ + auto xrpl_inst_ = \ + xrpl_m_->CreateInt64ObservableUpDownCounter((name), (description)); \ + xrpl_inst_->AddCallback( \ + [](opentelemetry::metrics::ObserverResult result, void* state) { \ + auto* fn = static_cast*>(state); \ + try \ + { \ + opentelemetry::nostd::get>>(result) \ + ->Observe((*fn)()); \ + } \ + catch (...) \ + { \ + } \ + }, \ + xrpl_fn_); \ + } \ + } \ + } while (false) + +#else // !XRPL_ENABLE_TELEMETRY + +#define XRPL_METRIC_COUNTER_INC(app, name, description) \ + do \ + { \ + } while (false) +#define XRPL_METRIC_COUNTER_INC_LABELED(app, name, description, ...) \ + do \ + { \ + } while (false) +#define XRPL_METRIC_COUNTER_ADD(app, name, description, amount) \ + do \ + { \ + } while (false) +#define XRPL_METRIC_COUNTER_ADD_LABELED(app, name, description, amount, ...) \ + do \ + { \ + } while (false) +#define XRPL_METRIC_UPDOWN_ADD(app, name, description, amount) \ + do \ + { \ + } while (false) +#define XRPL_METRIC_UPDOWN_ADD_LABELED(app, name, description, amount, ...) \ + do \ + { \ + } while (false) +#define XRPL_METRIC_HISTOGRAM_RECORD(app, name, description, value) \ + do \ + { \ + } while (false) +#define XRPL_METRIC_HISTOGRAM_RECORD_LABELED(app, name, description, value, ...) \ + do \ + { \ + } while (false) +#define XRPL_METRIC_GAUGE_RECORD(app, name, description, value) \ + do \ + { \ + } while (false) +#define XRPL_METRIC_GAUGE_RECORD_LABELED(app, name, description, value, ...) \ + do \ + { \ + } while (false) +#define XRPL_METRIC_OBSERVABLE_GAUGE_REGISTER(app, name, description, valueFn) \ + do \ + { \ + } while (false) +#define XRPL_METRIC_OBSERVABLE_COUNTER_REGISTER(app, name, description, valueFn) \ + do \ + { \ + } while (false) +#define XRPL_METRIC_OBSERVABLE_UPDOWN_REGISTER(app, name, description, valueFn) \ + do \ + { \ + } while (false) + +#endif // XRPL_ENABLE_TELEMETRY diff --git a/src/xrpld/telemetry/MetricsRegistry.h b/src/xrpld/telemetry/MetricsRegistry.h index 85aed97e2c..17f0dc3718 100644 --- a/src/xrpld/telemetry/MetricsRegistry.h +++ b/src/xrpld/telemetry/MetricsRegistry.h @@ -204,8 +204,15 @@ namespace telemetry { * @note Extending: * - Adding a new CountedObject type is auto-picked up by the * object_count gauge via iteration. - * - Adding a new synchronous instrument requires a header field, - * an initializer in start(), and a record method. + * - Adding a new SYNCHRONOUS instrument (counter/histogram): prefer the + * XRPL_METRIC_* call-site macros in MetricMacros.h -- no header/cpp + * edit needed. Fall back to a dedicated member + init line + record + * method (the pattern below) only when the metric needs to be read + * back by other code (e.g. ValidationTracker-style accumulation) or + * needs a custom histogram bucket View (see MetricMacros.h Limitation + * 2 in tasks/metric-macro-plan.md). + * - Adding a new OBSERVABLE gauge still requires eager central + * registration -- pull-model instruments cannot be lazily created. */ class MetricsRegistry { @@ -337,8 +344,12 @@ public: /** * Increment the ledgers_closed_total counter. - * Called from RCLConsensus::Adaptor::doAccept() after a ledger is - * accepted by consensus. + * + * @note Currently has no callers: the ledgers_closed_total counter is + * incremented at its consensus call site via the XRPL_METRIC_COUNTER_INC + * macro (see MetricMacros.h). This method and its eagerly-created + * counter are retained as a fallback and are slated for removal in a + * separate cleanup once the macro path has proven out. */ void incrementLedgersClosed(); @@ -410,6 +421,22 @@ public: return validationTracker_; } +#ifdef XRPL_ENABLE_TELEMETRY + /** + * Access the shared OTel Meter for call-site instrument creation. + * Used by the XRPL_METRIC_* macros (MetricMacros.h) so new synchronous + * counters/histograms can be declared at their call site instead of as + * MetricsRegistry members. Returns an empty (falsy) shared_ptr before + * start() has run or when disabled. + * @return The shared Meter, or empty if not yet started. + */ + opentelemetry::nostd::shared_ptr + meter() const noexcept + { + return meter_; + } +#endif + private: /** * Master enable flag; when false all methods are no-ops.