From 3ad525a48a77df3241eb3e38d13601a4f66d68a9 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Wed, 29 Jul 2026 14:33:05 +0100 Subject: [PATCH 1/7] =?UTF-8?q?docs(telemetry):=20drop=20speculative=20fil?= =?UTF-8?q?e/line=20estimate=20tables=20from=20=C2=A73.9?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The §3.9.1 "Files Modified Summary" and §3.9.2 "Detailed File Impact" tables carried hand-maintained per-component line counts that had drifted from the plan: the Lines Added column summed to 1,565 while the Total row claimed ~1,670. Files (34) and Lines Changed (120) reconciled, so only the Added total was stale — residue from expanding Core Telemetry 5 -> 11 files. Rather than patch one cell, remove both tables. They were pre-implementation estimates with no source of truth, so any figure in them drifts again on the next phase. §3.1's directory tree remains the canonical list of the 11 new telemetry files, and §3.9.3-3.9.7 (risk, architectural impact, backward compatibility, rollback) carry the assessment without inventing line counts. No references to the removed sections exist elsewhere in the plan docs. --- .../03-implementation-strategy.md | 73 ------------------- 1 file changed, 73 deletions(-) diff --git a/OpenTelemetryPlan/03-implementation-strategy.md b/OpenTelemetryPlan/03-implementation-strategy.md index 4edcef6194..7cf0f9d445 100644 --- a/OpenTelemetryPlan/03-implementation-strategy.md +++ b/OpenTelemetryPlan/03-implementation-strategy.md @@ -339,79 +339,6 @@ Instrumentation is gated on two levels. A compile-time feature flag (`XRPL_ENABL This section provides a detailed assessment of how intrusive the OpenTelemetry integration is to the existing xrpld codebase. -### 3.9.1 Files Modified Summary - -| Component | Files Modified | Lines Added | Lines Changed | Architectural Impact | -| --------------------- | -------------- | ----------- | ------------- | -------------------- | -| **Core Telemetry** | 11 new files | ~980 | 0 | None (new module) | -| **Application Init** | 2 files | ~30 | ~5 | Minimal | -| **RPC Layer** | 3 files | ~80 | ~20 | Minimal | -| **Transaction Relay** | 4 files | ~120 | ~40 | Low | -| **Consensus** | 3 files | ~100 | ~30 | Low-Medium | -| **Protocol Buffers** | 1 file | ~25 | 0 | Low | -| **CMake/Build** | 3 files | ~50 | ~10 | Minimal | -| **PathFinding** | 2 | ~80 | ~5 | Minimal | -| **TxQ/Fee** | 2 | ~60 | ~5 | Minimal | -| **Validator/Amend** | 3 | ~40 | ~5 | Minimal | -| **Total** | **~34 files** | **~1,670** | **~120** | **Low** | - -### 3.9.2 Detailed File Impact - -```mermaid -pie title Code Changes by Component - "New Telemetry Module" : 800 - "Transaction Relay" : 160 - "Consensus" : 130 - "RPC Layer" : 100 - "PathFinding" : 80 - "TxQ/Fee" : 60 - "Validator/Amendment" : 40 - "Application Init" : 35 - "Protocol Buffers" : 25 - "Build System" : 60 -``` - -#### New Files (No Impact on Existing Code) - -| File | Lines | Purpose | -| ------------------------------------------------ | ----- | ------------------------ | -| `include/xrpl/telemetry/Telemetry.h` | ~160 | Main interface | -| `include/xrpl/telemetry/TelemetryConfig.h` | ~80 | Configuration structures | -| `include/xrpl/telemetry/TraceContext.h` | ~80 | Context propagation | -| `include/xrpl/telemetry/SpanGuard.h` | ~120 | RAII wrapper | -| `include/xrpl/telemetry/SpanAttributes.h` | ~60 | Attribute helpers | -| `src/libxrpl/telemetry/Telemetry.cpp` | ~200 | Implementation | -| `src/libxrpl/telemetry/TelemetryConfig.cpp` | ~60 | Config parsing | -| `src/libxrpl/telemetry/TraceContext.cpp` | ~80 | Context serialization | -| `src/libxrpl/telemetry/NullTelemetry.cpp` | ~40 | No-op implementation | -| `src/xrpld/telemetry/TracingInstrumentation.h` | ~60 | Macros | -| `src/xrpld/telemetry/TracingInstrumentation.cpp` | ~40 | Instrumentation impl | - -#### Modified Files (Existing Xrpld Code) - -| File | Lines Added | Lines Changed | Risk Level | -| ------------------------------------------------- | ----------- | ------------- | ---------- | -| `src/xrpld/app/main/Application.cpp` | ~15 | ~3 | Low | -| `include/xrpl/core/ServiceRegistry.h` | ~5 | ~2 | Low | -| `src/xrpld/rpc/detail/ServerHandler.cpp` | ~40 | ~10 | Low | -| `src/xrpld/rpc/handlers/*.cpp` | ~30 | ~8 | Low | -| `src/xrpld/overlay/detail/PeerImp.cpp` | ~60 | ~15 | Medium | -| `src/xrpld/overlay/detail/OverlayImpl.cpp` | ~30 | ~10 | Medium | -| `src/xrpld/app/consensus/RCLConsensus.cpp` | ~50 | ~15 | Medium | -| `src/xrpld/app/consensus/RCLConsensusAdaptor.cpp` | ~40 | ~12 | Medium | -| `src/xrpld/core/JobQueue.cpp` | ~20 | ~5 | Low | -| `src/xrpld/app/paths/PathRequest.cpp` | ~40 | ~3 | Low | -| `src/xrpld/app/paths/Pathfinder.cpp` | ~40 | ~2 | Low | -| `src/xrpld/app/misc/TxQ.cpp` | ~40 | ~3 | Low | -| `src/xrpld/app/main/LoadManager.cpp` | ~20 | ~2 | Low | -| `src/xrpld/app/misc/ValidatorList.cpp` | ~20 | ~2 | Low | -| `src/xrpld/app/misc/AmendmentTable.cpp` | ~10 | ~2 | Low | -| `src/xrpld/app/misc/Manifest.cpp` | ~10 | ~1 | Low | -| `src/xrpld/shamap/SHAMap.cpp` | ~20 | ~3 | Low | -| `src/xrpld/overlay/detail/ripple.proto` | ~25 | 0 | Low | -| `CMakeLists.txt` | ~40 | ~8 | Low | -| `cmake/FindOpenTelemetry.cmake` | ~50 | 0 | None (new) | - ### 3.9.3 Risk Assessment by Component
From 17ff797633db4b7eb006dedac8c566dfe93f3af8 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Wed, 29 Jul 2026 15:06:31 +0100 Subject: [PATCH 2/7] refactor(telemetry): build span names with std::format SpanGuard::span() and freshRoot() joined the prefix and suffix by hand with reserve() + three append() calls. std::format expresses the same join in one line; libstdc++ has shipped since GCC 13 and the project floor is GCC 15.2 (BUILD.md), so it is available on every supported compiler. Both callers are noexcept and std::format can throw (std::bad_alloc, or std::format_error on a malformed spec), so an escaping exception would terminate the process. The call is wrapped in a joinSpanName() helper that catches and returns std::nullopt; the caller then returns a null guard, which is the same degrade-to-no-op path already taken when telemetry is disabled. Telemetry must never bring the node down. One helper rather than a try/catch at each site keeps both call sites a single line and puts the exception-safety requirement in one documented place. Addresses a review comment on PR #6437. --- src/libxrpl/telemetry/SpanGuard.cpp | 45 ++++++++++++++++++++++++----- 1 file changed, 37 insertions(+), 8 deletions(-) diff --git a/src/libxrpl/telemetry/SpanGuard.cpp b/src/libxrpl/telemetry/SpanGuard.cpp index 84b23f44cd..a165d0ad5a 100644 --- a/src/libxrpl/telemetry/SpanGuard.cpp +++ b/src/libxrpl/telemetry/SpanGuard.cpp @@ -47,6 +47,7 @@ #include #include +#include #include #include #include @@ -186,6 +187,34 @@ categoryToSpanKind(TraceCategory cat) return otel_trace::SpanKind::kInternal; // unreachable } +/** + * Join a span-name prefix and suffix into the dotted full name. + * + * Wraps std::format because the callers are noexcept: std::format can throw + * (std::bad_alloc, or std::format_error on a malformed spec) and an escaping + * exception would terminate the process. Telemetry must never take the node + * down, so a failure yields std::nullopt and the caller returns a null guard — + * the same degrade-to-no-op path already used when telemetry is disabled. + * + * @param prefix Segment before the dot (e.g. "consensus"). + * @param name Segment after the dot (e.g. "round"). + * @return The joined name, or std::nullopt if formatting failed. + */ +[[nodiscard]] std::optional +joinSpanName(std::string_view prefix, std::string_view name) noexcept +{ + try + { + return std::format("{}.{}", prefix, name); + } + catch (std::exception const&) + { + // Out of memory or a bad format spec. Drop the span rather than + // propagate out of a noexcept factory. + return std::nullopt; + } +} + } // namespace SpanGuard @@ -194,10 +223,10 @@ SpanGuard::span(TraceCategory cat, std::string_view prefix, std::string_view nam auto* tel = Telemetry::getInstance(); if ((tel == nullptr) || !tel->isEnabled() || !isCategoryEnabled(*tel, cat)) return {}; - std::string fullName; - fullName.reserve(prefix.size() + 1 + name.size()); - fullName.append(prefix).append(1, '.').append(name); - return SpanGuard(std::make_unique(tel->startSpan(fullName, categoryToSpanKind(cat)))); + auto const fullName = joinSpanName(prefix, name); + if (!fullName) + return {}; + return SpanGuard(std::make_unique(tel->startSpan(*fullName, categoryToSpanKind(cat)))); } SpanGuard @@ -206,13 +235,13 @@ SpanGuard::freshRoot(TraceCategory cat, std::string_view prefix, std::string_vie auto* tel = Telemetry::getInstance(); if ((tel == nullptr) || !tel->isEnabled() || !isCategoryEnabled(*tel, cat)) return {}; - std::string fullName; - fullName.reserve(prefix.size() + 1 + name.size()); - fullName.append(prefix).append(1, '.').append(name); + auto const fullName = joinSpanName(prefix, name); + if (!fullName) + return {}; // Force a fresh trace root: do NOT inherit this thread's active span. auto rootCtx = opentelemetry::context::Context{otel_trace::kIsRootSpanKey, true}; return SpanGuard( - std::make_unique(tel->startSpan(fullName, rootCtx, categoryToSpanKind(cat)))); + std::make_unique(tel->startSpan(*fullName, rootCtx, categoryToSpanKind(cat)))); } // ===== Child / linked span creation ======================================== From 3715b7a2a3318078e4154bf4cab045378ee97fd7 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Wed, 29 Jul 2026 16:02:19 +0100 Subject: [PATCH 3/7] fix(telemetry): correct RPC and gRPC span status reporting Six related defects in the RPC/gRPC span surface, all cases where a failure was recorded as success or an attribute was missing on an error path. GRPCServer: the non-exception branch set the span Ok unconditionally, then sent a possibly-failed grpc::Status. The handler can return a non-OK status without throwing, so every failed call traced as successful. Status now follows result.second, with the error message as the span description. ServerHandler: eight per-item error branches appended an error reply without recording that the request failed. Batch responses and ripplerpc < 3.0 always carry HTTP 200, so those failures were invisible and an entirely failed batch ended its span as successful. Added an appendItemError() helper next to the existing httpReplyError() lambda and routed all eight sites through it, so the flag cannot be forgotten at a new call site. ServerHandler: the early-return validation paths set the span error but not the rpc_status attribute. Added it to httpReplyError() so every such path gets it. RPCHandler: the fillHandler error path set only command and rpc_status, while callMethod sets command, version and rpc_role. Error spans were therefore not filterable by API version or role. The error path now mirrors that set. RPCHandler: resolveCommandSpanName() checked only that command/method were present, not that they agreed, while fillHandler rejects a mismatch as rpcUNKNOWN_COMMAND. A request supplying both with different values was labelled with one of the two names, misattributing the error to a command that never dispatched. It now mirrors fillHandler's rule and collapses to "unknown". ServerHandler: processRequest returned bool solely so the caller could set its span status. Telemetry should read state, not shape the signature of the code it observes, so the signature returns to void and rpc.process sets its own status from spanHadError. The enclosing rpc.http_request span now leaves status unset: the OTel spec has instrumentation leave status unset unless the operation itself errored, and reserves Ok for an operator asserting verified success. --- src/xrpld/app/main/GRPCServer.cpp | 15 +++++- src/xrpld/rpc/ServerHandler.h | 4 +- src/xrpld/rpc/detail/RPCHandler.cpp | 15 ++++++ src/xrpld/rpc/detail/ServerHandler.cpp | 69 ++++++++++++++------------ 4 files changed, 67 insertions(+), 36 deletions(-) diff --git a/src/xrpld/app/main/GRPCServer.cpp b/src/xrpld/app/main/GRPCServer.cpp index 566362bb0f..f217771025 100644 --- a/src/xrpld/app/main/GRPCServer.cpp +++ b/src/xrpld/app/main/GRPCServer.cpp @@ -243,8 +243,19 @@ GRPCServerImpl::CallData::process(std::shared_ptr result = handler_(context); setIsUnlimited(result.first, isUnlimited); - span.setAttribute(grpc_span::attr::grpcStatus, grpc_span::val::success); - span.setOk(); + // The handler can return a non-OK status without throwing, so + // the span status must follow result.second rather than assume + // success — otherwise every failed call traces as OK. + if (result.second.ok()) + { + span.setAttribute(grpc_span::attr::grpcStatus, grpc_span::val::success); + span.setOk(); + } + else + { + span.setAttribute(grpc_span::attr::grpcStatus, grpc_span::val::error); + span.setError(result.second.error_message()); + } responder_.Finish(result.first, result.second, this); } } diff --git a/src/xrpld/rpc/ServerHandler.h b/src/xrpld/rpc/ServerHandler.h index fbe487654e..5dfd2476da 100644 --- a/src/xrpld/rpc/ServerHandler.h +++ b/src/xrpld/rpc/ServerHandler.h @@ -194,10 +194,8 @@ private: /** * Process an RPC request and write the reply to `output`. - * @return false if the request resulted in an error response, true - * otherwise. Lets the caller's enclosing span reflect the outcome. */ - bool + void processRequest( Port const& port, std::string const& request, diff --git a/src/xrpld/rpc/detail/RPCHandler.cpp b/src/xrpld/rpc/detail/RPCHandler.cpp index 27dacdc560..8efe957435 100644 --- a/src/xrpld/rpc/detail/RPCHandler.cpp +++ b/src/xrpld/rpc/detail/RPCHandler.cpp @@ -238,6 +238,14 @@ resolveCommandSpanName(JsonContext const& context) if (!context.params.isMember(jss::command) && !context.params.isMember(jss::method)) return rpc_span::val::unknownCommand; + // fillHandler() rejects a request that supplies both fields with differing + // values as rpcUNKNOWN_COMMAND. Mirror that here, or the span would be + // labelled with one of the two names and misattribute the error to a + // command that was never dispatched. + if (context.params.isMember(jss::command) && context.params.isMember(jss::method) && + context.params[jss::command].asString() != context.params[jss::method].asString()) + return rpc_span::val::unknownCommand; + std::string const cmd = context.params.isMember(jss::command) ? context.params[jss::command].asString() : context.params[jss::method].asString(); @@ -261,6 +269,13 @@ doCommand(RPC::JsonContext& context, json::Value& result) auto const cmdName = resolveCommandSpanName(context); auto span = ScopedSpanGuard(TraceCategory::Rpc, rpc_span::prefix::command, cmdName); span.setAttribute(rpc_span::attr::command, cmdName); + // Mirror the attribute set callMethod() puts on a successful command + // span, so error spans stay filterable by API version and role. + span.setAttribute(rpc_span::attr::version, static_cast(context.apiVersion)); + span.setAttribute( + rpc_span::attr::rpcRole, + context.role == Role::ADMIN ? std::string_view(rpc_span::val::admin) + : std::string_view(rpc_span::val::user)); span.setAttribute(rpc_span::attr::rpcStatus, rpc_span::val::error); span.setError(getErrorInfo(error).token.cStr()); diff --git a/src/xrpld/rpc/detail/ServerHandler.cpp b/src/xrpld/rpc/detail/ServerHandler.cpp index 4abee4c79f..0661d78826 100644 --- a/src/xrpld/rpc/detail/ServerHandler.cpp +++ b/src/xrpld/rpc/detail/ServerHandler.cpp @@ -593,7 +593,7 @@ ServerHandler::processSession( auto span = ScopedSpanGuard::freshRoot( TraceCategory::Rpc, rpc_span::prefix::rpc, rpc_span::op::httpRequest); - bool const ok = processRequest( + processRequest( session->port(), buffersToString(session->request().body().data()), session->remoteAddress().atPort(0), @@ -615,15 +615,11 @@ ServerHandler::processSession( { session->close(true); } - // Reflect the request outcome on the wrapper span instead of always OK. - if (ok) - { - span.setOk(); - } - else - { - span.setError(rpc_span::val::error); - } + // Status is left unset: the OTel spec says instrumentation should leave it + // unset unless the operation itself errored, and reserves Ok for an + // operator asserting success. This span only delimits the HTTP request and + // has no error of its own to report — the outcome is determined inside + // rpc.process, which sets its own status. } static json::Value @@ -642,7 +638,7 @@ constexpr json::Int kServerOverloaded = -32604; constexpr json::Int kForbidden = -32605; constexpr json::Int kWrongVersion = -32606; -bool +void ServerHandler::processRequest( Port const& port, std::string const& request, @@ -671,6 +667,7 @@ ServerHandler::processRequest( // (the span would otherwise end UNSET, invisible to {status.code=error}). auto httpReplyError = [&](int status, std::string const& message) { spanHadError = true; + span.setAttribute(rpc_span::attr::rpcStatus, rpc_span::val::error); span.setError(message); httpReply(status, message, output, rpcJ); }; @@ -682,7 +679,7 @@ ServerHandler::processRequest( !jsonOrig || !jsonOrig.isObject()) { httpReplyError(400, "Unable to parse request: " + reader.getFormattedErrorMessages()); - return false; + return; } } @@ -694,12 +691,23 @@ ServerHandler::processRequest( if (!jsonOrig.isMember(jss::params) || !jsonOrig[jss::params].isArray()) { httpReplyError(400, "Malformed batch request"); - return false; + return; } size = jsonOrig[jss::params].size(); } json::Value reply(batch ? json::ValueType::Array : json::ValueType::Object); + + // Append a per-request error item and record that the request failed. + // Batch responses (and ripplerpc < 3.0) always carry HTTP 200, so the + // span status can only learn about these failures through spanHadError. + // Every per-item error path must go through here, or an entirely failed + // batch would end its span as successful. + auto appendItemError = [&](json::Value&& item) { + spanHadError = true; + reply.append(std::move(item)); + }; + auto const start(std::chrono::high_resolution_clock::now()); for (unsigned i = 0; i < size; ++i) { @@ -710,7 +718,7 @@ ServerHandler::processRequest( json::Value r(json::ValueType::Object); r[jss::request] = jsonRPC; r[jss::error] = makeJsonError(kMethodNotFound, "Method not found"); - reply.append(r); + appendItemError(std::move(r)); continue; } @@ -733,12 +741,12 @@ ServerHandler::processRequest( if (!batch) { httpReplyError(400, jss::invalid_API_version.cStr()); - return false; + return; } json::Value r(json::ValueType::Object); r[jss::request] = jsonRPC; r[jss::error] = makeJsonError(kWrongVersion, jss::invalid_API_version.cStr()); - reply.append(r); + appendItemError(std::move(r)); continue; } @@ -776,11 +784,11 @@ ServerHandler::processRequest( if (!batch) { httpReplyError(503, "Server is overloaded"); - return false; + return; } json::Value r = jsonRPC; r[jss::error] = makeJsonError(kServerOverloaded, "Server is overloaded"); - reply.append(r); + appendItemError(std::move(r)); continue; } } @@ -791,11 +799,11 @@ ServerHandler::processRequest( if (!batch) { httpReplyError(403, "Forbidden"); - return false; + return; } json::Value r = jsonRPC; r[jss::error] = makeJsonError(kForbidden, "Forbidden"); - reply.append(r); + appendItemError(std::move(r)); continue; } @@ -805,11 +813,11 @@ ServerHandler::processRequest( if (!batch) { httpReplyError(400, "Null method"); - return false; + return; } json::Value r = jsonRPC; r[jss::error] = makeJsonError(kMethodNotFound, "Null method"); - reply.append(r); + appendItemError(std::move(r)); continue; } @@ -820,11 +828,11 @@ ServerHandler::processRequest( if (!batch) { httpReplyError(400, "method is not string"); - return false; + return; } json::Value r = jsonRPC; r[jss::error] = makeJsonError(kMethodNotFound, "method is not string"); - reply.append(r); + appendItemError(std::move(r)); continue; } @@ -835,11 +843,11 @@ ServerHandler::processRequest( if (!batch) { httpReplyError(400, "method is empty"); - return false; + return; } json::Value r = jsonRPC; r[jss::error] = makeJsonError(kMethodNotFound, "method is empty"); - reply.append(r); + appendItemError(std::move(r)); continue; } @@ -861,7 +869,7 @@ ServerHandler::processRequest( { usage.charge(Resource::kFeeMalformedRpc); httpReplyError(400, "params unparsable"); - return false; + return; } else { @@ -870,7 +878,7 @@ ServerHandler::processRequest( { usage.charge(Resource::kFeeMalformedRpc); httpReplyError(400, "params unparsable"); - return false; + return; } } } @@ -888,12 +896,12 @@ ServerHandler::processRequest( if (!batch) { httpReplyError(400, "ripplerpc is not a string"); - return false; + return; } json::Value r = jsonRPC; r[jss::error] = makeJsonError(kMethodNotFound, "ripplerpc is not a string"); - reply.append(r); + appendItemError(std::move(r)); continue; } ripplerpc = params[jss::ripplerpc].asString(); @@ -1094,7 +1102,6 @@ ServerHandler::processRequest( span.setOk(); } httpReply(httpStatus, response, output, rpcJ); - return !spanHadError; } //------------------------------------------------------------------------------ From 05f545ff5addf127dd895f45574903cea2e81c33 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Thu, 30 Jul 2026 12:05:18 +0100 Subject: [PATCH 4/7] text alignment Signed-off-by: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> --- docker/telemetry/grafana/dashboards/node-health.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docker/telemetry/grafana/dashboards/node-health.json b/docker/telemetry/grafana/dashboards/node-health.json index 327f222543..0350b6d4a1 100644 --- a/docker/telemetry/grafana/dashboards/node-health.json +++ b/docker/telemetry/grafana/dashboards/node-health.json @@ -2375,7 +2375,7 @@ "options": { "mergeValues": true, "showValue": "auto", - "alignValue": "left", + "alignValue": "center", "rowHeight": 0.9, "tooltip": { "mode": "single", From c66f9a391d9b9c7ce4ad3155964c74237be2b1c1 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Thu, 30 Jul 2026 18:05:56 +0100 Subject: [PATCH 5/7] fix(telemetry): state the real noun in dashboard rate units Grafana renders `unit: "ops"` as the literal string "ops/s", so every rate panel read as "operations per second" regardless of what it actually counted. `Ledger Build Rate` showed "0.258 ops/s" where the value is one ledger every 3.9s -- the number was right, the unit was meaningless. Replace the generic units with Grafana custom-suffix units naming the quantity, following the existing `suffix:/hr` and `si:drops` precedent in this repo. Nine of these are `stat` panels with no axis, so the unit string was the only text a reader ever saw. Also switch the two trusted/untrusted piecharts and the transaction path piechart from rate() to increase(): a per-slice "per second" reading is not a share of a total, counts in the window are. Queries are unchanged apart from those three; the values were already correct. Alongside, bring the touched panels up to the dashboard guidelines: tooltip mode/sort/max-height, 30-minute null spanning, and axis labels in title case. Hoist the stat panels above the fold on ledger-operations and rpc-performance. Panels that a later branch in this chain removes are deliberately left alone -- fixing them would only add merge conflicts. --- .../grafana/dashboards/consensus-health.json | 128 ++++++++++++------ .../grafana/dashboards/ledger-operations.json | 60 ++++---- .../grafana/dashboards/peer-network.json | 22 +-- .../grafana/dashboards/rpc-performance.json | 54 +++++--- .../dashboards/transaction-overview.json | 125 ++++++++++------- docs/telemetry-runbook.md | 14 +- 6 files changed, 246 insertions(+), 157 deletions(-) diff --git a/docker/telemetry/grafana/dashboards/consensus-health.json b/docker/telemetry/grafana/dashboards/consensus-health.json index 72eac8871b..3f03f9bbd7 100644 --- a/docker/telemetry/grafana/dashboards/consensus-health.json +++ b/docker/telemetry/grafana/dashboards/consensus-health.json @@ -21,7 +21,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -45,7 +46,7 @@ "unit": "ms", "custom": { "axisLabel": "Duration (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -67,7 +68,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -81,10 +83,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: proposals/s", "custom": { "axisLabel": "Proposals / Sec", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -106,7 +108,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -123,7 +126,7 @@ "unit": "ms", "custom": { "axisLabel": "Duration (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -159,7 +162,7 @@ ], "fieldConfig": { "defaults": { - "unit": "ops" + "unit": "suffix: validations/s" }, "overrides": [] } @@ -190,12 +193,19 @@ "legendFormat": "P50 Apply Duration [{{service_instance_id}}]" } ], + "options": { + "tooltip": { + "mode": "multi", + "sort": "desc", + "maxHeight": 600 + } + }, "fieldConfig": { "defaults": { "unit": "ms", "custom": { "axisLabel": "Duration (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -223,12 +233,19 @@ "legendFormat": "Close Time Correct={{close_time_correct}} [{{service_instance_id}}]" } ], + "options": { + "tooltip": { + "mode": "multi", + "sort": "desc", + "maxHeight": 600 + } + }, "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: rounds/s", "custom": { "axisLabel": "Rounds / Sec", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -250,7 +267,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -264,10 +282,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: closes/s", "custom": { - "axisLabel": "Events / Sec", - "spanNulls": true, + "axisLabel": "Closes / Sec", + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -289,7 +307,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -310,10 +329,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: events/s", "custom": { "axisLabel": "Events / Sec", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -335,7 +354,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -356,10 +376,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: events/s", "custom": { "axisLabel": "Events / Sec", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -381,7 +401,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 }, "yAxis": { "axisLabel": "Duration (ms)", @@ -400,7 +421,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ms" + "unit": "ms", + "custom": { + "spanNulls": 1800000 + } } } }, @@ -418,6 +442,7 @@ "defaults": { "unit": "dateTimeFromNow", "custom": { + "spanNulls": 1800000, "drawStyle": "points", "pointSize": 6, "showPoints": "always" @@ -428,7 +453,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 }, "legend": { "displayMode": "table", @@ -461,6 +487,7 @@ "defaults": { "unit": "dateTimeFromNow", "custom": { + "spanNulls": 1800000, "drawStyle": "points", "pointSize": 6, "showPoints": "always" @@ -471,7 +498,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 }, "legend": { "displayMode": "table", @@ -503,6 +531,7 @@ "fieldConfig": { "defaults": { "custom": { + "spanNulls": 1800000, "drawStyle": "line", "lineInterpolation": "stepAfter", "pointSize": 5, @@ -547,7 +576,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 }, "legend": { "displayMode": "table", @@ -587,6 +617,7 @@ "fieldConfig": { "defaults": { "custom": { + "spanNulls": 1800000, "drawStyle": "bars", "fillOpacity": 40, "pointSize": 5, @@ -598,7 +629,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 }, "legend": { "displayMode": "table", @@ -631,6 +663,7 @@ "defaults": { "unit": "short", "custom": { + "spanNulls": 1800000, "fillOpacity": 60 } }, @@ -639,7 +672,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 }, "legend": { "displayMode": "table", @@ -710,7 +744,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -731,10 +766,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: failures/s", "custom": { "axisLabel": "Failures / Sec", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -756,7 +791,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -780,7 +816,7 @@ "unit": "ms", "custom": { "axisLabel": "Duration (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -802,7 +838,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -826,7 +863,7 @@ "unit": "ms", "custom": { "axisLabel": "Duration (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -848,7 +885,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -872,7 +910,7 @@ "unit": "ms", "custom": { "axisLabel": "Duration (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -894,7 +932,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -915,10 +954,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: checks/s", "custom": { "axisLabel": "Checks / Sec", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -940,7 +979,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -954,10 +994,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: changes/s", "custom": { - "axisLabel": "Mode Changes / Sec", - "spanNulls": true, + "axisLabel": "Changes / Sec", + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 diff --git a/docker/telemetry/grafana/dashboards/ledger-operations.json b/docker/telemetry/grafana/dashboards/ledger-operations.json index 2cf30b1f16..37673e5b62 100644 --- a/docker/telemetry/grafana/dashboards/ledger-operations.json +++ b/docker/telemetry/grafana/dashboards/ledger-operations.json @@ -14,7 +14,7 @@ "type": "stat", "gridPos": { "h": 8, - "w": 12, + "w": 8, "x": 0, "y": 0 }, @@ -35,7 +35,7 @@ ], "fieldConfig": { "defaults": { - "unit": "ops" + "unit": "suffix: ledgers/s" }, "overrides": [] } @@ -47,13 +47,14 @@ "gridPos": { "h": 8, "w": 12, - "x": 12, - "y": 0 + "x": 0, + "y": 8 }, "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -77,7 +78,7 @@ "unit": "ms", "custom": { "axisLabel": "Duration (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -92,9 +93,9 @@ "type": "stat", "gridPos": { "h": 8, - "w": 12, - "x": 0, - "y": 8 + "w": 8, + "x": 8, + "y": 0 }, "options": { "tooltip": { @@ -113,7 +114,7 @@ ], "fieldConfig": { "defaults": { - "unit": "ops" + "unit": "suffix: ledgers/s" }, "overrides": [] } @@ -131,7 +132,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 }, "yAxis": { "axisLabel": "Duration (ms)" @@ -149,7 +151,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ms" + "unit": "ms", + "custom": { + "spanNulls": 1800000 + } }, "overrides": [] } @@ -167,7 +172,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -191,7 +197,7 @@ "unit": "ms", "custom": { "axisLabel": "Duration (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -213,7 +219,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -227,10 +234,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: transactions/s", "custom": { - "axisLabel": "Operations / Sec", - "spanNulls": true, + "axisLabel": "Transactions / Sec", + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -245,9 +252,9 @@ "type": "stat", "gridPos": { "h": 8, - "w": 12, - "x": 0, - "y": 24 + "w": 8, + "x": 16, + "y": 0 }, "options": { "tooltip": { @@ -266,7 +273,7 @@ ], "fieldConfig": { "defaults": { - "unit": "ops" + "unit": "suffix: ledgers/s" }, "overrides": [] } @@ -277,14 +284,15 @@ "type": "timeseries", "gridPos": { "h": 8, - "w": 12, - "x": 12, + "w": 24, + "x": 0, "y": 24 }, "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -308,7 +316,7 @@ "unit": "ms", "custom": { "axisLabel": "Duration (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 diff --git a/docker/telemetry/grafana/dashboards/peer-network.json b/docker/telemetry/grafana/dashboards/peer-network.json index 56f0d756d8..9d9de27f58 100644 --- a/docker/telemetry/grafana/dashboards/peer-network.json +++ b/docker/telemetry/grafana/dashboards/peer-network.json @@ -22,7 +22,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -36,10 +37,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: proposals/s", "custom": { "axisLabel": "Proposals / Sec", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -61,7 +62,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -75,10 +77,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: validations/s", "custom": { "axisLabel": "Validations / Sec", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -108,13 +110,13 @@ "datasource": { "type": "prometheus" }, - "expr": "sum by (proposal_trusted, service_instance_id) (rate(traces_span_metrics_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", proposal_trusted=~\"$proposal_trusted\", span_name=\"peer.proposal.receive\"}[5m]))", + "expr": "sum by (proposal_trusted, service_instance_id) (increase(traces_span_metrics_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", proposal_trusted=~\"$proposal_trusted\", span_name=\"peer.proposal.receive\"}[5m]))", "legendFormat": "Trusted = {{proposal_trusted}} [{{service_instance_id}}]" } ], "fieldConfig": { "defaults": { - "unit": "ops" + "unit": "short" }, "overrides": [] } @@ -140,13 +142,13 @@ "datasource": { "type": "prometheus" }, - "expr": "sum by (validation_trusted, service_instance_id) (rate(traces_span_metrics_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", validation_trusted=~\"$validation_trusted\", span_name=\"peer.validation.receive\"}[5m]))", + "expr": "sum by (validation_trusted, service_instance_id) (increase(traces_span_metrics_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", validation_trusted=~\"$validation_trusted\", span_name=\"peer.validation.receive\"}[5m]))", "legendFormat": "Trusted = {{validation_trusted}} [{{service_instance_id}}]" } ], "fieldConfig": { "defaults": { - "unit": "ops" + "unit": "short" }, "overrides": [] } diff --git a/docker/telemetry/grafana/dashboards/rpc-performance.json b/docker/telemetry/grafana/dashboards/rpc-performance.json index 9fe962e6a3..b54b98a174 100644 --- a/docker/telemetry/grafana/dashboards/rpc-performance.json +++ b/docker/telemetry/grafana/dashboards/rpc-performance.json @@ -16,12 +16,13 @@ "h": 8, "w": 12, "x": 0, - "y": 0 + "y": 24 }, "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -38,7 +39,7 @@ "unit": "reqps", "custom": { "axisLabel": "Requests / Sec", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -55,12 +56,13 @@ "h": 8, "w": 12, "x": 12, - "y": 0 + "y": 24 }, "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -77,7 +79,7 @@ "unit": "ms", "custom": { "axisLabel": "Latency (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -147,7 +149,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 }, "yAxis": { "axisLabel": "Duration (ms)", @@ -166,7 +169,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ms" + "unit": "ms", + "custom": { + "spanNulls": 1800000 + } } } }, @@ -183,7 +189,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -207,7 +214,7 @@ "unit": "reqps", "custom": { "axisLabel": "Requests / Sec", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -229,7 +236,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -250,10 +258,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: commands/s", "custom": { "axisLabel": "Commands / Sec", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -270,7 +278,7 @@ "h": 8, "w": 12, "x": 0, - "y": 24 + "y": 0 }, "options": { "tooltip": { @@ -302,7 +310,7 @@ "h": 8, "w": 12, "x": 12, - "y": 24 + "y": 0 }, "options": { "tooltip": { @@ -321,7 +329,7 @@ ], "fieldConfig": { "defaults": { - "unit": "ops" + "unit": "suffix: messages/s" }, "overrides": [] } @@ -339,7 +347,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 }, "legend": { "displayMode": "table", @@ -358,10 +367,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: requests/s", "custom": { "axisLabel": "Requests / Sec", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -383,7 +392,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -404,10 +414,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: requests/s", "custom": { "axisLabel": "Requests / Sec", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 diff --git a/docker/telemetry/grafana/dashboards/transaction-overview.json b/docker/telemetry/grafana/dashboards/transaction-overview.json index c5d54e9e4a..4efec5de79 100644 --- a/docker/telemetry/grafana/dashboards/transaction-overview.json +++ b/docker/telemetry/grafana/dashboards/transaction-overview.json @@ -21,7 +21,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -42,10 +43,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: transactions/s", "custom": { "axisLabel": "Transactions / Sec", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -67,7 +68,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 }, "legend": { "displayMode": "table", @@ -96,7 +98,7 @@ "unit": "ms", "custom": { "axisLabel": "Latency (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -126,10 +128,16 @@ "datasource": { "type": "prometheus" }, - "expr": "sum by (local, service_instance_id) (rate(traces_span_metrics_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", local=~\"$tx_origin\", span_name=\"tx.process\", tx_type=~\"$tx_type\"}[5m]))", + "expr": "sum by (local, service_instance_id) (increase(traces_span_metrics_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", local=~\"$tx_origin\", span_name=\"tx.process\", tx_type=~\"$tx_type\"}[5m]))", "legendFormat": "Local = {{local}} [{{service_instance_id}}]" } - ] + ], + "fieldConfig": { + "defaults": { + "unit": "short" + }, + "overrides": [] + } }, { "title": "Transaction Receive vs Suppressed", @@ -144,7 +152,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -158,10 +167,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: transactions/s", "custom": { "axisLabel": "Transactions / Sec", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -183,7 +192,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 }, "yAxis": { "axisLabel": "Duration (ms)", @@ -202,7 +212,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ms" + "unit": "ms", + "custom": { + "spanNulls": 1800000 + } } } }, @@ -219,7 +232,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -243,7 +257,7 @@ "unit": "ms", "custom": { "axisLabel": "Latency (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -265,7 +279,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -282,7 +297,7 @@ "unit": "ops", "custom": { "axisLabel": "Transactions / Sec", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -318,7 +333,7 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: transactions/s", "thresholds": { "steps": [ { @@ -352,7 +367,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 }, "legend": { "displayMode": "table", @@ -371,10 +387,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: transactions/s", "custom": { - "axisLabel": "TX / Sec", - "spanNulls": true, + "axisLabel": "Transactions / Sec", + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -396,7 +412,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 }, "legend": { "displayMode": "table", @@ -415,10 +432,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: transactions/s", "custom": { - "axisLabel": "Failed TX / Sec", - "spanNulls": true, + "axisLabel": "Transactions / Sec", + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -448,7 +465,9 @@ "showLegend": true }, "tooltip": { - "mode": "single" + "mode": "multi", + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -477,6 +496,7 @@ ] }, "custom": { + "spanNulls": 1800000, "fillOpacity": 80, "lineWidth": 0 } @@ -497,7 +517,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 }, "legend": { "displayMode": "table", @@ -519,7 +540,7 @@ "unit": "ms", "custom": { "axisLabel": "Duration (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -541,7 +562,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -555,10 +577,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: enqueues/s", "custom": { "axisLabel": "Enqueues / Sec", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -580,7 +602,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -597,7 +620,7 @@ "unit": "percentunit", "custom": { "axisLabel": "Bypass Fraction", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -619,7 +642,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -643,7 +667,7 @@ "unit": "ms", "custom": { "axisLabel": "Duration (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -665,7 +689,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 } }, "targets": [ @@ -679,10 +704,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: cleanups/s", "custom": { "axisLabel": "Cleanups / Sec", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -704,7 +729,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 }, "legend": { "displayMode": "table", @@ -726,7 +752,7 @@ "unit": "ops", "custom": { "axisLabel": "Spans / Sec", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -748,7 +774,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 }, "legend": { "displayMode": "table", @@ -770,7 +797,7 @@ "unit": "ms", "custom": { "axisLabel": "Duration (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -792,7 +819,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 }, "legend": { "displayMode": "table", @@ -811,10 +839,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: failures/s", "custom": { - "axisLabel": "Failed Spans / Sec", - "spanNulls": true, + "axisLabel": "Failures / Sec", + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -836,7 +864,8 @@ "options": { "tooltip": { "mode": "multi", - "sort": "desc" + "sort": "desc", + "maxHeight": 600 }, "legend": { "displayMode": "table", @@ -858,7 +887,7 @@ "unit": "ms", "custom": { "axisLabel": "Duration (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index 2a0ff66091..ef4d3f2dec 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -667,7 +667,7 @@ Ten dashboards are pre-provisioned in `docker/telemetry/grafana/dashboards/`: | ---------------------------------- | -------------- | --------------------------------------------------------------------------------------------------- | ----------------------------------- | | Transaction Processing Rate | timeseries | `rate(traces_span_metrics_calls_total{span_name="tx.process"}[5m])` and `tx.receive` | `span_name` | | Transaction Processing Latency | timeseries | `histogram_quantile(0.95 / 0.50, ... {span_name="tx.process"})` | — | -| Transaction Path Distribution | piechart | `sum by (local) (rate(traces_span_metrics_calls_total{span_name="tx.process"}[5m]))` | `local` | +| Transaction Path Distribution | piechart | `sum by (local) (increase(traces_span_metrics_calls_total{span_name="tx.process"}[5m]))` | `local` | | Transaction Receive vs Suppressed | timeseries | `rate(traces_span_metrics_calls_total{span_name="tx.receive"}[5m])` | — | | TX Processing Duration Heatmap | heatmap | `tx.process` histogram buckets | `le` | | TX Apply Duration per Ledger | timeseries | p95/p50 of `tx.apply` | — | @@ -707,12 +707,12 @@ Ten dashboards are pre-provisioned in `docker/telemetry/grafana/dashboards/`: Requires `trace_peer=1` in the `[telemetry]` config section. -| Panel | Type | PromQL | Labels Used | -| -------------------------------- | ---------- | ------------------------------ | -------------------- | -| Proposal Receive Rate | timeseries | `peer.proposal.receive` rate | — | -| Validation Receive Rate | timeseries | `peer.validation.receive` rate | — | -| Proposals Trusted vs Untrusted | piechart | by `proposal_trusted` | `proposal_trusted` | -| Validations Trusted vs Untrusted | piechart | by `validation_trusted` | `validation_trusted` | +| Panel | Type | PromQL | Labels Used | +| -------------------------------- | ---------- | ------------------------------------------------------------------------- | -------------------- | +| Proposal Receive Rate | timeseries | `peer.proposal.receive` rate | — | +| Validation Receive Rate | timeseries | `peer.validation.receive` rate | — | +| Proposals Trusted vs Untrusted | piechart | `increase()` counts in the selected window, split by `proposal_trusted` | `proposal_trusted` | +| Validations Trusted vs Untrusted | piechart | `increase()` counts in the selected window, split by `validation_trusted` | `validation_trusted` | ### Node Health -- StatsD (`xrpld-statsd-node-health`) From 2095a3d2f17238cad2ee07bc7425b5c4fdf4a0f6 Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Thu, 30 Jul 2026 18:06:23 +0100 Subject: [PATCH 6/7] fix(telemetry): correct job latency units and name dashboard rate nouns The four job-latency panels on node-health declared milliseconds while querying `job_running_us` / `job_queued_us`, which record microseconds (MetricsRegistry records the raw value, and the instrument description says microseconds). Every reading was therefore a thousand times too large: the p95 for acceptLedger, 241495us, rendered as "241 sec" instead of 241ms. job-queue.json already read these same metrics as microseconds, so the two dashboards disagreed by 1000x on identical data. Switch node-health to microseconds to match. Also replace the generic `ops` and `cps` units, which Grafana renders as the literal "ops/s" and "counts/s", with custom-suffix units naming what each panel counts -- messages, fetches, calls, mismatches. Two panels plot more than one quantity on a single axis, which no single unit can describe. Give each series its own unit through field overrides: reads per second beside two queue depths on NuDB Read Pressure, and ledgers beside fetches on Ledger Close Rate, the latter on a right-hand axis. State Duration Rate plots a seconds-per-second time share, which can exceed 1.0 and so is not a percentage; label it as the ratio it is. The normalised share already exists as its own panel. Queries are unchanged; the values were already correct. Alongside, bring the touched panels up to the dashboard guidelines and hoist the stat panels above the fold. --- .../grafana/dashboards/ledger-data-sync.json | 85 +++++++++++++++---- .../grafana/dashboards/network-traffic.json | 20 ++--- .../grafana/dashboards/node-health.json | 60 +++++++------ .../dashboards/overlay-traffic-detail.json | 12 +-- .../grafana/dashboards/rpc-pathfinding.json | 48 ++++++----- docs/telemetry-runbook.md | 40 ++++----- 6 files changed, 168 insertions(+), 97 deletions(-) diff --git a/docker/telemetry/grafana/dashboards/ledger-data-sync.json b/docker/telemetry/grafana/dashboards/ledger-data-sync.json index 42916eb34a..2f01f834d9 100644 --- a/docker/telemetry/grafana/dashboards/ledger-data-sync.json +++ b/docker/telemetry/grafana/dashboards/ledger-data-sync.json @@ -496,9 +496,9 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "cps", + "unit": "suffix: messages/s", "custom": { - "axisLabel": "Messages In", + "axisLabel": "Messages / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -543,9 +543,9 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "cps", + "unit": "suffix: messages/s", "custom": { - "axisLabel": "Messages In", + "axisLabel": "Messages / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -584,9 +584,9 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "cps", + "unit": "suffix: messages/s", "custom": { - "axisLabel": "Messages In", + "axisLabel": "Messages / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -637,9 +637,9 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "cps", + "unit": "suffix: messages/s", "custom": { - "axisLabel": "Messages In", + "axisLabel": "Messages / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -985,8 +985,9 @@ "alignValue": "left", "rowHeight": 0.9, "tooltip": { - "mode": "single", - "sort": "none" + "maxHeight": 600, + "mode": "multi", + "sort": "desc" }, "legend": { "displayMode": "list", @@ -1012,7 +1013,7 @@ "fillOpacity": 80, "lineWidth": 0, "insertNulls": false, - "spanNulls": false + "spanNulls": 1800000 }, "mappings": [ { @@ -1117,16 +1118,33 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "unit": "reqps", + "unit": "suffix: ledgers/s", "custom": { - "axisLabel": "Rate (per second)", + "axisLabel": "Ledgers / Sec & Fetches / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 } }, - "overrides": [] + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/^Ledger Fetches/" + }, + "properties": [ + { + "id": "unit", + "value": "suffix: fetches/s" + }, + { + "id": "custom.axisPlacement", + "value": "right" + } + ] + } + ] }, "id": 19 }, @@ -1376,16 +1394,49 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "unit": "short", + "unit": "suffix: reads/s", "custom": { - "axisLabel": "Reads/s & Depth", + "axisLabel": "Reads / Sec & Depth", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 } }, - "overrides": [] + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/^Read Queue/" + }, + "properties": [ + { + "id": "unit", + "value": "short" + }, + { + "id": "custom.axisPlacement", + "value": "right" + } + ] + }, + { + "matcher": { + "id": "byRegexp", + "options": "/^Read Threads Running/" + }, + "properties": [ + { + "id": "unit", + "value": "short" + }, + { + "id": "custom.axisPlacement", + "value": "right" + } + ] + } + ] }, "id": 24 }, diff --git a/docker/telemetry/grafana/dashboards/network-traffic.json b/docker/telemetry/grafana/dashboards/network-traffic.json index 7323a32bef..724f05f652 100644 --- a/docker/telemetry/grafana/dashboards/network-traffic.json +++ b/docker/telemetry/grafana/dashboards/network-traffic.json @@ -63,7 +63,7 @@ "h": 8, "w": 12, "x": 12, - "y": 0 + "y": 24 }, "options": { "tooltip": { @@ -175,9 +175,9 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "cps", + "unit": "suffix: messages/s", "custom": { - "axisLabel": "Messages", + "axisLabel": "Messages / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -227,9 +227,9 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "cps", + "unit": "suffix: messages/s", "custom": { - "axisLabel": "Messages", + "axisLabel": "Messages / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -285,9 +285,9 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "cps", + "unit": "suffix: messages/s", "custom": { - "axisLabel": "Messages", + "axisLabel": "Messages / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -343,9 +343,9 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "cps", + "unit": "suffix: messages/s", "custom": { - "axisLabel": "Messages", + "axisLabel": "Messages / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -363,7 +363,7 @@ "h": 8, "w": 12, "x": 12, - "y": 24 + "y": 0 }, "options": { "tooltip": { diff --git a/docker/telemetry/grafana/dashboards/node-health.json b/docker/telemetry/grafana/dashboards/node-health.json index a21bb2d574..44ea4cc5cb 100644 --- a/docker/telemetry/grafana/dashboards/node-health.json +++ b/docker/telemetry/grafana/dashboards/node-health.json @@ -113,10 +113,11 @@ "h": 8, "w": 12, "x": 0, - "y": 8 + "y": 24 }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -163,7 +164,7 @@ "unit": "percentunit", "custom": { "axisLabel": "Time in mode", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3, @@ -183,10 +184,11 @@ "h": 8, "w": 12, "x": 12, - "y": 8 + "y": 24 }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -233,7 +235,7 @@ "unit": "short", "custom": { "axisLabel": "Transitions", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -254,6 +256,7 @@ }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -279,7 +282,7 @@ "unit": "ms", "custom": { "axisLabel": "Latency (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -300,6 +303,7 @@ }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -318,7 +322,7 @@ "unit": "short", "custom": { "axisLabel": "Jobs", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -335,7 +339,7 @@ "h": 8, "w": 12, "x": 0, - "y": 24 + "y": 8 }, "options": { "tooltip": { @@ -354,7 +358,7 @@ ], "fieldConfig": { "defaults": { - "unit": "ops" + "unit": "suffix: fetches/s" }, "overrides": [] } @@ -367,7 +371,7 @@ "h": 8, "w": 12, "x": 12, - "y": 24 + "y": 8 }, "options": { "tooltip": { @@ -386,7 +390,7 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: mismatches/s", "thresholds": { "steps": [ { @@ -427,6 +431,7 @@ }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -512,10 +517,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ms", + "unit": "µs", "custom": { - "axisLabel": "Duration (ms)", - "spanNulls": true, + "axisLabel": "Duration (µs)", + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -536,6 +541,7 @@ }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -621,10 +627,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ms", + "unit": "µs", "custom": { - "axisLabel": "Wait Time (ms)", - "spanNulls": true, + "axisLabel": "Wait Time (µs)", + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -645,6 +651,7 @@ }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -663,7 +670,7 @@ "unit": "short", "custom": { "axisLabel": "Entries", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -782,6 +789,7 @@ }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -804,10 +812,10 @@ ], "fieldConfig": { "defaults": { - "unit": "short", + "unit": "suffix: s/s", "custom": { "axisLabel": "Rate (s/s)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -828,6 +836,7 @@ }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -843,10 +852,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ms", + "unit": "µs", "custom": { - "axisLabel": "Duration (ms)", - "spanNulls": true, + "axisLabel": "Duration (µs)", + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -867,6 +876,7 @@ }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -882,10 +892,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ms", + "unit": "µs", "custom": { - "axisLabel": "Wait Time (ms)", - "spanNulls": true, + "axisLabel": "Wait Time (µs)", + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 diff --git a/docker/telemetry/grafana/dashboards/overlay-traffic-detail.json b/docker/telemetry/grafana/dashboards/overlay-traffic-detail.json index f86a75b68c..2b1ffc8ea1 100644 --- a/docker/telemetry/grafana/dashboards/overlay-traffic-detail.json +++ b/docker/telemetry/grafana/dashboards/overlay-traffic-detail.json @@ -67,9 +67,9 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "cps", + "unit": "suffix: messages/s", "custom": { - "axisLabel": "Messages", + "axisLabel": "Messages / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -195,9 +195,9 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "cps", + "unit": "suffix: messages/s", "custom": { - "axisLabel": "Count", + "axisLabel": "Messages / Sec & Bytes / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -328,9 +328,9 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "cps", + "unit": "suffix: messages/s", "custom": { - "axisLabel": "Messages", + "axisLabel": "Messages / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", diff --git a/docker/telemetry/grafana/dashboards/rpc-pathfinding.json b/docker/telemetry/grafana/dashboards/rpc-pathfinding.json index 28f9a51ad5..bae50fa772 100644 --- a/docker/telemetry/grafana/dashboards/rpc-pathfinding.json +++ b/docker/telemetry/grafana/dashboards/rpc-pathfinding.json @@ -54,6 +54,7 @@ }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -81,7 +82,7 @@ "unit": "ms", "custom": { "axisLabel": "Latency (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -98,10 +99,11 @@ "h": 8, "w": 12, "x": 0, - "y": 8 + "y": 24 }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -129,7 +131,7 @@ "unit": "decbytes", "custom": { "axisLabel": "Size (Bytes)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -146,10 +148,11 @@ "h": 8, "w": 12, "x": 12, - "y": 8 + "y": 24 }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -193,7 +196,7 @@ "unit": "ms", "custom": { "axisLabel": "Latency (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -214,6 +217,7 @@ }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -241,7 +245,7 @@ "unit": "ms", "custom": { "axisLabel": "Duration (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -262,6 +266,7 @@ }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -289,7 +294,7 @@ "unit": "ms", "custom": { "axisLabel": "Duration (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -306,7 +311,7 @@ "h": 8, "w": 12, "x": 0, - "y": 24 + "y": 8 }, "options": { "tooltip": { @@ -326,7 +331,7 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: warnings/s", "thresholds": { "steps": [ { @@ -355,7 +360,7 @@ "h": 8, "w": 12, "x": 12, - "y": 24 + "y": 8 }, "options": { "tooltip": { @@ -375,7 +380,7 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: drops/s", "thresholds": { "steps": [ { @@ -408,6 +413,7 @@ }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -424,10 +430,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: calls/s", "custom": { "axisLabel": "Calls / Sec", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -448,6 +454,7 @@ }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -467,7 +474,7 @@ "unit": "ms", "custom": { "axisLabel": "Duration (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -488,6 +495,7 @@ }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -504,10 +512,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: calls/s", "custom": { "axisLabel": "Calls / Sec", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -528,6 +536,7 @@ }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -555,7 +564,7 @@ "unit": "ms", "custom": { "axisLabel": "Duration (ms)", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 @@ -576,6 +585,7 @@ }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -600,10 +610,10 @@ ], "fieldConfig": { "defaults": { - "unit": "ops", + "unit": "suffix: operations/s", "custom": { "axisLabel": "Operations / Sec", - "spanNulls": true, + "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index 57a5c7ecba..22aded49fc 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -718,24 +718,24 @@ Requires `trace_peer=1` in the `[telemetry]` config section. ### Node Health -- System Metrics (`node-health`) -| Panel | Type | PromQL | Labels Used | -| -------------------------------------- | ---------- | ---------------------------------------------------------- | ----------- | -| Validated Ledger Age | stat | `ledgermaster_validated_ledger_age` | — | -| Published Ledger Age | stat | `ledgermaster_published_ledger_age` | — | -| Operating Mode (Time Share) | timeseries | `rate(state_accounting_X_duration) / sum(rate(all modes))` | — | -| Operating Mode Transitions | timeseries | `state_accounting_*_transitions` | — | -| I/O Latency | timeseries | `histogram_quantile(0.95, ios_latency_bucket)` | — | -| Job Queue Depth | timeseries | `job_count` | — | -| Ledger Fetch Rate | stat | `rate(ledger_fetches[5m])` | — | -| Ledger History Mismatches | stat | `rate(ledger_history_mismatch[5m])` | — | -| Key Jobs Execution Time | timeseries | `acceptledger{quantile="$quantile"}` (+ 10 more key jobs) | `quantile` | -| Key Jobs Dequeue Wait Time | timeseries | `acceptledger_q{quantile="$quantile"}` (+ 10 more) | `quantile` | -| FullBelowCache Size | timeseries | `node_family_full_below_cache_size` | — | -| FullBelowCache Hit Rate | gauge | `node_family_full_below_cache_hit_rate` | — | -| Ledger Publish Gap | stat | `Published_Ledger_Age - Validated_Ledger_Age` | — | -| State Duration Rate (Full vs Tracking) | timeseries | `rate(state_accounting_full_duration[5m]) / 1000000` | — | -| All Jobs Execution Time (Detail) | timeseries | `{__name__=~"", quantile="$quantile"}` | `quantile` | -| All Jobs Dequeue Wait (Detail) | timeseries | `{__name__=~"_q", quantile="$quantile"}` | `quantile` | +| Panel | Type | PromQL | Labels Used | +| -------------------------------------- | ---------- | --------------------------------------------------------------------------------- | ----------- | +| Validated Ledger Age | stat | `ledgermaster_validated_ledger_age` | — | +| Published Ledger Age | stat | `ledgermaster_published_ledger_age` | — | +| Operating Mode (Time Share) | timeseries | `rate(state_accounting_X_duration) / sum(rate(all modes))` | — | +| Operating Mode Transitions | timeseries | `state_accounting_*_transitions` | — | +| I/O Latency | timeseries | `histogram_quantile(0.95, ios_latency_bucket)` | — | +| Job Queue Depth | timeseries | `job_count` | — | +| Ledger Fetch Rate | stat | `rate(ledger_fetches_total[$__rate_interval])` | — | +| Ledger History Mismatches | stat | `rate(ledger_history_mismatch_total[$__rate_interval])` | — | +| Key Jobs Execution Time | timeseries | `acceptledger{quantile="$quantile"}` (+ 10 more key jobs) | `quantile` | +| Key Jobs Dequeue Wait Time | timeseries | `acceptledger_q{quantile="$quantile"}` (+ 10 more) | `quantile` | +| FullBelowCache Size | timeseries | `node_family_full_below_cache_size` | — | +| FullBelowCache Hit Rate | gauge | `node_family_full_below_cache_hit_rate` | — | +| Ledger Publish Gap | stat | `Published_Ledger_Age - Validated_Ledger_Age` | — | +| State Duration Rate (Full vs Tracking) | timeseries | `rate(state_accounting_full_duration[5m]) / 1000000` | — | +| All Jobs Execution Time (Detail) | timeseries | `histogram_quantile($quantile, rate(job_running_us_bucket[5m])) by job_type` — µs | `quantile` | +| All Jobs Dequeue Wait (Detail) | timeseries | `histogram_quantile($quantile, rate(job_queued_us_bucket[5m])) by job_type` — µs | `quantile` | ### Network Traffic -- System Metrics (`network-traffic`) @@ -762,8 +762,8 @@ Requires `trace_peer=1` in the `[telemetry]` config section. | RPC Response Time Heatmap | heatmap | `rpc_time_bucket` | — | | Pathfinding Fast Duration | timeseries | `histogram_quantile(0.95, pathfind_fast_bucket)` | — | | Pathfinding Full Duration | timeseries | `histogram_quantile(0.95, pathfind_full_bucket)` | — | -| Resource Warnings Rate | stat | `rate(warn[5m])` | — | -| Resource Drops Rate | stat | `rate(drop[5m])` | — | +| Resource Warnings Rate | stat | `rate(warn_total[$__rate_interval])` | — | +| Resource Drops Rate | stat | `rate(drop_total[$__rate_interval])` | — | ### Span → Metric → Dashboard Summary From 370130a9f9adc2d842f76bdb6c57e54a4199974a Mon Sep 17 00:00:00 2001 From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com> Date: Thu, 30 Jul 2026 18:06:56 +0100 Subject: [PATCH 7/7] fix(telemetry): correct job latency units and name dashboard rate nouns The two key-job latency panels on node-health declared milliseconds while querying `job_running_us` / `job_queued_us`, which record microseconds. Every reading was a thousand times too large: the p95 for acceptLedger, 241495us, rendered as "241 sec" instead of 241ms. job-queue.json already read the same metrics as microseconds, so the two dashboards disagreed by 1000x on identical data. Also replace the generic `ops`, `cps` and `cpm` units, which Grafana renders as the literal "ops/s", "counts/s" and "counts/min", with custom-suffix units naming what each panel counts -- jobs, ledgers, validations, lookups. The per-minute panels already scale correctly in their queries; only the noun was missing. Normalise the micro sign to U+00B5 throughout. Four axis labels used the visually identical Greek mu, U+03BC, while every unit field used the micro sign. Ledger Close Rate plots ledgers closed beside ledger fetches, which one unit cannot describe; give the fetches series its own unit on a right-hand axis. Drop two field overrides on NodeStore Read Latency that restated the panel unit as a custom suffix -- a suffix is appended verbatim, so it would have suppressed magnitude scaling and left large values unreadable, the same defect fixed above. Queries are unchanged apart from the transaction path piechart, which moves from rate() to increase(): a per-slice "per second" reading is not a share of a total. Alongside, widen the Complete Ledger Ranges table to full width, hoist the stat panels above the fold, and bring the touched panels up to the tooltip and null-spanning guidelines. --- .../grafana/dashboards/consensus-health.json | 5 +- .../grafana/dashboards/fee-market.json | 8 +- .../grafana/dashboards/job-queue.json | 20 ++--- .../grafana/dashboards/ledger-data-sync.json | 39 ++++++--- .../grafana/dashboards/node-health.json | 79 ++++++++++--------- .../dashboards/overlay-traffic-detail.json | 12 ++- .../grafana/dashboards/rpc-performance.json | 12 +-- .../dashboards/transaction-overview.json | 4 +- .../grafana/dashboards/validator-health.json | 22 +++--- docs/telemetry-runbook.md | 52 ++++++------ 10 files changed, 140 insertions(+), 113 deletions(-) diff --git a/docker/telemetry/grafana/dashboards/consensus-health.json b/docker/telemetry/grafana/dashboards/consensus-health.json index 8ae07fe919..ab848401b1 100644 --- a/docker/telemetry/grafana/dashboards/consensus-health.json +++ b/docker/telemetry/grafana/dashboards/consensus-health.json @@ -230,6 +230,7 @@ }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -283,6 +284,7 @@ }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -376,6 +378,7 @@ }, "options": { "tooltip": { + "maxHeight": 600, "mode": "multi", "sort": "desc" } @@ -1006,7 +1009,7 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "ops", + "unit": "suffix: mismatches/s", "custom": { "axisLabel": "Mismatches / Sec", "spanNulls": 1800000, diff --git a/docker/telemetry/grafana/dashboards/fee-market.json b/docker/telemetry/grafana/dashboards/fee-market.json index 7ab2fde491..b8ff971de7 100644 --- a/docker/telemetry/grafana/dashboards/fee-market.json +++ b/docker/telemetry/grafana/dashboards/fee-market.json @@ -332,9 +332,9 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "ops", + "unit": "suffix: transactions/s", "custom": { - "axisLabel": "Expired / Sec", + "axisLabel": "Transactions / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -372,9 +372,9 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "ops", + "unit": "suffix: transactions/s", "custom": { - "axisLabel": "Dropped / Sec", + "axisLabel": "Transactions / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", diff --git a/docker/telemetry/grafana/dashboards/job-queue.json b/docker/telemetry/grafana/dashboards/job-queue.json index fa84c49770..3c7d229316 100644 --- a/docker/telemetry/grafana/dashboards/job-queue.json +++ b/docker/telemetry/grafana/dashboards/job-queue.json @@ -110,12 +110,12 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "ops", + "unit": "suffix: jobs/s", "custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 10, - "axisLabel": "Operations / Sec", + "axisLabel": "Jobs / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -162,12 +162,12 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "ops", + "unit": "suffix: jobs/s", "custom": { "drawStyle": "line", "lineWidth": 1, "fillOpacity": 5, - "axisLabel": "Operations / Sec", + "axisLabel": "Jobs / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -214,12 +214,12 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "ops", + "unit": "suffix: jobs/s", "custom": { "drawStyle": "line", "lineWidth": 1, "fillOpacity": 5, - "axisLabel": "Operations / Sec", + "axisLabel": "Jobs / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -272,7 +272,7 @@ "drawStyle": "line", "lineWidth": 2, "fillOpacity": 5, - "axisLabel": "Duration (μs)", + "axisLabel": "Duration (µs)", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -325,7 +325,7 @@ "drawStyle": "line", "lineWidth": 2, "fillOpacity": 5, - "axisLabel": "Duration (μs)", + "axisLabel": "Duration (µs)", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -377,7 +377,7 @@ "drawStyle": "line", "lineWidth": 1, "fillOpacity": 5, - "axisLabel": "Duration (μs)", + "axisLabel": "Duration (µs)", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -419,7 +419,7 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "cpm", + "unit": "suffix: overflows/min", "thresholds": { "steps": [ { diff --git a/docker/telemetry/grafana/dashboards/ledger-data-sync.json b/docker/telemetry/grafana/dashboards/ledger-data-sync.json index 51f4ec9f40..f598ba5808 100644 --- a/docker/telemetry/grafana/dashboards/ledger-data-sync.json +++ b/docker/telemetry/grafana/dashboards/ledger-data-sync.json @@ -1117,16 +1117,33 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "unit": "reqps", + "unit": "suffix: ledgers/s", "custom": { - "axisLabel": "Rate (per second)", + "axisLabel": "Ledgers / Sec & Fetches / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", "pointSize": 3 } }, - "overrides": [] + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "/^Ledger Fetches/" + }, + "properties": [ + { + "id": "unit", + "value": "suffix: fetches/s" + }, + { + "id": "custom.axisPlacement", + "value": "right" + } + ] + } + ] }, "id": 19 }, @@ -1826,9 +1843,9 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "unit": "ops", + "unit": "suffix: events/s", "custom": { - "axisLabel": "Events (Per Second)", + "axisLabel": "Events / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -1882,9 +1899,9 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "unit": "ops", + "unit": "suffix: outcomes/s", "custom": { - "axisLabel": "Outcomes (Per Second)", + "axisLabel": "Outcomes / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -1931,9 +1948,9 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "unit": "ops", + "unit": "suffix: acquisitions/s", "custom": { - "axisLabel": "Discarded Acquisitions (Per Second)", + "axisLabel": "Acquisitions / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -1980,9 +1997,9 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} ${__field.labels.xrpl_ident}", - "unit": "ops", + "unit": "suffix: events/s", "custom": { - "axisLabel": "Ledger Lane Events (Per Second)", + "axisLabel": "Events / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", diff --git a/docker/telemetry/grafana/dashboards/node-health.json b/docker/telemetry/grafana/dashboards/node-health.json index 0350b6d4a1..a0bb801fe0 100644 --- a/docker/telemetry/grafana/dashboards/node-health.json +++ b/docker/telemetry/grafana/dashboards/node-health.json @@ -526,7 +526,7 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "ops", + "unit": "suffix: operations/s", "custom": { "axisLabel": "Operations / Sec", "drawStyle": "line", @@ -1196,7 +1196,7 @@ "type": "timeseries", "gridPos": { "x": 0, - "y": 88, + "y": 96, "w": 12, "h": 8 }, @@ -1240,7 +1240,7 @@ "type": "timeseries", "gridPos": { "x": 12, - "y": 88, + "y": 96, "w": 12, "h": 8 }, @@ -1285,8 +1285,8 @@ "gridPos": { "h": 8, "w": 12, - "x": 12, - "y": 96 + "x": 0, + "y": 88 }, "options": { "tooltip": { @@ -1319,8 +1319,8 @@ "gridPos": { "h": 8, "w": 12, - "x": 0, - "y": 104 + "x": 12, + "y": 88 }, "options": { "tooltip": { @@ -1353,7 +1353,7 @@ "gridPos": { "h": 8, "w": 12, - "x": 12, + "x": 0, "y": 104 }, "options": { @@ -1407,9 +1407,9 @@ "type": "table", "gridPos": { "h": 8, - "w": 12, + "w": 24, "x": 0, - "y": 113 + "y": 121 }, "options": { "showHeader": true @@ -1440,8 +1440,8 @@ "gridPos": { "h": 8, "w": 12, - "x": 12, - "y": 113 + "x": 0, + "y": 129 }, "options": { "tooltip": { @@ -1494,9 +1494,9 @@ "type": "stat", "gridPos": { "h": 8, - "w": 12, + "w": 24, "x": 0, - "y": 121 + "y": 113 }, "options": { "tooltip": { @@ -1516,7 +1516,7 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "cpm", + "unit": "suffix: fetches/min", "custom": {} }, "overrides": [] @@ -1530,7 +1530,7 @@ "h": 8, "w": 12, "x": 12, - "y": 121 + "y": 129 }, "options": { "tooltip": { @@ -1571,7 +1571,7 @@ "h": 1, "w": 24, "x": 0, - "y": 129 + "y": 137 }, "collapsed": false, "panels": [] @@ -1584,7 +1584,7 @@ "h": 8, "w": 6, "x": 0, - "y": 130 + "y": 138 }, "options": { "tooltip": { @@ -1618,7 +1618,7 @@ "h": 8, "w": 6, "x": 6, - "y": 130 + "y": 138 }, "options": { "tooltip": { @@ -1652,7 +1652,7 @@ "h": 8, "w": 6, "x": 12, - "y": 130 + "y": 138 }, "options": { "tooltip": { @@ -1686,7 +1686,7 @@ "h": 8, "w": 6, "x": 18, - "y": 130 + "y": 138 }, "options": { "tooltip": { @@ -1748,7 +1748,7 @@ "h": 8, "w": 24, "x": 0, - "y": 138 + "y": 146 }, "options": { "tooltip": { @@ -1768,9 +1768,9 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "cps", + "unit": "suffix: transactions/s", "custom": { - "axisLabel": "Transactions", + "axisLabel": "Transactions / Sec", "drawStyle": "line", "lineWidth": 2, "fillOpacity": 10, @@ -1886,9 +1886,9 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "ms", + "unit": "µs", "custom": { - "axisLabel": "Duration (ms)", + "axisLabel": "Duration (µs)", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -1986,9 +1986,9 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "ms", + "unit": "µs", "custom": { - "axisLabel": "Wait Time (ms)", + "axisLabel": "Wait Time (µs)", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -2290,7 +2290,7 @@ "h": 8, "w": 12, "x": 0, - "y": 130 + "y": 138 }, "options": { "tooltip": { @@ -2330,7 +2330,7 @@ "h": 8, "w": 12, "x": 12, - "y": 130 + "y": 138 }, "options": { "tooltip": { @@ -2350,7 +2350,7 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "ops", + "unit": "suffix: acquisitions/s", "custom": { "axisLabel": "Acquisitions / Sec", "spanNulls": 1800000, @@ -2370,7 +2370,7 @@ "h": 8, "w": 18, "x": 0, - "y": 138 + "y": 146 }, "options": { "mergeValues": true, @@ -2378,8 +2378,9 @@ "alignValue": "center", "rowHeight": 0.9, "tooltip": { - "mode": "single", - "sort": "none" + "maxHeight": 600, + "mode": "multi", + "sort": "desc" } }, "targets": [ @@ -2444,7 +2445,9 @@ ], "custom": { "fillOpacity": 80, - "lineWidth": 0 + "lineWidth": 0, + "spanNulls": 1800000, + "insertNulls": false } }, "overrides": [] @@ -2458,7 +2461,7 @@ "h": 8, "w": 6, "x": 18, - "y": 138 + "y": 146 }, "options": { "reduceOptions": { @@ -2519,7 +2522,7 @@ "h": 1, "w": 24, "x": 0, - "y": 146 + "y": 154 }, "collapsed": false, "panels": [] @@ -2532,7 +2535,7 @@ "h": 8, "w": 24, "x": 0, - "y": 147 + "y": 155 }, "options": { "tooltip": { diff --git a/docker/telemetry/grafana/dashboards/overlay-traffic-detail.json b/docker/telemetry/grafana/dashboards/overlay-traffic-detail.json index f7bf0823c6..adc685ce6f 100644 --- a/docker/telemetry/grafana/dashboards/overlay-traffic-detail.json +++ b/docker/telemetry/grafana/dashboards/overlay-traffic-detail.json @@ -590,7 +590,7 @@ "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", "unit": "µs", "custom": { - "axisLabel": "Duration (μs)", + "axisLabel": "Duration (µs)", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -639,7 +639,11 @@ ], "fieldConfig": { "defaults": { - "unit": "short" + "unit": "short", + "custom": { + "spanNulls": 1800000, + "insertNulls": false + } }, "overrides": [] } @@ -672,7 +676,7 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "cps", + "unit": "suffix: lookups/s", "custom": { "axisLabel": "Lookups / Sec", "spanNulls": 1800000, @@ -712,7 +716,7 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "cps", + "unit": "suffix: rejections/s", "custom": { "axisLabel": "Rejections / Sec", "spanNulls": 1800000, diff --git a/docker/telemetry/grafana/dashboards/rpc-performance.json b/docker/telemetry/grafana/dashboards/rpc-performance.json index 5d436d2427..230dcbe0fb 100644 --- a/docker/telemetry/grafana/dashboards/rpc-performance.json +++ b/docker/telemetry/grafana/dashboards/rpc-performance.json @@ -498,12 +498,12 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "ops", + "unit": "suffix: calls/s", "custom": { "drawStyle": "line", "lineWidth": 2, "fillOpacity": 10, - "axisLabel": "Operations / Sec", + "axisLabel": "Calls / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -550,12 +550,12 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "ops", + "unit": "suffix: calls/s", "custom": { "drawStyle": "line", "lineWidth": 1, "fillOpacity": 5, - "axisLabel": "Operations / Sec", + "axisLabel": "Calls / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", @@ -602,12 +602,12 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "ops", + "unit": "suffix: errors/s", "custom": { "drawStyle": "line", "lineWidth": 1, "fillOpacity": 5, - "axisLabel": "Operations / Sec", + "axisLabel": "Errors / Sec", "spanNulls": 1800000, "insertNulls": false, "showPoints": "auto", diff --git a/docker/telemetry/grafana/dashboards/transaction-overview.json b/docker/telemetry/grafana/dashboards/transaction-overview.json index 2a4fc54292..c14b31013d 100644 --- a/docker/telemetry/grafana/dashboards/transaction-overview.json +++ b/docker/telemetry/grafana/dashboards/transaction-overview.json @@ -297,7 +297,7 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "cps" + "unit": "short" }, "overrides": [] }, @@ -313,7 +313,7 @@ "datasource": { "type": "prometheus" }, - "expr": "label_replace(sum by (local, service_instance_id, xrpl_branch, xrpl_node_role) (rate(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", local=~\"$tx_origin\", span_name=\"tx.process\", tx_type=~\"$tx_type\"}[$__rate_interval])), \"series\", \"Local $1\", \"local\", \"(.*)\")" + "expr": "label_replace(sum by (local, service_instance_id, xrpl_branch, xrpl_node_role) (increase(span_calls_total{service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", local=~\"$tx_origin\", span_name=\"tx.process\", tx_type=~\"$tx_type\"}[$__rate_interval])), \"series\", \"Local $1\", \"local\", \"(.*)\")" } ], "id": 7 diff --git a/docker/telemetry/grafana/dashboards/validator-health.json b/docker/telemetry/grafana/dashboards/validator-health.json index 7c53c2d593..e1f1abf1f4 100644 --- a/docker/telemetry/grafana/dashboards/validator-health.json +++ b/docker/telemetry/grafana/dashboards/validator-health.json @@ -297,7 +297,7 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "cpm", + "unit": "suffix: validations/min", "thresholds": { "steps": [ { @@ -347,7 +347,7 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "cpm", + "unit": "suffix: validations/min", "custom": {} }, "overrides": [] @@ -569,9 +569,9 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "cpm", + "unit": "suffix: validations/min", "custom": { - "axisLabel": "Per Minute", + "axisLabel": "Validations / Min", "drawStyle": "line", "lineWidth": 2, "fillOpacity": 10, @@ -657,7 +657,7 @@ "h": 8, "w": 24, "x": 0, - "y": 91 + "y": 123 }, "options": { "tooltip": { @@ -703,7 +703,7 @@ "h": 8, "w": 24, "x": 0, - "y": 99 + "y": 91 }, "options": { "tooltip": { @@ -737,7 +737,7 @@ "h": 8, "w": 24, "x": 0, - "y": 107 + "y": 99 }, "options": { "tooltip": { @@ -757,7 +757,7 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "suffix:/hr", + "unit": "suffix: changes/hr", "thresholds": { "steps": [ { @@ -787,7 +787,7 @@ "h": 8, "w": 24, "x": 0, - "y": 115 + "y": 107 }, "options": { "tooltip": { @@ -807,7 +807,7 @@ "fieldConfig": { "defaults": { "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]", - "unit": "cpm", + "unit": "suffix: ledgers/min", "thresholds": { "steps": [ { @@ -837,7 +837,7 @@ "h": 8, "w": 24, "x": 0, - "y": 123 + "y": 115 }, "options": { "tooltip": { diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md index aaee9385fe..b41fbffa7b 100644 --- a/docs/telemetry-runbook.md +++ b/docs/telemetry-runbook.md @@ -1095,32 +1095,32 @@ Requires `trace_peer=1` in the `[telemetry]` config section. ### Node Health -- System Metrics (`node-health`) -| Panel | Type | PromQL | Labels Used | -| -------------------------------------- | ---------- | ---------------------------------------------------------- | ---------------- | -| Validated Ledger Age | stat | `ledgermaster_validated_ledger_age` | — | -| Published Ledger Age | stat | `ledgermaster_published_ledger_age` | — | -| Operating Mode (Time Share) | timeseries | `rate(state_accounting_X_duration) / sum(rate(all modes))` | — | -| Operating Mode Transitions | timeseries | `state_accounting_*_transitions` | — | -| I/O Latency | timeseries | `histogram_quantile(0.95, ios_latency_bucket)` | — | -| Job Queue Depth | timeseries | `job_count` | — | -| Ledger Fetch Rate | stat | `rate(ledger_fetches[5m])` | — | -| Ledger History Mismatches | stat | `rate(ledger_history_mismatch[5m])` | — | -| Key Jobs Execution Time | timeseries | `acceptledger{quantile="$quantile"}` (+ 10 more key jobs) | `quantile` | -| Key Jobs Dequeue Wait Time | timeseries | `acceptledger_q{quantile="$quantile"}` (+ 10 more) | `quantile` | -| FullBelowCache Size | timeseries | `node_family_full_below_cache_size` | — | -| FullBelowCache Hit Rate | gauge | `node_family_full_below_cache_hit_rate` | — | -| Ledger Publish Gap | stat | `Published_Ledger_Age - Validated_Ledger_Age` | — | -| State Duration Rate (Full vs Tracking) | timeseries | `rate(state_accounting_full_duration[5m]) / 1000000` | — | -| All Jobs Execution Time (Detail) | timeseries | `{__name__=~"", quantile="$quantile"}` | `quantile` | -| All Jobs Dequeue Wait (Detail) | timeseries | `{__name__=~"_q", quantile="$quantile"}` | `quantile` | -| Server State | stat | `server_info{metric="server_state"}` | `metric` | -| Uptime | stat | `server_info{metric="uptime"}` | `metric` | -| Peer Count | stat | `server_info{metric="peers"}` | `metric` | -| Validated Ledger Seq | stat | `server_info{metric="validated_ledger_seq"}` | `metric` | -| Build Version | stat | `build_info` | `version` | -| Complete Ledger Ranges | table | `complete_ledgers` | `bound`, `index` | -| Database Sizes | timeseries | `db_metrics{metric=~"db_kb_.*"}` | `metric` | -| Historical Fetch Rate | stat | `db_metrics{metric="historical_perminute"}` | `metric` | +| Panel | Type | PromQL | Labels Used | +| -------------------------------- | ---------- | ---------------------------------------------------------- | ---------------- | +| Validated Ledger Age | stat | `ledgermaster_validated_ledger_age` | — | +| Published Ledger Age | stat | `ledgermaster_published_ledger_age` | — | +| Operating Mode (Time Share) | timeseries | `rate(state_accounting_X_duration) / sum(rate(all modes))` | — | +| Operating Mode Transitions | timeseries | `state_accounting_*_transitions` | — | +| I/O Latency | timeseries | `histogram_quantile(0.95, ios_latency_bucket)` | — | +| Job Queue Depth | timeseries | `job_count` | — | +| Ledger Fetch Rate | stat | `rate(ledger_fetches[5m])` | — | +| Ledger History Mismatches | stat | `rate(ledger_history_mismatch[5m])` | — | +| Key Jobs Execution Time | timeseries | `acceptledger{quantile="$quantile"}` (+ 10 more key jobs) | `quantile` | +| Key Jobs Dequeue Wait Time | timeseries | `acceptledger_q{quantile="$quantile"}` (+ 10 more) | `quantile` | +| FullBelowCache Size | timeseries | `node_family_full_below_cache_size` | — | +| FullBelowCache Hit Rate | gauge | `node_family_full_below_cache_hit_rate` | — | +| Ledger Publish Gap | stat | `Published_Ledger_Age - Validated_Ledger_Age` | — | +| State Duration Rate (All States) | timeseries | `rate(state_accounting__duration[5m]) / 1000000` | — | +| All Jobs Execution Time (Detail) | timeseries | `{__name__=~"", quantile="$quantile"}` | `quantile` | +| All Jobs Dequeue Wait (Detail) | timeseries | `{__name__=~"_q", quantile="$quantile"}` | `quantile` | +| Server State | stat | `server_info{metric="server_state"}` | `metric` | +| Uptime | stat | `server_info{metric="uptime"}` | `metric` | +| Peer Count | stat | `server_info{metric="peers"}` | `metric` | +| Validated Ledger Seq | stat | `server_info{metric="validated_ledger_seq"}` | `metric` | +| Build Version | stat | `build_info` | `version` | +| Complete Ledger Ranges | table | `complete_ledgers` | `bound`, `index` | +| Database Sizes | timeseries | `db_metrics{metric=~"db_kb_.*"}` | `metric` | +| Historical Fetch Rate | stat | `db_metrics{metric="historical_perminute"}` | `metric` | ### Network Traffic -- System Metrics (`network-traffic`)