From c1a1421aa058f37d555663c86426de4665215a3b Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Thu, 20 Aug 2026 16:41:38 +0100
Subject: [PATCH 01/38] docs: Document the [telemetry] trace toggles as 0 or 1

The configuration reference typed the five trace_* switches as bool with
default true. An xrpld config section carries integers, and these keys are
read with an integer cast, so a literal "true" fails to convert rather
than enabling the switch.

Type them as 0 or 1 with default 1, matching the other integer-valued
keys in the same table.
---
 OpenTelemetryPlan/05-configuration-reference.md | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/OpenTelemetryPlan/05-configuration-reference.md b/OpenTelemetryPlan/05-configuration-reference.md
index 9ea29ade1d..579ee10a66 100644
--- a/OpenTelemetryPlan/05-configuration-reference.md
+++ b/OpenTelemetryPlan/05-configuration-reference.md
@@ -24,11 +24,11 @@ The authoritative `[telemetry]` example lives in `cfg/xrpld-example.cfg`. Teleme
 | `batch_size`          | uint   | `512`                             | Spans per export batch                               |
 | `batch_delay_ms`      | uint   | `5000`                            | Max delay before sending batch (ms)                  |
 | `max_queue_size`      | uint   | `2048`                            | Maximum queued spans                                 |
-| `trace_transactions`  | bool   | `true`                            | Enable transaction tracing                           |
-| `trace_consensus`     | bool   | `true`                            | Enable consensus tracing                             |
-| `trace_rpc`           | bool   | `true`                            | Enable RPC tracing                                   |
-| `trace_peer`          | bool   | `true`                            | Enable peer message tracing (high volume)            |
-| `trace_ledger`        | bool   | `true`                            | Enable ledger tracing                                |
+| `trace_transactions`  | 0 or 1 | `1`                               | Enable transaction tracing                           |
+| `trace_consensus`     | 0 or 1 | `1`                               | Enable consensus tracing                             |
+| `trace_rpc`           | 0 or 1 | `1`                               | Enable RPC tracing                                   |
+| `trace_peer`          | 0 or 1 | `1`                               | Enable peer message tracing (high volume)            |
+| `trace_ledger`        | 0 or 1 | `1`                               | Enable ledger tracing                                |
 | `service_name`        | string | `"xrpld"`                         | Service name (`service.name`) for traces and metrics |
 | `service_instance_id` | string | `<node_pubkey>`                   | Instance identifier                                  |
 

From 251cd181a7a9cc3315a9cbe208e9dc6cb18b446c Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Fri, 21 Aug 2026 12:29:02 +0100
Subject: [PATCH 02/38] fix: Reject unreadable [telemetry] TLS certificate
 paths at startup

With telemetry enabled and use_tls=1, makeTelemetrySetup now reads each
non-empty tls_ca_cert / tls_client_cert / tls_client_key path and refuses to
start when the file is missing or cannot be read. The message names the config
key, the path and the OS error, instead of leaving the problem to surface much
later as an opaque TLS handshake failure inside the exporter.

Reading the file with getFileContents, as the gRPC server already does for its
own ssl_cert and ssl_key pair, proves the file is both present and readable; an
existence test alone would miss a permissions problem. The contents are
discarded.

Both gates are deliberate. The check is skipped when enabled is 0, so a stale
cert line still cannot stop a node from booting, and when use_tls is 0, where
the exporter never opens the files. An empty path stays valid; for tls_ca_cert
it selects the system CA store.

Six GTest cases cover the three keys that can fail, the all-readable case, and
each gate on its own.
---
 src/libxrpl/telemetry/TelemetryConfig.cpp     |  43 ++++
 .../libxrpl/telemetry/TelemetryConfig.cpp     | 193 ++++++++++++++++--
 2 files changed, 220 insertions(+), 16 deletions(-)

diff --git a/src/libxrpl/telemetry/TelemetryConfig.cpp b/src/libxrpl/telemetry/TelemetryConfig.cpp
index 006f062009..f0cc61517f 100644
--- a/src/libxrpl/telemetry/TelemetryConfig.cpp
+++ b/src/libxrpl/telemetry/TelemetryConfig.cpp
@@ -8,6 +8,7 @@
  * See cfg/xrpld-example.cfg for the full list of available options.
  */
 
+#include <xrpl/basics/FileUtilities.h>
 #include <xrpl/basics/contract.h>
 #include <xrpl/config/BasicConfig.h>
 #include <xrpl/telemetry/Telemetry.h>
@@ -16,6 +17,7 @@
 #include <cstdint>
 #include <stdexcept>
 #include <string>
+#include <system_error>
 
 namespace xrpl::telemetry {
 
@@ -85,6 +87,35 @@ networkTypeFromId(std::uint32_t networkId)
     }
 }
 
+/**
+ * Throw unless the given path names a file this process can read.
+ *
+ * An empty path means the option is unset, which every caller allows. Reading
+ * the file proves it is both present and readable; testing existence alone
+ * would miss a permissions problem. The contents are discarded — nothing here
+ * checks that they parse as PEM.
+ *
+ * @param path       Path taken from the config, possibly empty.
+ * @param configKey  Config key the path came from, named in the message. Not
+ * called `key`, which would hide the `key` namespace above.
+ * @throws std::runtime_error  If the path is non-empty and cannot be read.
+ */
+void
+requireReadableFile(std::string const& path, char const* configKey)
+{
+    if (path.empty())
+        return;
+
+    std::error_code ec;
+    getFileContents(ec, path);
+    if (ec)
+    {
+        Throw<std::runtime_error>(
+            std::string{"[telemetry] "} + configKey + " cannot be read: " + path + " - " +
+            ec.message());
+    }
+}
+
 }  // namespace
 
 Telemetry::Setup
@@ -139,6 +170,18 @@ makeTelemetrySetup(
                 "[telemetry] tls_client_cert/tls_client_key require use_tls=1 "
                 "(set use_tls=1 to enable mutual TLS, or remove the cert paths).");
         }
+
+        // Still inside the enabled branch. The exporter opens these files only
+        // when TLS is on, so check them only then: a bad path behind use_tls=0
+        // stops nothing. Checking here turns what would otherwise surface much
+        // later as an opaque handshake failure into a startup error naming the
+        // key. Each path is optional; an empty one is skipped.
+        if (setup.useTls)
+        {
+            requireReadableFile(setup.tlsCertPath, key::tlsCaCert);
+            requireReadableFile(setup.tlsClientCertPath, key::tlsClientCert);
+            requireReadableFile(setup.tlsClientKeyPath, key::tlsClientKey);
+        }
     }
 
     // Head sampling is intentionally fixed at 1.0 (sample everything) and is
diff --git a/src/tests/libxrpl/telemetry/TelemetryConfig.cpp b/src/tests/libxrpl/telemetry/TelemetryConfig.cpp
index 2aff850c84..56a3ef4447 100644
--- a/src/tests/libxrpl/telemetry/TelemetryConfig.cpp
+++ b/src/tests/libxrpl/telemetry/TelemetryConfig.cpp
@@ -1,3 +1,4 @@
+#include <xrpl/basics/FileUtilities.h>
 #include <xrpl/beast/utility/Journal.h>
 #include <xrpl/config/BasicConfig.h>
 #include <xrpl/telemetry/Telemetry.h>
@@ -5,10 +6,13 @@
 #include <gmock/gmock.h>
 #include <gtest/gtest.h>
 
+#include <fstream>
 #include <stdexcept>
+#include <string>
 
 using namespace xrpl;
 
+using ::testing::AllOf;
 using ::testing::HasSubstr;
 using ::testing::ThrowsMessage;
 
@@ -25,15 +29,18 @@ namespace {
  * or an unexpected throw. Tests that never set the key are unaffected. One
  * source of truth still keeps the two files from drifting apart.
  *
- * clientCert and clientKey are the paths written to those keys. They are
+ * clientCert and clientKey are the paths written to those keys. They name files
+ * that do not exist, so they suit only the cases the readability check cannot
+ * reach: telemetry off, or use_tls off. A case with enabled=1 and use_tls=1
+ * must write real files with writeCertFile() below instead. They are
  * declared as `char const*` so they pass to Section::set() (which takes
  * `std::string const&`) and compare against the parsed std::string members
  * without an explicit conversion, exactly as a literal would.
  *
- * pairingError and useTlsError are message fragments. Both guards throw
- * std::runtime_error, so the exception type alone cannot tell them apart.
- * Each fragment occurs in exactly one of the two messages, so matching it
- * proves which guard fired.
+ * pairingError, useTlsError and readError are message fragments. All three
+ * guards throw std::runtime_error, so the exception type alone cannot tell
+ * them apart. Each fragment occurs in exactly one of the three messages, so
+ * matching it proves which guard fired.
  */
 namespace mtls {
 constexpr char const* keyClientCert = "tls_client_cert";
@@ -42,6 +49,7 @@ constexpr char const* clientCert = "/etc/ssl/client.pem";
 constexpr char const* clientKey = "/etc/ssl/client.key";
 constexpr char const* pairingError = "must be set together";
 constexpr char const* useTlsError = "require use_tls=1";
+constexpr char const* readError = "cannot be read";
 
 /**
  * Build a [telemetry] section carrying only the `enabled` key.
@@ -75,6 +83,27 @@ parseSection(Section const& section)
 {
     return telemetry::makeTelemetrySetup(section, "nHUtest123", "2.0.0", 0);
 }
+
+/**
+ * Write a placeholder certificate file at the given path.
+ *
+ * The parser only needs the file to exist and be readable, so the contents are
+ * irrelevant — nothing checks that they parse as PEM. The stream state is
+ * asserted, so a failed write shows up as a setup failure here rather than as a
+ * confusing failure in the case under test.
+ *
+ * @param path  Where to write the file, typically from TempDir::file().
+ * @return The same path, ready to pass to Section::set().
+ */
+std::string
+writeCertFile(std::string const& path)
+{
+    std::ofstream out{path};
+    out << "placeholder\n";
+    out.close();
+    EXPECT_TRUE(out.good()) << "could not create " << path;
+    return path;
+}
 }  // namespace mtls
 
 }  // namespace
@@ -121,6 +150,10 @@ TEST(TelemetryConfig, parse_empty_section)
 
 TEST(TelemetryConfig, parse_full_section)
 {
+    // The CA path has to name a real file: with enabled=1 and use_tls=1 the
+    // parser opens it, so a placeholder path would make this case throw.
+    TempDir const dir;
+    auto const caCert = mtls::writeCertFile(dir.file("ca.pem"));
     Section section;
     section.set("enabled", "1");
     section.set("service_name", "my-rippled");
@@ -128,7 +161,7 @@ TEST(TelemetryConfig, parse_full_section)
     section.set("exporter", "otlp_http");
     section.set("endpoint", "http://collector:4318/v1/traces");
     section.set("use_tls", "1");
-    section.set("tls_ca_cert", "/etc/ssl/ca.pem");
+    section.set("tls_ca_cert", caCert);
     section.set("batch_size", "256");
     section.set("batch_delay_ms", "3000");
     section.set("max_queue_size", "4096");
@@ -145,7 +178,7 @@ TEST(TelemetryConfig, parse_full_section)
     EXPECT_EQ(setup.serviceInstanceId, "custom-id");
     EXPECT_EQ(setup.exporterEndpoint, "http://collector:4318/v1/traces");
     EXPECT_TRUE(setup.useTls);
-    EXPECT_EQ(setup.tlsCertPath, "/etc/ssl/ca.pem");
+    EXPECT_EQ(setup.tlsCertPath, caCert);
     EXPECT_EQ(setup.batchSize, 256u);
     EXPECT_EQ(setup.batchDelay, std::chrono::milliseconds{3000});
     EXPECT_EQ(setup.maxQueueSize, 4096u);
@@ -158,17 +191,24 @@ TEST(TelemetryConfig, parse_full_section)
 
 TEST(TelemetryConfig, mtls_cert_and_key_both_set)
 {
-    // Telemetry on and use_tls=1, so both guards run and neither may fire.
+    // Telemetry on and use_tls=1, so all three checks run and none may fire.
+    // Both paths have to name real files, because the parser opens them here.
+    // No CA bundle is set, which is the case this covers: mTLS against a
+    // collector whose certificate the system CA store already vouches for.
+    TempDir const dir;
+    auto const cert = mtls::writeCertFile(dir.file("client.pem"));
+    auto const key = mtls::writeCertFile(dir.file("client.key"));
     Section section = mtls::makeSection(true);
     section.set("use_tls", "1");
-    section.set(mtls::keyClientCert, mtls::clientCert);
-    section.set(mtls::keyClientKey, mtls::clientKey);
+    section.set(mtls::keyClientCert, cert);
+    section.set(mtls::keyClientKey, key);
 
     auto const setup = mtls::parseSection(section);
     EXPECT_TRUE(setup.enabled);
     EXPECT_TRUE(setup.useTls);
-    EXPECT_EQ(setup.tlsClientCertPath, mtls::clientCert);
-    EXPECT_EQ(setup.tlsClientKeyPath, mtls::clientKey);
+    EXPECT_TRUE(setup.tlsCertPath.empty());
+    EXPECT_EQ(setup.tlsClientCertPath, cert);
+    EXPECT_EQ(setup.tlsClientKeyPath, key);
 }
 
 TEST(TelemetryConfig, mtls_cert_without_key_throws)
@@ -252,20 +292,141 @@ TEST(TelemetryConfig, mtls_default_no_client_tls_is_accepted)
 
 TEST(TelemetryConfig, mtls_neither_set_is_one_way_tls)
 {
-    // Telemetry is on so the guards run, and this config must pass both:
-    // one-way TLS with a CA bundle and no client certificate.
+    // Telemetry is on so the checks run, and this config must pass all of
+    // them: one-way TLS with a CA bundle and no client certificate. The CA
+    // path has to name a real file, because the parser opens it here.
+    TempDir const dir;
+    auto const caCert = mtls::writeCertFile(dir.file("ca.pem"));
     Section section = mtls::makeSection(true);
     section.set("use_tls", "1");
-    section.set("tls_ca_cert", "/etc/ssl/ca.pem");
+    section.set("tls_ca_cert", caCert);
 
     auto const setup = mtls::parseSection(section);
     EXPECT_TRUE(setup.enabled);
     EXPECT_TRUE(setup.useTls);
-    EXPECT_EQ(setup.tlsCertPath, "/etc/ssl/ca.pem");
+    EXPECT_EQ(setup.tlsCertPath, caCert);
     EXPECT_TRUE(setup.tlsClientCertPath.empty());
     EXPECT_TRUE(setup.tlsClientKeyPath.empty());
 }
 
+TEST(TelemetryConfig, tls_missing_client_cert_file_throws)
+{
+    // Both client paths are set and use_tls=1, so neither contradiction guard
+    // can fire and the readability check is the only reachable throw. Only the
+    // certificate is absent, so the message must name that key and that path.
+    //
+    // This case and the two below use an absent file. A file that exists but
+    // denies read permission is deliberately not covered: a test process
+    // running as root reads it anyway, so the case would not be reliable.
+    TempDir const dir;
+    auto const absentCert = dir.file("absent.pem");
+    Section section = mtls::makeSection(true);
+    section.set("use_tls", "1");
+    section.set(mtls::keyClientCert, absentCert);
+    section.set(mtls::keyClientKey, mtls::writeCertFile(dir.file("k.pem")));
+
+    EXPECT_THAT(
+        [&section] { mtls::parseSection(section); },
+        ThrowsMessage<std::runtime_error>(AllOf(
+            HasSubstr(mtls::readError), HasSubstr(mtls::keyClientCert), HasSubstr(absentCert))));
+}
+
+TEST(TelemetryConfig, tls_missing_client_key_file_throws)
+{
+    // The mirror image of the case above: the certificate is readable and only
+    // the private key is absent, so the key's name must appear instead.
+    TempDir const dir;
+    auto const absentKey = dir.file("absent.key");
+    Section section = mtls::makeSection(true);
+    section.set("use_tls", "1");
+    section.set(mtls::keyClientCert, mtls::writeCertFile(dir.file("c.pem")));
+    section.set(mtls::keyClientKey, absentKey);
+
+    EXPECT_THAT(
+        [&section] { mtls::parseSection(section); },
+        ThrowsMessage<std::runtime_error>(AllOf(
+            HasSubstr(mtls::readError), HasSubstr(mtls::keyClientKey), HasSubstr(absentKey))));
+}
+
+TEST(TelemetryConfig, tls_missing_ca_cert_file_throws)
+{
+    // One-way TLS with no client certificate, so the CA bundle is the only
+    // path checked.
+    TempDir const dir;
+    auto const absentCa = dir.file("absent-ca.pem");
+    Section section = mtls::makeSection(true);
+    section.set("use_tls", "1");
+    section.set("tls_ca_cert", absentCa);
+
+    EXPECT_THAT(
+        [&section] { mtls::parseSection(section); },
+        ThrowsMessage<std::runtime_error>(
+            AllOf(HasSubstr(mtls::readError), HasSubstr("tls_ca_cert"), HasSubstr(absentCa))));
+}
+
+TEST(TelemetryConfig, tls_readable_files_are_accepted)
+{
+    // Full mTLS with all three files present and readable: parsing must
+    // succeed and keep every path verbatim.
+    TempDir const dir;
+    auto const ca = mtls::writeCertFile(dir.file("ca.pem"));
+    auto const cert = mtls::writeCertFile(dir.file("c.pem"));
+    auto const key = mtls::writeCertFile(dir.file("k.pem"));
+    Section section = mtls::makeSection(true);
+    section.set("use_tls", "1");
+    section.set("tls_ca_cert", ca);
+    section.set(mtls::keyClientCert, cert);
+    section.set(mtls::keyClientKey, key);
+
+    telemetry::Telemetry::Setup setup;
+    ASSERT_NO_THROW(setup = mtls::parseSection(section));
+    EXPECT_TRUE(setup.enabled);
+    EXPECT_TRUE(setup.useTls);
+    EXPECT_EQ(setup.tlsCertPath, ca);
+    EXPECT_EQ(setup.tlsClientCertPath, cert);
+    EXPECT_EQ(setup.tlsClientKeyPath, key);
+}
+
+TEST(TelemetryConfig, tls_paths_not_checked_when_telemetry_disabled)
+{
+    // Telemetry off, so the files are never opened and absent paths must not
+    // stop the node from booting. use_tls stays 1 here, so the `enabled` gate
+    // is the only thing that can be suppressing the check.
+    TempDir const dir;
+    auto const absentCert = dir.file("absent.pem");
+    auto const absentKey = dir.file("absent.key");
+    Section section = mtls::makeSection(false);
+    section.set("use_tls", "1");
+    section.set(mtls::keyClientCert, absentCert);
+    section.set(mtls::keyClientKey, absentKey);
+
+    telemetry::Telemetry::Setup setup;
+    ASSERT_NO_THROW(setup = mtls::parseSection(section));
+    EXPECT_FALSE(setup.enabled);
+    EXPECT_TRUE(setup.useTls);
+    EXPECT_EQ(setup.tlsClientCertPath, absentCert);
+    EXPECT_EQ(setup.tlsClientKeyPath, absentKey);
+}
+
+TEST(TelemetryConfig, tls_ca_cert_not_checked_when_use_tls_off)
+{
+    // With TLS off the exporter never reads the CA path, so a missing file
+    // must not stop startup. Telemetry stays on here, so the use_tls gate is
+    // the only thing that can be suppressing the check. The client-cert keys
+    // cannot be used for this case: they trip the use_tls contradiction guard
+    // before any file is opened.
+    TempDir const dir;
+    auto const absentCa = dir.file("absent-ca.pem");
+    Section section = mtls::makeSection(true);
+    section.set("tls_ca_cert", absentCa);
+
+    telemetry::Telemetry::Setup setup;
+    ASSERT_NO_THROW(setup = mtls::parseSection(section));
+    EXPECT_TRUE(setup.enabled);
+    EXPECT_FALSE(setup.useTls);
+    EXPECT_EQ(setup.tlsCertPath, absentCa);
+}
+
 TEST(TelemetryConfig, null_telemetry_factory)
 {
     telemetry::Telemetry::Setup setup;

From a24db2e995b29ea749b298e522ce7ef1de798541 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Fri, 21 Aug 2026 12:29:35 +0100
Subject: [PATCH 03/38] docs: Document the [telemetry] TLS path readability
 check

Bring the three documentation surfaces in line with the new parse-time check:

- The @throws clause on makeTelemetrySetup now names the third failure
  condition and records that an empty path is skipped.
- cfg/xrpld-example.cfg states, under all three TLS keys, that with enabled=1
  and use_tls=1 a path that does not exist or cannot be read stops startup. The
  tls_ca_cert wording still says that empty selects the system CA store, since
  only a path that is set is checked.
- The runbook troubleshooting entry gains a third bullet for the "cannot be
  read" message, whose remedy is the path or its permissions rather than the
  certificate and key pairing.

Documentation only; no behaviour change.
---
 cfg/xrpld-example.cfg              | 19 ++++++++++++++-----
 docs/telemetry-runbook.md          | 21 ++++++++++++++-------
 include/xrpl/telemetry/Telemetry.h |  5 ++++-
 3 files changed, 32 insertions(+), 13 deletions(-)

diff --git a/cfg/xrpld-example.cfg b/cfg/xrpld-example.cfg
index a3bd8673ce..9f74c99c82 100644
--- a/cfg/xrpld-example.cfg
+++ b/cfg/xrpld-example.cfg
@@ -1714,6 +1714,11 @@ validators.txt
 #   Path to a PEM-encoded CA certificate bundle for TLS verification.
 #   Only used when use_tls=1. Default: empty (system CA store).
 #
+#   Leaving this empty stays valid and selects the system CA store. A path
+#   that is set is checked like the client paths below: with enabled=1 and
+#   use_tls=1, one that does not exist or cannot be read makes xrpld fail
+#   to start.
+#
 # tls_client_cert=
 #
 #   Path to this node's PEM-encoded client certificate, presented to the
@@ -1723,15 +1728,19 @@ validators.txt
 #   To enable mTLS, both tls_client_cert and tls_client_key must be
 #   specified. If only one is provided, xrpld will fail to start. Providing
 #   them while use_tls=0 also fails to start, rather than being ignored.
-#   Both checks apply only when enabled=1; with telemetry disabled these
-#   settings are read but never validated.
+#   With use_tls=1 each path is opened at startup, so one that does not
+#   exist or cannot be read fails to start too, rather than failing later
+#   as an opaque TLS handshake error. All three checks apply only when
+#   enabled=1; with telemetry disabled these settings are read but never
+#   validated.
 #
 # tls_client_key=
 #
 #   Path to the PEM-encoded private key for tls_client_cert. Required
-#   whenever tls_client_cert is set. Requires use_tls=1. Both conditions
-#   are enforced exactly as described under tls_client_cert above: when
-#   enabled=1, breaking either one makes xrpld fail to start.
+#   whenever tls_client_cert is set. Requires use_tls=1, and must be
+#   readable. All three conditions are enforced exactly as described under
+#   tls_client_cert above: when enabled=1, breaking any one of them makes
+#   xrpld fail to start.
 #   Default: empty.
 #
 #   Head sampling is intentionally fixed at 1.0 (sample everything) and is
diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md
index f30b87eaea..eb3a6e9c28 100644
--- a/docs/telemetry-runbook.md
+++ b/docs/telemetry-runbook.md
@@ -612,12 +612,13 @@ Three dashboards are pre-provisioned in `docker/telemetry/grafana/dashboards/`:
   exception thrown while the `Application` object is constructed prints the same
   `Unable to start` prefix, so confirm the text after the colon begins with
   `[telemetry]` before using this entry
-- Cause: the `[telemetry]` mTLS keys (`tls_client_cert` and `tls_client_key`)
-  contradict each other. Only these two mTLS checks are gated on `enabled=1`;
-  the rest of the section is still read when telemetry is off, so a malformed
-  value in any key — including `enabled` itself, which is read before the gate
-  — still fails startup with a different message
-- Fix: the two checks need different remedies, and the printed message says
+- Cause: either the `[telemetry]` mTLS keys (`tls_client_cert` and
+  `tls_client_key`) contradict each other, or one of the TLS certificate paths
+  cannot be read. Only these three checks are gated on `enabled=1`; the rest of
+  the section is still read when telemetry is off, so a malformed value in any
+  key — including `enabled` itself, which is read before the gate — still fails
+  startup with a different message
+- Fix: the three checks need different remedies, and the printed message says
   which one fired
   - `tls_client_cert and tls_client_key must be set together` — exactly one of
     the two paths is set. Either delete the one that is set, or add the missing
@@ -626,8 +627,14 @@ Three dashboards are pre-provisioned in `docker/telemetry/grafana/dashboards/`:
   - `tls_client_cert/tls_client_key require use_tls=1` — both paths are set but
     TLS is off. Either set `use_tls=1`, or delete **both** paths. Deleting only
     one of them trips the first check
+  - `<key> cannot be read` — the named key (`tls_ca_cert`, `tls_client_cert` or
+    `tls_client_key`) points at a file the node cannot open; the message also
+    prints the path and the OS error. Fix the path or its permissions — the
+    pairing is not what is wrong here. This check runs only when `use_tls=1`,
+    and an empty `tls_ca_cert` is always accepted (it selects the system CA
+    store)
   - If you did not mean to enable telemetry at all, set `enabled=0` — that
-    clears both checks whichever one fired
+    clears all three checks whichever one fired
 
 ## Performance Tuning
 
diff --git a/include/xrpl/telemetry/Telemetry.h b/include/xrpl/telemetry/Telemetry.h
index a34e3cf19b..bf5af0156c 100644
--- a/include/xrpl/telemetry/Telemetry.h
+++ b/include/xrpl/telemetry/Telemetry.h
@@ -431,7 +431,10 @@ makeTelemetry(Telemetry::Setup const& setup, beast::Journal journal);
  * @return A populated Setup struct with defaults for missing values.
  * @throws std::runtime_error  If `enabled` is set and the mutual TLS (mTLS)
  * settings contradict each other: only one of `tls_client_cert`/`tls_client_key`
- * is given, or a client certificate is given while `use_tls` is 0. Those two
+ * is given, or a client certificate is given while `use_tls` is 0. Also if
+ * `enabled` and `use_tls` are both set and a non-empty `tls_ca_cert`,
+ * `tls_client_cert` or `tls_client_key` cannot be read; an empty path is skipped,
+ * so an empty `tls_ca_cert` still means "use the system CA store". All three
  * checks are skipped when `enabled` is 0.
  * @throws boost::bad_lexical_cast  If any numeric key (`enabled`, `use_tls`,
  * `batch_size`, the trace switches, ...) holds a value Section::valueOr cannot

From d1b80e47a29a040eddb55d68c611e117ec8ffa66 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 19:25:24 +0100
Subject: [PATCH 04/38] test(telemetry): void span baselines captured on the
 old span ladder

---
 docker/telemetry/workload/baselines/README.md |  22 ++++
 .../workload/baselines/baseline-timings.json  | 102 +-----------------
 .../workload/regression-thresholds.json       |   2 +-
 3 files changed, 26 insertions(+), 100 deletions(-)

diff --git a/docker/telemetry/workload/baselines/README.md b/docker/telemetry/workload/baselines/README.md
index fffea2e34d..4adfbe9b18 100644
--- a/docker/telemetry/workload/baselines/README.md
+++ b/docker/telemetry/workload/baselines/README.md
@@ -25,6 +25,28 @@ was invoked with. Capture and comparison are profile-agnostic — they only read
 Prometheus — so all existing profiles (`full-validation`, `quick-smoke`, `stress`)
 continue to work unchanged.
 
+## Current state: the baseline is a placeholder
+
+`baseline-timings.json` currently carries `"placeholder": true` and an empty `metrics` object,
+so **no metric gates right now**. Its entries were captured on 2026-06-05 against a spanmetrics
+ladder that was re-cut on 2026-08-04 in `3860c93db2`, which makes every sub-millisecond quantile
+in that capture bucket-edge arithmetic rather than a latency (a p95 of `0.95` ms is `0.95 × 1 ms`).
+Because the comparator only flags a metric when the current value _exceeds_ the baseline, a
+stale-high baseline passes everything silently — so the entries were voided instead of left in
+place. The file's `_note` records why, and which numbers were dropped.
+
+To restore gating, follow [Bootstrapping the baseline](#bootstrapping-the-baseline) below — a
+placeholder is exactly the state that loop expects. Pasting the CI block **replaces the whole
+file**, `_note` included; that is intended, and the voided numbers stay retrievable from this
+file's git history.
+
+**Do not let the placeholder outlive one run.** CI stays green the whole time the placeholder
+stands, so an un-copied block is not a failure anyone will notice — it is a silent loss of
+regression coverage that looks identical to a passing gate.
+
+Voiding a baseline is the one hand edit this file allows; _setting_ one always comes from a
+printed CI block, per the "Refreshing the baseline" rule below.
+
 ## Bootstrapping the baseline
 
 1. Merge a CI run with a `"placeholder": true` baseline. The telemetry-validation
diff --git a/docker/telemetry/workload/baselines/baseline-timings.json b/docker/telemetry/workload/baselines/baseline-timings.json
index 4e9633568b..0f9b858e5e 100644
--- a/docker/telemetry/workload/baselines/baseline-timings.json
+++ b/docker/telemetry/workload/baselines/baseline-timings.json
@@ -1,105 +1,9 @@
 {
-  "_note": "job.* entries were removed on 2026-08-21. They were captured against the old microsecond ladder whose first edge was 100us, with 99.3% of job_queued_us samples beneath it, so job.acceptLedger.queued.p95 = 96.79us was 0.95/0.9926 x 100 -- arithmetic on the bucket edge, not a latency. Recapture them on a node running the re-cut ladder (floor 1us); until then the comparer reports them as \"new metric (not in baseline)\" and gates only the span metrics, which are unaffected. Removed values, for reference: job.acceptLedger.queued.p95=96.79us, job.acceptLedger.running.p95=10562.50us, job.transaction.queued.p95=478.97us, job.transaction.running.p95=494.14us.",
+  "_note": "PLACEHOLDER -- every entry was voided, and the next telemetry-validation run reprints a fresh block to paste back (see baselines/README.md). Span entries were captured 2026-06-05 against a spanmetrics ladder whose first edge was 1ms. That ladder had no edge below 1ms at all, so for the sub-millisecond spans nearly every sample landed in bucket 0 and their quantiles are interpolation across that single edge, not latencies: span.ledger.store read p50/p95/p99 = 0.5/0.95/0.99, exactly quantile x 1ms, and span.tx.process read 0.50428/0.95813/0.99847 for the same reason. Sub-millisecond edges (floor 0.01ms) landed 2026-08-04 in 3860c93db2, so the ladder those numbers came from no longer exists. The gate is one-sided -- compare_to_baseline.py flags a regression only when current exceeds baseline -- so a stale-high baseline passes everything in silence, including a real 10x regression from 0.05ms to 0.5ms hiding under the 0.95ms entry. The 11 entries at or above 1ms (consensus.accept p50/p95/p99, consensus.ledger_close p95/p99, ledger.build p95/p99, ledger.validate p95/p99, tx.apply p95/p99) were undistorted -- but only because every edge from 1ms to 1s is byte-identical between the two ladders and all 11 happen to fall in that band, NOT because the re-cut spared everything above 1ms. It did not: it also added 2s/3s/4s/10s/30s edges above 1s, so a span whose quantiles reach the second scale is distorted just as much. Those 11 are voided with the rest anyway: a half-filled metrics object is not a placeholder, so the gate would stay half-dead with nothing to signal it. Removed span values, for reference (ms, p50/p95/p99): consensus.accept = 1.059405940594059/9.749999999999996/23.704545454545432; consensus.ledger_close = 0.5284697508896797/1.511111111111103/7.878571428571429; ledger.build = 0.7412060301507538/4.611111111111112/7.541666666666674; ledger.store = 0.5/0.95/0.9900000000000001; ledger.validate = 0.5283687943262412/1.3666666666666627/6.699999999999978; rpc.ws_message = 0.5026522773001647/0.9550393268703128/0.9952515090543261; tx.apply = 0.6330472103004292/4.203389830508474/5.083333333333319; tx.process = 0.5042801992591597/0.9581323781882418/0.998474791584883. job.* entries were removed earlier, on 2026-08-21, for the same reason on the native microsecond ladder: its first edge was 100us with 99.3% of job_queued_us samples beneath it, so job.acceptLedger.queued.p95 = 96.79us was 0.95/0.9926 x 100. That earlier note also claimed the gate then \"gates only the span metrics, which are unaffected\" -- that was wrong, for the reasons recorded above. Removed job values, for reference (us): job.acceptLedger.queued.p95=96.79, job.acceptLedger.running.p95=10562.50, job.transaction.queued.p95=478.97, job.transaction.running.p95=494.14. Both groups need recapture on a node running the re-cut ladders. captured_at, git_sha, profile and window below describe the voided run, not a live baseline.",
   "captured_at": "2026-06-05T18:41:52Z",
   "git_sha": "fd1c8c6060f7a15cc9e65b16f99629d9ab7ac7dc",
-  "metrics": {
-    "span.consensus.accept.p50": {
-      "unit": "ms",
-      "value": 1.059405940594059
-    },
-    "span.consensus.accept.p95": {
-      "unit": "ms",
-      "value": 9.749999999999996
-    },
-    "span.consensus.accept.p99": {
-      "unit": "ms",
-      "value": 23.704545454545432
-    },
-    "span.consensus.ledger_close.p50": {
-      "unit": "ms",
-      "value": 0.5284697508896797
-    },
-    "span.consensus.ledger_close.p95": {
-      "unit": "ms",
-      "value": 1.511111111111103
-    },
-    "span.consensus.ledger_close.p99": {
-      "unit": "ms",
-      "value": 7.878571428571429
-    },
-    "span.ledger.build.p50": {
-      "unit": "ms",
-      "value": 0.7412060301507538
-    },
-    "span.ledger.build.p95": {
-      "unit": "ms",
-      "value": 4.611111111111112
-    },
-    "span.ledger.build.p99": {
-      "unit": "ms",
-      "value": 7.541666666666674
-    },
-    "span.ledger.store.p50": {
-      "unit": "ms",
-      "value": 0.5
-    },
-    "span.ledger.store.p95": {
-      "unit": "ms",
-      "value": 0.95
-    },
-    "span.ledger.store.p99": {
-      "unit": "ms",
-      "value": 0.9900000000000001
-    },
-    "span.ledger.validate.p50": {
-      "unit": "ms",
-      "value": 0.5283687943262412
-    },
-    "span.ledger.validate.p95": {
-      "unit": "ms",
-      "value": 1.3666666666666627
-    },
-    "span.ledger.validate.p99": {
-      "unit": "ms",
-      "value": 6.699999999999978
-    },
-    "span.rpc.ws_message.p50": {
-      "unit": "ms",
-      "value": 0.5026522773001647
-    },
-    "span.rpc.ws_message.p95": {
-      "unit": "ms",
-      "value": 0.9550393268703128
-    },
-    "span.rpc.ws_message.p99": {
-      "unit": "ms",
-      "value": 0.9952515090543261
-    },
-    "span.tx.apply.p50": {
-      "unit": "ms",
-      "value": 0.6330472103004292
-    },
-    "span.tx.apply.p95": {
-      "unit": "ms",
-      "value": 4.203389830508474
-    },
-    "span.tx.apply.p99": {
-      "unit": "ms",
-      "value": 5.083333333333319
-    },
-    "span.tx.process.p50": {
-      "unit": "ms",
-      "value": 0.5042801992591597
-    },
-    "span.tx.process.p95": {
-      "unit": "ms",
-      "value": 0.9581323781882418
-    },
-    "span.tx.process.p99": {
-      "unit": "ms",
-      "value": 0.998474791584883
-    }
-  },
+  "metrics": {},
+  "placeholder": true,
   "profile": "full-validation",
   "schema_version": 1,
   "window": "3m"
diff --git a/docker/telemetry/workload/regression-thresholds.json b/docker/telemetry/workload/regression-thresholds.json
index 0dba5b6845..7e474569a9 100644
--- a/docker/telemetry/workload/regression-thresholds.json
+++ b/docker/telemetry/workload/regression-thresholds.json
@@ -1,6 +1,6 @@
 {
   "_description": "Per-metric regression thresholds. A metric regresses when current - baseline exceeds BOTH the percentage and absolute bounds (AND, not OR — this tolerates small-value noise). Defaults apply unless a per-metric override exists.",
-  "_bucket_note": "SpanMetrics latency histograms use explicit buckets [0.01,0.05,0.1,0.25,0.5,1,5,10,25,50,100,250,500]ms then [1,2,3,4,5,10,30]s (20 edges; docker/telemetry/otel-collector-config.yaml is the authoritative list). An earlier version of this note claimed 15 edges starting at 1ms and justified the 10ms absolute span bound as \"~2 low-end bucket widths\" — that derivation is void, because the sub-millisecond edges make the low-end bucket width 0.01ms, not 5ms. The 10ms bound is retained on its own merit: it is roughly two bucket widths in the 5-25ms band where most span quantiles actually sit, so it still absorbs single-bucket quantization jitter while catching multi-bucket regressions. Second-scale consensus spans have 2s/3s/4s boundaries, so their quantiles quantize to ~1s widths there. The job_queue running bound is widened similarly — per-ledger apply work scales with TxQ burst load. NOTE: the native job_queue histograms are microsecond-valued and their ladder was re-cut (floor 100us → 1us), so any job_queue baseline captured before that change is an interpolation artefact, not a latency.",
+  "_bucket_note": "SpanMetrics latency histograms use explicit buckets [0.01,0.05,0.1,0.25,0.5,1,5,10,25,50,100,250,500]ms then [1,2,3,4,5,10,30]s (20 edges; docker/telemetry/otel-collector-config.yaml is the authoritative list). An earlier version of this note claimed 15 edges starting at 1ms and justified the 10ms absolute span bound as \"~2 low-end bucket widths\" — that derivation is void, because the sub-millisecond edges make the low-end bucket width 0.01ms, not 5ms. The 10ms bound is retained on its own merit: it is roughly two bucket widths in the 5-25ms band where most span quantiles actually sit, so it still absorbs single-bucket quantization jitter while catching multi-bucket regressions. Second-scale consensus spans have 2s/3s/4s boundaries, so their quantiles quantize to ~1s widths there. The job_queue running bound is widened similarly — per-ledger apply work scales with TxQ burst load. NOTE: BOTH ladders were re-cut, and a baseline captured before its own ladder changed is an interpolation artefact, not a latency. The native job_queue histograms are microsecond-valued and their floor moved 100us → 1us. The span ladder was re-cut too, on 2026-08-04 in 3860c93db2, moving the floor 1ms → 0.01ms; so any sub-millisecond span quantile captured before that date is equally void — a p95 reading 0.95ms is 0.95 × the old 1ms first edge, not a measurement. An earlier note asserted that the surviving span baselines were unaffected by the ladder work; that is wrong for every span quantile below 1ms. Only the band from 1ms to 1s is safe: those edges are byte-identical across the two ladders. The re-cut also ADDED edges above 1s (2s/3s/4s/10s/30s), so a span whose quantiles land in the second-scale range — consensus.round ~3.9s, consensus.establish ~1.9s, the ledger.acquire tail — is distorted just as much, and any pre-2026-08-04 baseline for it is equally void. Do not read this note as licensing a stale second-scale baseline.",
   "defaults": {
     "span": {
       "p50": { "max_pct_increase": 50.0, "max_abs_increase_ms": 10.0 },

From 14badbfdd744e22049243b78de98f6f03f5d7764 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 19:46:18 +0100
Subject: [PATCH 05/38] docs(telemetry): record rpc_size_bytes and the other
 unasserted histograms

---
 docker/telemetry/workload/expected_metrics.json | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/docker/telemetry/workload/expected_metrics.json b/docker/telemetry/workload/expected_metrics.json
index 2aa1d0fcb2..078f6caa84 100644
--- a/docker/telemetry/workload/expected_metrics.json
+++ b/docker/telemetry/workload/expected_metrics.json
@@ -175,7 +175,11 @@
       "getobject_request_objects": "GetObjectMetricNames.h:86, emitted from PeerImp.cpp:2926 only while serving an inbound TMGetObjectByHash. The XRPL_METRIC_* macros create their instrument lazily on first use (MetricMacros.h:174-285), so no series exists until a peer actually requests objects by hash — which a 5-node cluster started at genesis and already in sync may never do.",
       "getobject_lookup_us": "GetObjectMetricNames.h:95, PeerImp.cpp:2929. Same lazy-creation and same inbound-request gate as getobject_request_objects.",
       "getobject_lookups_total": "GetObjectMetricNames.h:100, PeerImp.cpp:2949/:2956. Same gate.",
-      "getobject_charge": "GetObjectMetricNames.h:105, PeerImp.cpp:2931. Same gate."
+      "getobject_charge": "GetObjectMetricNames.h:105, PeerImp.cpp:2931. Same gate.",
+      "rpc_size_bytes": "ServerHandler.cpp:191, group('rpc')->makeEvent('size', Unit::Bytes). The OTLP Prometheus exporter derives the metric-name suffix from the declared unit, so a byte unit yields rpc_size_bytes; before 24094e427b the event declared no unit and exported as rpc_size_milliseconds on the millisecond bucket ladder. Neither name was ever recorded here, so the harness could confirm neither the rename nor a regression back onto that ladder. Notified from ServerHandler::processRequest:1133, the HTTP JSON-RPC path — it computes an HTTP status and appends a trailing newline — and the load generators are WebSocket-only, the same gate regression-metrics.json:4 records for rpc.process, so only the harness's handful of HTTP health polls reach it. Real coverage needs an HTTP JSON-RPC phase in rpc_load_generator.py; that is a workload change rather than a harness correction, and is deliberately out of scope here.",
+      "rpc_time_milliseconds": "ServerHandler.cpp:192, group('rpc')->makeEvent('time') with the default millisecond unit. Notified from ServerHandler::processRequest:1129, the same HTTP JSON-RPC call site as rpc_size_bytes and behind the same WebSocket-only gate.",
+      "ios_latency_milliseconds": "Application.cpp:515, makeEvent('ios_latency') with the default millisecond unit. Emitted and dashboarded, but it measures io-service scheduling delay rather than anything the workload drives, so no workload phase guarantees it.",
+      "jobq_*_milliseconds, jobq_*_q_milliseconds": "Created per job type in JobTypeData.h:97-98 from info.name() and info.name() + kSuffixQueued ('_q'), so the exported names are jobq_<jobtype>_milliseconds and jobq_<jobtype>_q_milliseconds with the job type lowercased by formatName. Which job types appear depends on which jobs a run happens to schedule, so no individual name is guaranteed. They are also rounded up to a whole millisecond at source (Event.h:47-51 applies ceil to a millisecond value type), which is why 6e2b2da772 moved the ledger-data-sync q-wait panels off jobq_<jobtype>_q_milliseconds_bucket onto job_queued_us_bucket — they are poor assertion targets regardless."
     }
   },
   "grafana_dashboards": {

From 2afae6655ddd3c226825d52cb7fb8929981c8bd9 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 19:53:43 +0100
Subject: [PATCH 06/38] test(telemetry): assert xrpl_node_id reaches the metric
 series

---
 docker/telemetry/workload/README.md           |   5 +-
 .../telemetry/workload/expected_metrics.json  |   5 +-
 .../telemetry/workload/validate_telemetry.py  | 219 +++++++++++++++---
 3 files changed, 199 insertions(+), 30 deletions(-)

diff --git a/docker/telemetry/workload/README.md b/docker/telemetry/workload/README.md
index 9221b47713..9d96499051 100644
--- a/docker/telemetry/workload/README.md
+++ b/docker/telemetry/workload/README.md
@@ -416,7 +416,8 @@ them again — load shape comes entirely from `--profile` and
   "description": "Top-level doc string — skipped by the validator.",
   "category_name": {
     "description": "Human-readable description.",
-    "metrics": ["metric_1", "metric_2"]
+    "metrics": ["metric_1", "metric_2"],
+    "required_labels": ["label_1"]
   },
   "grafana_dashboards": {
     "uids": ["rpc-performance", "node-health"]
@@ -430,6 +431,8 @@ them again — load shape comes entirely from `--profile` and
 
 Every metric listed under a `metrics` array must produce > 0 Prometheus series during the validation run. If a metric doesn't fire, the workload generators need to produce enough load to trigger it.
 
+`required_labels` is optional and read for every category that declares one. Each label becomes one additional check, named `metric.<category>.label.<label>`, that at least one of that category's series carries the label with a non-empty value. It is matched as `<label>!=""` rather than `<label>=~".*"` because Prometheus cannot tell an absent label from an empty one, so a regex match would pass on a node that lost the label entirely. The guarantee rule is the same as for `metrics`: list a label only where the workload guarantees it. Declaring `required_labels` on a category with no `metrics` fails the check rather than skipping it, so the key can never sit unenforced.
+
 Three top-level keys are not metric categories:
 
 - `description` and `grafana_dashboards` are skipped explicitly by
diff --git a/docker/telemetry/workload/expected_metrics.json b/docker/telemetry/workload/expected_metrics.json
index 078f6caa84..7049b49593 100644
--- a/docker/telemetry/workload/expected_metrics.json
+++ b/docker/telemetry/workload/expected_metrics.json
@@ -1,5 +1,5 @@
 {
-  "description": "Expected metric inventory for xrpld telemetry validation. Metric names have no prefix (the xrpld_ prefix was removed). beast::insight metrics are lowercased by formatName. Every name here was verified against its declaration in MetricsRegistry.cpp or include/xrpl/telemetry/GetObjectMetricNames.h and against a panel query under docker/telemetry/grafana/dashboards/. IMPORTANT: validate_telemetry.py has no notion of an optional metric — validate_metrics() iterates every group that has a \"metrics\" key and hard-fails any name with 0 Prometheus series after a 45 s poll. A metric is therefore listed only when the harness workload guarantees it will appear: observable gauges/counters whose callbacks Observe unconditionally (series exist at value 0), or push counters/histograms on a path every run exercises. Workload-gated and defect-gated names are recorded in the \"not_asserted\" group, which intentionally has no \"metrics\" key so the validator skips it. Only series existence is checked, never a value, except for the four bounds checks hardcoded in PARITY_VALUE_SANITY.",
+  "description": "Expected metric inventory for xrpld telemetry validation. Metric names have no prefix (the xrpld_ prefix was removed). beast::insight metrics are lowercased by formatName. Every name here was verified against its declaration in MetricsRegistry.cpp or include/xrpl/telemetry/GetObjectMetricNames.h and against a panel query under docker/telemetry/grafana/dashboards/. IMPORTANT: validate_telemetry.py has no notion of an optional metric — validate_metrics() iterates every group that has a \"metrics\" key and hard-fails any name with 0 Prometheus series after a 45 s poll. A metric is therefore listed only when the harness workload guarantees it will appear: observable gauges/counters whose callbacks Observe unconditionally (series exist at value 0), or push counters/histograms on a path every run exercises. Workload-gated and defect-gated names are recorded in the \"not_asserted\" group, which intentionally has no \"metrics\" key so the validator skips it. Only series existence is checked, never a value, except for the four bounds checks hardcoded in PARITY_VALUE_SANITY. A group may additionally declare \"required_labels\": each label there becomes one check that at least one of the group's series carries it with a non-empty value, matched as <label>!=\"\" because Prometheus cannot tell an absent label from an empty one. The same guarantee rule applies — list a label only where the workload guarantees it.",
   "spanmetrics": {
     "description": "SpanMetrics-derived RED metrics from the OTel Collector spanmetrics connector.",
     "metrics": [
@@ -78,7 +78,8 @@
     ]
   },
   "job_queue": {
-    "description": "Job-queue counters and latency histograms (MetricsRegistry.cpp:360-366). Every xrpld job passes through these, so they populate under any workload. Both histograms are recorded in the same function bodies as job_started_total / job_finished_total, under the same guard and with the same labels, so their presence is equally guaranteed. They are named with the _bucket/_count/_sum suffixes the Prometheus exporter emits: regression-metrics.json and the job-queue dashboard both query job_queued_us_bucket / job_running_us_bucket, and no bare series exists.",
+    "description": "Job-queue counters and latency histograms (MetricsRegistry.cpp:360-366). Every xrpld job passes through these, so they populate under any workload. Both histograms are recorded in the same function bodies as job_started_total / job_finished_total, under the same guard and with the same labels, so their presence is equally guaranteed. They are named with the _bucket/_count/_sum suffixes the Prometheus exporter emits: regression-metrics.json and the job-queue dashboard both query job_queued_us_bucket / job_running_us_bucket, and no bare series exists. This group also carries the xrpl_node_id required_labels assertion. 070d29b465 added the xrpl.node.id resource attribute so Grafana Cloud trace ingest would stop folding distinct nodes into one — ingest groups ResourceSpans but ignores service.instance.id, so before the attribute existed the boards reported one node's ledger.build five times over and saw 2 of 9 nodes. Nothing asserted it, so a regression would silently restore the folding. It is asserted here rather than on a statsd_* group because these metrics are MetricsRegistry-backed and guaranteed by any workload, whereas the beast::insight meter provider is constructed before the wallet DB exists and so has no node public key to stamp: those metrics legitimately lack the label and 070d29b465 omits it rather than writing it blank.",
+    "required_labels": ["xrpl_node_id"],
     "metrics": [
       "job_queued_total",
       "job_started_total",
diff --git a/docker/telemetry/workload/validate_telemetry.py b/docker/telemetry/workload/validate_telemetry.py
index 1ac076ea92..51b4f7bd74 100644
--- a/docker/telemetry/workload/validate_telemetry.py
+++ b/docker/telemetry/workload/validate_telemetry.py
@@ -9,7 +9,8 @@ Validation categories:
   1. Span validation     — Every required span type in expected_spans.json, each
                            carrying its required attributes
   2. Metric validation   — SpanMetrics, StatsD, and MetricsRegistry OTLP metrics
-                           are non-zero
+                           are non-zero, and each group's required_labels reach
+                           Prometheus with non-empty values
   3. Log-trace correlation — Loki logs contain trace_id/span_id fields
   4. Dashboard validation — Every dashboard uid in expected_metrics.json
                            provisions and loads (panel count only, not panel data)
@@ -700,6 +701,41 @@ async def _log_prometheus_metric_names(
         logger.warning("Failed to fetch Prometheus metric names: %s", exc)
 
 
+def _metric_check_targets(
+    expected: dict[str, Any],
+) -> tuple[list[tuple[str, str]], list[tuple[str, str, list[str]]]]:
+    """Flatten expected_metrics.json into the two lists of check targets.
+
+    Args:
+        expected: The parsed expected_metrics.json contract.
+
+    Returns:
+        A (metric targets, label targets) pair. Metric targets are
+        (group, metric selector) for every name under a group's "metrics".
+        Label targets are (group, label, that group's metric selectors) for
+        every name under a group's "required_labels" — read for every group
+        that declares it, not just spanmetrics. Nothing read the key at all
+        until this was added, so the four labels the spanmetrics group
+        documented as required had never actually been checked.
+    """
+    groups = [
+        (category_key, category_data)
+        for category_key, category_data in expected.items()
+        if category_key not in ("description", "grafana_dashboards")
+    ]
+    targets = [
+        (category_key, metric_name)
+        for category_key, category_data in groups
+        for metric_name in category_data.get("metrics", [])
+    ]
+    label_targets = [
+        (category_key, label, category_data.get("metrics", []))
+        for category_key, category_data in groups
+        for label in category_data.get("required_labels", [])
+    ]
+    return targets, label_targets
+
+
 async def validate_metrics(
     session: aiohttp.ClientSession,
     prometheus_url: str,
@@ -707,6 +743,11 @@ async def validate_metrics(
 ) -> None:
     """Validate that expected metrics appear in Prometheus with non-zero values.
 
+    Two kinds of check come out of expected_metrics.json: every name under a
+    group's "metrics" must have at least one series, and every label under a
+    group's "required_labels" must reach at least one of that group's series
+    with a non-empty value.
+
     Args:
         session:        aiohttp client session.
         prometheus_url: Base URL for Prometheus API (e.g., http://localhost:9090).
@@ -719,44 +760,86 @@ async def validate_metrics(
     with open(EXPECTED_METRICS_FILE) as f:
         expected = json.load(f)
 
-    # Flatten every (category, metric) pair the contract asserts, then poll
-    # them concurrently against ONE shared deadline. Polling them serially made
-    # each metric own its own timeout, so the waits were additive: 58 metrics x
-    # 45 s = 43.5 min, which overran the CI job budget and lost the
-    # artifact-upload and summary diagnostics. Sharing the deadline bounds the
-    # whole phase to a single poll window.
-    targets = [
-        (category_key, metric_name)
-        for category_key, category_data in expected.items()
-        if category_key not in ("description", "grafana_dashboards")
-        for metric_name in category_data.get("metrics", [])
-    ]
+    # Flatten the contract, then poll every target concurrently against ONE
+    # shared deadline. Polling them serially made each metric own its own
+    # timeout, so the waits were additive: 58 metrics x 45 s = 43.5 min, which
+    # overran the CI job budget and lost the artifact-upload and summary
+    # diagnostics. Sharing the deadline bounds the whole phase to a single
+    # poll window.
+    targets, label_targets = _metric_check_targets(expected)
 
     deadline = time.monotonic() + METRIC_POLL_TIMEOUT_SEC
     sem = asyncio.Semaphore(METRIC_POLL_CONCURRENCY)
-    checks = await asyncio.gather(
-        *(
-            _check_prometheus_metric(
-                session, prometheus_url, metric_name, category, deadline, sem
+    # Both kinds of check share the one deadline and the one concurrency bound,
+    # so the label checks cost no extra poll window and add no extra load.
+    metric_checks, label_checks = await asyncio.gather(
+        asyncio.gather(
+            *(
+                _check_prometheus_metric(
+                    session, prometheus_url, metric_name, category, deadline, sem
+                )
+                for category, metric_name in targets
             )
-            for category, metric_name in targets
-        )
+        ),
+        asyncio.gather(
+            *(
+                _check_metric_label(
+                    session,
+                    prometheus_url,
+                    category,
+                    label,
+                    metric_selectors,
+                    deadline,
+                    sem,
+                )
+                for category, label, metric_selectors in label_targets
+            )
+        ),
     )
 
     # Add in contract order, not completion order, so the report and its log
-    # lines stay deterministic across runs.
-    for check in checks:
+    # lines stay deterministic across runs. Existence checks keep their place
+    # ahead of the label checks, so no existing check's position moves.
+    for check in [*metric_checks, *label_checks]:
         report.add(check)
 
 
+def _selector_with_label(metric_selector: str, label: str) -> str:
+    """Add a "label is present and non-empty" matcher to a metric selector.
+
+    ``<label>!=""`` is the only matcher that expresses the requirement. An
+    absent Prometheus label is indistinguishable from an empty-string one, so
+    ``<label>=~".*"`` matches series that never carried the label at all and a
+    check written that way would pass on a node that lost the attribute
+    entirely. ``!=""`` rejects both the absent and the blank case.
+
+    Selectors in expected_metrics.json are usually bare names, but some already
+    carry a matcher (``ledger_economy{metric="base_fee_xrp"}``), so the matcher
+    is merged into an existing brace group rather than appended after it.
+
+    Args:
+        metric_selector: A metric name, optionally with a brace matcher group.
+        label:           Prometheus label name that must be present, non-empty.
+
+    Returns:
+        The selector with the label matcher merged in.
+    """
+    matcher = label + '!=""'
+    if metric_selector.endswith("}"):
+        head = metric_selector[:-1].rstrip()
+        separator = "" if head.endswith("{") else ", "
+        return head + separator + matcher + "}"
+    return metric_selector + "{" + matcher + "}"
+
+
 async def _poll_series_count(
     session: aiohttp.ClientSession,
     prometheus_url: str,
-    metric_name: str,
+    selectors: list[str],
     deadline: float,
     sem: asyncio.Semaphore,
 ) -> int:
-    """Poll Prometheus until a metric has series or the deadline passes.
+    """Poll Prometheus until a selector has series or the deadline passes.
 
     Uses the /api/v1/series endpoint instead of an instant query.
     Beast::insight StatsD gauges only mark dirty on value *changes*, so a gauge
@@ -771,16 +854,19 @@ async def _poll_series_count(
     Args:
         session:        aiohttp client session.
         prometheus_url: Prometheus base URL.
-        metric_name:    Prometheus metric name.
+        selectors:      One or more Prometheus selectors. /api/v1/series takes
+                        repeated match[] parameters and unions their results,
+                        so several selectors answer "does ANY of these have a
+                        series" in a single request.
         deadline:       Monotonic deadline shared by every metric in the run.
         sem:            Bounds how many requests reach Prometheus at once. It
                         is held only across the request, never across the
                         sleep, so one absent metric cannot starve the others.
 
     Returns:
-        Number of series found, or 0 if the metric never appeared.
+        Number of series found, or 0 if nothing ever appeared.
     """
-    params: dict[str, str] = {"match[]": metric_name}
+    params: list[tuple[str, str]] = [("match[]", s) for s in selectors]
     while True:
         async with sem:
             async with session.get(
@@ -818,7 +904,7 @@ async def _check_prometheus_metric(
     """
     try:
         series_count = await _poll_series_count(
-            session, prometheus_url, metric_name, deadline, sem
+            session, prometheus_url, [metric_name], deadline, sem
         )
         return CheckResult(
             name=f"metric.{category}.{metric_name}",
@@ -840,6 +926,85 @@ async def _check_prometheus_metric(
         )
 
 
+async def _check_metric_label(
+    session: aiohttp.ClientSession,
+    prometheus_url: str,
+    category: str,
+    label: str,
+    metric_selectors: list[str],
+    deadline: float,
+    sem: asyncio.Semaphore,
+) -> CheckResult:
+    """Check that a group's required label reaches Prometheus non-empty.
+
+    A "required_labels" entry names a label that at least one series of that
+    group must carry with a non-empty value. It is checked per group rather
+    than per metric because the requirement is about the label reaching
+    Prometheus at all, and one series carrying it proves the pipeline works.
+
+    The failure this guards against is a label going missing while every metric
+    in the group still exists, so the existence checks stay green. xrpl_node_id
+    is the case that motivated it: losing it makes Grafana Cloud trace ingest
+    fold distinct nodes into one.
+
+    Args:
+        session:          aiohttp client session.
+        prometheus_url:   Prometheus base URL.
+        category:         Group key from expected_metrics.json.
+        label:            Label name that must be present and non-empty.
+        metric_selectors: The group's asserted metric selectors.
+        deadline:         Monotonic deadline shared by every metric in the run.
+        sem:              Bounds how many requests reach Prometheus at once.
+
+    Returns:
+        The CheckResult for this (group, label) pair. The caller adds it to the
+        report so report order follows the contract file.
+    """
+    check_name = f"metric.{category}.label.{label}"
+
+    if not metric_selectors:
+        # required_labels on a group with no asserted metrics would be a check
+        # that silently never runs — the exact failure mode it exists to catch.
+        return CheckResult(
+            name=check_name,
+            category="metric",
+            passed=False,
+            message=(
+                f"{label}: group '{category}' declares required_labels but no "
+                "metrics, so there is nothing to check the label against"
+            ),
+        )
+
+    selectors = [_selector_with_label(m, label) for m in metric_selectors]
+    try:
+        series_count = await _poll_series_count(
+            session, prometheus_url, selectors, deadline, sem
+        )
+        return CheckResult(
+            name=check_name,
+            category="metric",
+            passed=series_count > 0,
+            message=(
+                f"{label}: present and non-empty on {series_count} "
+                f"{category} series"
+                if series_count > 0
+                else f"{label}: no {category} series carries it with a "
+                "non-empty value (an absent label and an empty one are the "
+                "same thing in Prometheus, so this covers both). If this "
+                "group's own metric checks also failed, the metrics are "
+                "missing rather than the label."
+            ),
+            details={"series_count": series_count, "selectors": selectors},
+        )
+    except Exception as exc:
+        return CheckResult(
+            name=check_name,
+            category="metric",
+            passed=False,
+            message=f"{label}: query failed ({exc})",
+        )
+
+
 # ---------------------------------------------------------------------------
 # Log-Trace Correlation Validation (Loki API)
 # ---------------------------------------------------------------------------

From 31619219cc514d8a9e0c76f64231f68a4ef61cd1 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 19:54:15 +0100
Subject: [PATCH 07/38] feat(telemetry): add set-once state attrs to consensus
 phase spans

consensus.phase.open and consensus.establish carried almost no state of
their own. Span attributes are not inherited, so the ledger context on the
parent consensus.round span does not describe either child, and the few
attributes the establish span did carry are rewritten on every iteration
and therefore only ever report the final value.

Add seven attributes that are read from state already in scope, are
written exactly once, and are not duplicates of the parent round span:

  consensus.phase.open (start)
    start_reason            initial, or recovered on a handleWrongLedger
                            re-entry, which emplaces a SECOND phase.open
                            span under the same round
    previous_close_agree    feeds the sinceClose branch in phaseOpen()
    peer_positions_at_open  positions in hand after playbackProposals(),
                            the head start the round began with
    early_close_triggered   the round skipped the timer because enough
                            peers had already closed

  consensus.phase.open (end)
    tx_sets_acquired        candidate tx sets held at close, read before
                            our own position is added; a low count against
                            a high peer_positions_at_close means tx-set
                            fetches did not land, not disagreement

  consensus.establish (start)
    disputes_count_initial  disputes carried in from the positions held at
                            close, as opposed to disputes_count, which is
                            overwritten each iteration

  consensus.establish (end)
    avalanche_state         terminal close-time convergence regime; the
                            derived avalanche_threshold is a weight and
                            cannot be inverted back to the state

The avalanche label is mapped by a new constexpr avalancheStateLabel() in
ConsensusSpanNames.h rather than an inline switch, so the four labels stay
under the naming check's L1 ownership and are unit-testable.

Deliberately not added: ledger_seq and consensus_mode, which would only
copy the parent round span's values down; tx set size and position hash,
which the TxSet concept does not expose portably across RCLTxSet and the
csf simulator; and the peer-unchanged and dead-node counters, whose
underlying state is reset mid-round and so would report a misleading value.

Behaviour is unchanged. The early-close condition is hoisted into a named
local so the annotation happens before timerEntry(), which can reach
closeLedger() and end the open-phase span.

Tests pin the wire strings for every new key and value and cover all four
enumerators of the avalanche mapping. They need no telemetry runtime: the
csf simulator returns an invalid round span context and a null Telemetry,
so consensus spans there are null guards and attribute writes are no-ops.
---
 include/xrpl/consensus/Consensus.h            | 67 +++++++++++++-
 include/xrpl/consensus/ConsensusSpanNames.h   | 82 ++++++++++++++++-
 .../libxrpl/telemetry/ConsensusSpanNames.cpp  | 90 +++++++++++++++++++
 3 files changed, 234 insertions(+), 5 deletions(-)
 create mode 100644 src/tests/libxrpl/telemetry/ConsensusSpanNames.cpp

diff --git a/include/xrpl/consensus/Consensus.h b/include/xrpl/consensus/Consensus.h
index 0baef3acfc..207534012b 100644
--- a/include/xrpl/consensus/Consensus.h
+++ b/include/xrpl/consensus/Consensus.h
@@ -669,7 +669,8 @@ private:
 
     /**
      * Create the establish-phase span if not yet active.
-     *  Called on each phaseEstablish() invocation; no-op while span is live.
+     *  Called on each phaseEstablish() invocation; no-op while span is live,
+     *  so the entry-state attributes it sets are written exactly once.
      */
     void
     startEstablishTracing();
@@ -683,6 +684,9 @@ private:
 
     /**
      * End the establish span when transitioning to the accepted phase.
+     *  Records the terminal avalanche_state before ending the span. A round
+     *  that instead loses the establish span to a wrongLedger recovery omits
+     *  the attribute rather than reporting a stale regime.
      */
     void
     endEstablishTracing();
@@ -783,6 +787,22 @@ Consensus<Adaptor>::startRoundInternal(
     openSpan_.emplace(
         telemetry::SpanGuard::childSpan(
             telemetry::consensus::span::phaseOpen, adaptor_.roundSpanContext()));
+    if (*openSpan_)
+    {
+        namespace cs = telemetry::consensus::span;
+        // Which entry path created this span. A handleWrongLedger recovery
+        // re-enters startRoundInternal and emplaces a SECOND phase.open span
+        // under the same round span, so without this the two are
+        // indistinguishable in a trace.
+        openSpan_->setAttribute(
+            cs::attr::startReason,
+            reason == StartRoundReason::Recovered ? std::string_view{cs::val::startRecovered}
+                                                  : std::string_view{cs::val::startInitial});
+        // Read from the parameter, not previousLedger_, which is not assigned
+        // until further down this function. Feeds the sinceClose calculation
+        // in phaseOpen().
+        openSpan_->setAttribute(cs::attr::previousCloseAgree, prevLedger.closeAgree());
+    }
     // On the Recovered path, fire phase.open here because startRoundTracing
     // (which fires it for the Initial path) is not called on re-entry. On
     // the Initial path this is a no-op because the round span hasn't been
@@ -817,10 +837,25 @@ Consensus<Adaptor>::startRoundInternal(
     playbackProposals();
     CLOG(clog) << "number of peer proposals,previous proposers: " << currPeerPositions_.size()
                << ',' << prevProposers_ << ". ";
-    if (currPeerPositions_.size() > (prevProposers_ / 2))
+    // We may be falling behind, don't wait for the timer
+    // consider closing the ledger immediately
+    bool const closeImmediately = currPeerPositions_.size() > (prevProposers_ / 2);
+    // Annotate before the timerEntry() below, which can reach closeLedger()
+    // and end this span.
+    if (openSpan_ && *openSpan_)
+    {
+        namespace cs = telemetry::consensus::span;
+        // Positions already in hand once playbackProposals() has replayed the
+        // buffered ones — the head start this round began with. Pairs with
+        // peer_positions_at_close on the same span.
+        openSpan_->setAttribute(
+            cs::attr::peerPositionsAtOpen, static_cast<int64_t>(currPeerPositions_.size()));
+        // Whether the round skipped waiting for the timer because enough peers
+        // had already closed.
+        openSpan_->setAttribute(cs::attr::earlyCloseTriggered, closeImmediately);
+    }
+    if (closeImmediately)
     {
-        // We may be falling behind, don't wait for the timer
-        // consider closing the ledger immediately
         CLOG(clog) << "consider closing the ledger immediately. ";
         timerEntry(now_, clog);
     }
@@ -1560,6 +1595,11 @@ Consensus<Adaptor>::closeLedger(std::unique_ptr<std::stringstream> const& clog)
             cs::attr::openDurationMs, static_cast<int64_t>(openTime_.read().count()));
         openSpan_->setAttribute(
             cs::attr::peerPositionsAtClose, static_cast<int64_t>(currPeerPositions_.size()));
+        // Candidate transaction sets held at close. Read before our own
+        // position is added below, so this counts only what peers shared. A
+        // low count against a high peer_positions_at_close means tx-set
+        // fetches did not land, not that peers disagreed.
+        openSpan_->setAttribute(cs::attr::txSetsAcquired, static_cast<int64_t>(acquired_.size()));
     }
     openSpan_.reset();
     phase_ = ConsensusPhase::Establish;
@@ -2113,6 +2153,16 @@ Consensus<Adaptor>::startEstablishTracing()
     if (*establishSpan_)
     {
         establishSpanContext_ = establishSpan_->spanContext();
+        // Disputes carried into the phase from the positions held at close.
+        // Set once (this function early-returns while the span is live),
+        // unlike disputes_count, which updateEstablishTracing() overwrites
+        // every iteration and so only ever reports the final value.
+        if (result_)
+        {
+            establishSpan_->setAttribute(
+                telemetry::consensus::span::attr::disputesCountInitial,
+                static_cast<int64_t>(result_->disputes.size()));
+        }
     }
 }
 
@@ -2138,6 +2188,15 @@ template <class Adaptor>
 void
 Consensus<Adaptor>::endEstablishTracing()
 {
+    // Terminal close-time convergence regime, recorded once before the span
+    // ends. The derived avalanche_threshold on consensus.update_positions is
+    // a weight, not the state, and cannot be inverted back to it.
+    if (establishSpan_ && *establishSpan_)
+    {
+        establishSpan_->setAttribute(
+            telemetry::consensus::span::attr::avalancheState,
+            telemetry::consensus::span::avalancheStateLabel(closeTimeAvalancheState_));
+    }
     establishSpan_.reset();
     establishSpanContext_ = telemetry::SpanContext{};
 }
diff --git a/include/xrpl/consensus/ConsensusSpanNames.h b/include/xrpl/consensus/ConsensusSpanNames.h
index 6c79cd3132..dfc492d962 100644
--- a/include/xrpl/consensus/ConsensusSpanNames.h
+++ b/include/xrpl/consensus/ConsensusSpanNames.h
@@ -21,6 +21,9 @@
  *    +-- consensus.phase.open                    [main thread, child]
  *    |     Created: Consensus::startRoundInternal()
  *    |     Ended:   Consensus::closeLedger()
+ *    |     Attrs:   start_reason, previous_close_agree, peer_positions_at_open,
+ *    |              early_close_triggered (all at start); open_duration_ms,
+ *    |              peer_positions_at_close, tx_sets_acquired (all at end)
  *    |
  *    +-- consensus.proposal.send                 [main thread]
  *    |     Created: Adaptor::propose()
@@ -33,7 +36,9 @@
  *    +-- consensus.establish                     [main thread, child]
  *    |     Created: Consensus::startEstablishTracing()
  *    |     Ended:   Consensus::phaseEstablish() on accept
- *    |     Attrs:   converge_percent, establish_count, proposers
+ *    |     Attrs:   disputes_count_initial (at start); converge_percent,
+ *    |              establish_count, proposers, disputes_count (overwritten
+ *    |              each iteration); avalanche_state (terminal, at end)
  *    |
  *    +-- consensus.update_positions              [main thread]
  *    |     Created: Consensus::updateOurPositions()
@@ -78,8 +83,11 @@
  *    +~~~ follows-from link (separate sub-tree, causal link)
  */
 
+#include <xrpl/consensus/ConsensusParms.h>
 #include <xrpl/telemetry/SpanNames.h>
 
+#include <string_view>
+
 namespace xrpl::telemetry::consensus::span {
 
 // ===== Span name segments ====================================================
@@ -169,11 +177,29 @@ inline constexpr auto previousProposers = makeStr("previous_proposers");
 inline constexpr auto previousRoundTimeMs = makeStr("previous_round_time_ms");
 inline constexpr auto previousLedgerSeq = makeStr("previous_ledger_seq");
 inline constexpr auto closeTimeResolutionMs = makeStr("close_time_resolution_ms");
+/**
+ * Open-phase start metadata (set on consensus.phase.open at creation).
+ *
+ * `start_reason` distinguishes a fresh round from a handleWrongLedger
+ * recovery, which emits a SECOND consensus.phase.open span under the same
+ * round span; without it the two are indistinguishable in a trace.
+ * `early_close_triggered` records that startRoundInternal itself forced an
+ * immediate timerEntry because peers had already closed.
+ */
+inline constexpr auto startReason = makeStr("start_reason");
+inline constexpr auto previousCloseAgree = makeStr("previous_close_agree");
+inline constexpr auto peerPositionsAtOpen = makeStr("peer_positions_at_open");
+inline constexpr auto earlyCloseTriggered = makeStr("early_close_triggered");
 /**
  * Open-phase end metadata (set on consensus.phase.open before reset).
+ *
+ * `tx_sets_acquired` counts the candidate transaction sets held when the
+ * ledger closed; a low count next to a high peer_positions_at_close points
+ * at missing tx-set fetches rather than at disagreement.
  */
 inline constexpr auto openDurationMs = makeStr("open_duration_ms");
 inline constexpr auto peerPositionsAtClose = makeStr("peer_positions_at_close");
+inline constexpr auto txSetsAcquired = makeStr("tx_sets_acquired");
 /**
  * Ledger-close inputs.
  */
@@ -182,6 +208,19 @@ inline constexpr auto txCountOpen = makeStr("tx_count_open");
  * Establish/check additional state.
  */
 inline constexpr auto proposersFinished = makeStr("proposers_finished");
+/**
+ * Establish-phase start/end metadata.
+ *
+ * `disputes_count_initial` is the dispute count carried into the establish
+ * phase from the positions held at close. It is set once, unlike
+ * `disputes_count`, which updateEstablishTracing() overwrites every
+ * iteration and so only ever reports the final value.
+ * `avalanche_state` is the terminal close-time convergence regime, set once
+ * when the establish span ends; the derived `avalanche_threshold` on
+ * consensus.update_positions cannot be inverted back to it.
+ */
+inline constexpr auto disputesCountInitial = makeStr("disputes_count_initial");
+inline constexpr auto avalancheState = makeStr("avalanche_state");
 /**
  * Accept/apply enrichment.
  */
@@ -291,6 +330,47 @@ inline constexpr auto unchanged = makeStr("unchanged");
 inline constexpr auto phaseOpen = makeStr("open");
 inline constexpr auto phaseEstablish = makeStr("establish");
 inline constexpr auto phaseAccepted = makeStr("accepted");
+// start_reason values (how startRoundInternal was entered).
+inline constexpr auto startInitial = makeStr("initial");
+inline constexpr auto startRecovered = makeStr("recovered");
+// avalanche_state values, one per ConsensusParms::AvalancheState enumerator.
+inline constexpr auto avalancheInit = makeStr("init");
+inline constexpr auto avalancheMid = makeStr("mid");
+inline constexpr auto avalancheLate = makeStr("late");
+inline constexpr auto avalancheStuck = makeStr("stuck");
 }  // namespace val
 
+/**
+ * Map a close-time avalanche state to its `avalanche_state` label.
+ *
+ * The regime escalates Init -> Mid -> Late -> Stuck as a round takes longer
+ * to converge, raising the close-time agreement threshold at each step. The
+ * label is recorded once, when the establish span ends, so the terminal
+ * regime of the round is queryable.
+ *
+ * @param state The state held by Consensus::closeTimeAvalancheState_.
+ * @return The wire label; one of val::avalanche*.
+ *
+ * @note constexpr so the call site costs nothing. Every enumerator is
+ * handled explicitly; there is no default arm, so adding an enumerator to
+ * ConsensusParms::AvalancheState makes the switch fall through to the
+ * unreachable return rather than silently mislabelling a new regime.
+ */
+[[nodiscard]] constexpr std::string_view
+avalancheStateLabel(ConsensusParms::AvalancheState const state)
+{
+    switch (state)
+    {
+        case ConsensusParms::AvalancheState::Init:
+            return val::avalancheInit;
+        case ConsensusParms::AvalancheState::Mid:
+            return val::avalancheMid;
+        case ConsensusParms::AvalancheState::Late:
+            return val::avalancheLate;
+        case ConsensusParms::AvalancheState::Stuck:
+            return val::avalancheStuck;
+    }
+    return val::avalancheInit;
+}
+
 }  // namespace xrpl::telemetry::consensus::span
diff --git a/src/tests/libxrpl/telemetry/ConsensusSpanNames.cpp b/src/tests/libxrpl/telemetry/ConsensusSpanNames.cpp
new file mode 100644
index 0000000000..ea8384038b
--- /dev/null
+++ b/src/tests/libxrpl/telemetry/ConsensusSpanNames.cpp
@@ -0,0 +1,90 @@
+#include <xrpl/consensus/ConsensusSpanNames.h>
+
+#include <xrpl/consensus/ConsensusParms.h>
+
+#include <gtest/gtest.h>
+
+#include <string_view>
+
+/**
+ * Contract tests for the consensus phase-span attribute constants.
+ *
+ *  The keys in ConsensusSpanNames.h are the single source of truth (L1) that
+ *  `.github/scripts/otel-naming/check_otel_naming.py` derives its valid key
+ *  set from, and that the collector's spanmetrics dimensions, the Tempo span
+ *  filters and the Grafana dashboards query by literal string. A silent rename
+ *  here compiles cleanly but blanks panels, so these tests pin the wire values.
+ *  They need no telemetry runtime and run in every build.
+ *
+ *  Scope: the attributes carried by `consensus.phase.open` and
+ *  `consensus.establish`. The round-level attrs are covered by the
+ *  pre-existing key set and are deliberately NOT duplicated onto the phase
+ *  children (a child span does not inherit parent attributes, but copying
+ *  `ledger_seq` down would store the same value twice per trace).
+ */
+
+using namespace xrpl::telemetry::consensus::span;
+
+TEST(ConsensusSpanNames, phase_open_start_attribute_keys)
+{
+    // Set once when the open-phase span is created in startRoundInternal().
+    EXPECT_EQ(std::string_view(attr::startReason), "start_reason");
+    EXPECT_EQ(std::string_view(attr::previousCloseAgree), "previous_close_agree");
+    EXPECT_EQ(std::string_view(attr::peerPositionsAtOpen), "peer_positions_at_open");
+    EXPECT_EQ(std::string_view(attr::earlyCloseTriggered), "early_close_triggered");
+}
+
+TEST(ConsensusSpanNames, phase_open_end_attribute_keys)
+{
+    // Existing end-of-phase metadata, pinned alongside the new key so a rename
+    // of either shows up here.
+    EXPECT_EQ(std::string_view(attr::openDurationMs), "open_duration_ms");
+    EXPECT_EQ(std::string_view(attr::peerPositionsAtClose), "peer_positions_at_close");
+    EXPECT_EQ(std::string_view(attr::txSetsAcquired), "tx_sets_acquired");
+}
+
+TEST(ConsensusSpanNames, establish_attribute_keys)
+{
+    EXPECT_EQ(std::string_view(attr::disputesCountInitial), "disputes_count_initial");
+    EXPECT_EQ(std::string_view(attr::avalancheState), "avalanche_state");
+}
+
+TEST(ConsensusSpanNames, start_reason_values_are_the_two_entry_paths)
+{
+    // startRoundInternal() is entered fresh, or re-entered by handleWrongLedger
+    // after acquiring the correct prior ledger. A round that recovers emits a
+    // SECOND consensus.phase.open span, so the label is what tells them apart.
+    EXPECT_EQ(std::string_view(val::startInitial), "initial");
+    EXPECT_EQ(std::string_view(val::startRecovered), "recovered");
+}
+
+TEST(ConsensusSpanNames, avalanche_state_values_match_the_parms_enum)
+{
+    EXPECT_EQ(std::string_view(val::avalancheInit), "init");
+    EXPECT_EQ(std::string_view(val::avalancheMid), "mid");
+    EXPECT_EQ(std::string_view(val::avalancheLate), "late");
+    EXPECT_EQ(std::string_view(val::avalancheStuck), "stuck");
+}
+
+TEST(ConsensusSpanNames, avalanche_state_label_maps_every_enum_state)
+{
+    // A missed branch here would silently report the wrong convergence regime
+    // for the round, so every enumerator is asserted explicitly rather than
+    // round-tripped through a table.
+    using AvalancheState = xrpl::ConsensusParms::AvalancheState;
+
+    EXPECT_EQ(avalancheStateLabel(AvalancheState::Init), "init");
+    EXPECT_EQ(avalancheStateLabel(AvalancheState::Mid), "mid");
+    EXPECT_EQ(avalancheStateLabel(AvalancheState::Late), "late");
+    EXPECT_EQ(avalancheStateLabel(AvalancheState::Stuck), "stuck");
+}
+
+TEST(ConsensusSpanNames, avalanche_state_label_is_usable_at_compile_time)
+{
+    // The mapping is consteval-safe so the label costs nothing at the call
+    // site in endEstablishTracing().
+    static_assert(
+        avalancheStateLabel(xrpl::ConsensusParms::AvalancheState::Stuck) == "stuck",
+        "avalancheStateLabel must be constexpr-evaluable");
+    SUCCEED();
+}

From 2ae47c66aaa82d1b2155b6c34ea2cf5c86140881 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 20:09:04 +0100
Subject: [PATCH 08/38] fix(telemetry): assert ios_latency and correct the
 rpc_size rename attribution

---
 docker/telemetry/workload/expected_metrics.json | 13 ++++++++++---
 1 file changed, 10 insertions(+), 3 deletions(-)

diff --git a/docker/telemetry/workload/expected_metrics.json b/docker/telemetry/workload/expected_metrics.json
index 7049b49593..3731a1f5c6 100644
--- a/docker/telemetry/workload/expected_metrics.json
+++ b/docker/telemetry/workload/expected_metrics.json
@@ -46,6 +46,14 @@
     "description": "beast::insight counters exported via OTLP/HTTP. The OTel Prometheus exporter appends _total to monotonic counters.",
     "metrics": ["rpc_requests_total", "ledger_fetches_total"]
   },
+  "io_latency": {
+    "description": "io_context scheduling-latency histogram — a beast::insight Event exported over OTLP (Application.cpp:515, makeEvent('ios_latency') with the default millisecond unit). It is the one beast Event the node itself guarantees, which is why it is asserted while rpc_time_milliseconds and the jobq_* pairs sit in not_asserted: the sampler starts on the unconditional startup path (Application.cpp:1697, outside the 'if (withTimers)' guard), and its handler always emits the first sample whatever its value (Application.cpp:185-189, 'firstSample_.exchange(false) || lastSample >= 10ms', with a comment stating the point is to register the metric downstream). The OTel histogram is cumulative, so that one notify creates series that persist for the rest of the run. Named with the _bucket/_count/_sum suffixes the Prometheus exporter emits for a histogram — no bare ios_latency_milliseconds series exists, the same convention rpc_method_us and span_duration_milliseconds follow, and all 6 panels that query it use ios_latency_milliseconds_bucket. Only existence is asserted: past the first sample the handler reports only latencies >= 10 ms, so neither the sample count nor the value is predictable.",
+    "metrics": [
+      "ios_latency_milliseconds_bucket",
+      "ios_latency_milliseconds_count",
+      "ios_latency_milliseconds_sum"
+    ]
+  },
   "overlay_traffic": {
     "description": "Overlay traffic metrics (subset — full list has 45+ categories).",
     "metrics": [
@@ -177,10 +185,9 @@
       "getobject_lookup_us": "GetObjectMetricNames.h:95, PeerImp.cpp:2929. Same lazy-creation and same inbound-request gate as getobject_request_objects.",
       "getobject_lookups_total": "GetObjectMetricNames.h:100, PeerImp.cpp:2949/:2956. Same gate.",
       "getobject_charge": "GetObjectMetricNames.h:105, PeerImp.cpp:2931. Same gate.",
-      "rpc_size_bytes": "ServerHandler.cpp:191, group('rpc')->makeEvent('size', Unit::Bytes). The OTLP Prometheus exporter derives the metric-name suffix from the declared unit, so a byte unit yields rpc_size_bytes; before 24094e427b the event declared no unit and exported as rpc_size_milliseconds on the millisecond bucket ladder. Neither name was ever recorded here, so the harness could confirm neither the rename nor a regression back onto that ladder. Notified from ServerHandler::processRequest:1133, the HTTP JSON-RPC path — it computes an HTTP status and appends a trailing newline — and the load generators are WebSocket-only, the same gate regression-metrics.json:4 records for rpc.process, so only the harness's handful of HTTP health polls reach it. Real coverage needs an HTTP JSON-RPC phase in rpc_load_generator.py; that is a workload change rather than a harness correction, and is deliberately out of scope here.",
+      "rpc_size_bytes": "ServerHandler.cpp:191, group('rpc')->makeEvent('size', Unit::Bytes). The OTLP Prometheus exporter derives the metric-name suffix from the declared unit, so a byte unit yields rpc_size_bytes. The Unit::Bytes declaration itself landed earlier, in 76c9051203; what 24094e427b changed was the exporter finally consuming it, replacing a hardcoded CreateDoubleHistogram(name, 'Duration in ms', 'ms') with otelUnitDescription(unit)/otelUnitCode(unit), and that is what renamed the series off rpc_size_milliseconds and the millisecond bucket ladder. Neither name was ever recorded here, so the harness could confirm neither the rename nor a regression back onto that ladder. Notified from ServerHandler::processRequest:1133, the HTTP JSON-RPC path — it computes an HTTP status and appends a trailing newline — and the load generators are WebSocket-only, the same gate regression-metrics.json:4 records for rpc.process, so only the harness's handful of HTTP health polls reach it. Real coverage needs an HTTP JSON-RPC phase in rpc_load_generator.py; that is a workload change rather than a harness correction, and is deliberately out of scope here.",
       "rpc_time_milliseconds": "ServerHandler.cpp:192, group('rpc')->makeEvent('time') with the default millisecond unit. Notified from ServerHandler::processRequest:1129, the same HTTP JSON-RPC call site as rpc_size_bytes and behind the same WebSocket-only gate.",
-      "ios_latency_milliseconds": "Application.cpp:515, makeEvent('ios_latency') with the default millisecond unit. Emitted and dashboarded, but it measures io-service scheduling delay rather than anything the workload drives, so no workload phase guarantees it.",
-      "jobq_*_milliseconds, jobq_*_q_milliseconds": "Created per job type in JobTypeData.h:97-98 from info.name() and info.name() + kSuffixQueued ('_q'), so the exported names are jobq_<jobtype>_milliseconds and jobq_<jobtype>_q_milliseconds with the job type lowercased by formatName. Which job types appear depends on which jobs a run happens to schedule, so no individual name is guaranteed. They are also rounded up to a whole millisecond at source (Event.h:47-51 applies ceil to a millisecond value type), which is why 6e2b2da772 moved the ledger-data-sync q-wait panels off jobq_<jobtype>_q_milliseconds_bucket onto job_queued_us_bucket — they are poor assertion targets regardless."
+      "jobq_*_milliseconds, jobq_*_q_milliseconds": "This key is a pattern rather than a literal metric name — unlike every other entry in this map it stands for a whole family, one pair per job type. Created per job type in JobTypeData.h:97-98 from info.name() and info.name() + kSuffixQueued ('_q'), so the exported names are jobq_<jobtype>_milliseconds and jobq_<jobtype>_q_milliseconds with the job type lowercased by formatName. Which job types appear depends on which jobs a run happens to schedule, so no individual name is guaranteed. They are also rounded up to a whole millisecond at source (Event.h:47-51 applies ceil to a millisecond value type), which is why 6e2b2da772 moved the ledger-data-sync q-wait panels off jobq_<jobtype>_q_milliseconds_bucket onto job_queued_us_bucket — they are poor assertion targets regardless."
     }
   },
   "grafana_dashboards": {

From 6eaf7316e5a146c3143e04f3c1add45369c70f3c Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 20:13:39 +0100
Subject: [PATCH 09/38] feat(telemetry): record the ledger close reason on
 consensus.phase.open

The open phase ended for one of four distinct reasons, but
shouldCloseLedger() collapsed them into a bool, so a trace could say when a
phase ended and never why. "The network closed without us" and "nothing was
waiting" are the same span today.

Add whyCloseLedger(), which holds the decision and returns
LedgerCloseReason. shouldCloseLedger() keeps its exact signature and becomes
a one-line delegation, so its callers and unit tests are untouched and the
branch logic is not duplicated. phaseOpen() calls whyCloseLedger() directly;
both emit the same journal and CLOG output, so only one is called.

New attributes on consensus.phase.open, both set once on the closing tick:

  close_reason         anomaly | others_closed | idle | normal
  proposers_validated  trusted peers that had already validated the prior
                       ledger, reusing the value the decision was made on

Absent on the simulate() close path, which bypasses the decision rather than
having a reason invented for it.

Skipped has_open_transactions: hasOpenTransactions() is
!getOpenLedger().empty(), which is false on a quiet network for most of a
round, and close_reason=idle already implies it. The sibling
consensus.ledger_close span carries tx_count_open, which is the same fact
with a count instead of a boolean.

shouldCloseLedger() now has no production caller; it stays exported so the
public API and its tests are unchanged.

Tests pin every input vector from should_close_ledger to its literal reason,
including that the anomaly check outranks others-closed, and cover the
inclusive idle boundary either side by one millisecond.
---
 include/xrpl/consensus/Consensus.h            | 99 ++++++++++++-------
 include/xrpl/consensus/ConsensusSpanNames.h   | 75 +++++++++-----
 include/xrpl/consensus/ConsensusTypes.h       | 35 +++++++
 src/libxrpl/consensus/Consensus.cpp           | 46 +++++++--
 src/tests/libxrpl/consensus/Consensus.cpp     | 78 +++++++++++++++
 .../libxrpl/telemetry/ConsensusSpanNames.cpp  | 33 +++++++
 6 files changed, 297 insertions(+), 69 deletions(-)

diff --git a/include/xrpl/consensus/Consensus.h b/include/xrpl/consensus/Consensus.h
index 207534012b..5922801e1c 100644
--- a/include/xrpl/consensus/Consensus.h
+++ b/include/xrpl/consensus/Consensus.h
@@ -31,12 +31,37 @@
 
 namespace xrpl {
 
+/**
+ * Determines why the current ledger should close at this time.
+ *
+ * Holds the close decision that shouldCloseLedger() reduces to a bool. Call
+ * this one when the deciding branch matters. Both log identically, so call
+ * one or the other, never both. Parameters match shouldCloseLedger().
+ *
+ * @return The deciding branch, or KeepOpen if no close condition is met.
+ */
+LedgerCloseReason
+whyCloseLedger(
+    bool anyTransactions,
+    std::size_t prevProposers,
+    std::size_t proposersClosed,
+    std::size_t proposersValidated,
+    std::chrono::milliseconds prevRoundTime,
+    std::chrono::milliseconds timeSincePrevClose,
+    std::chrono::milliseconds openTime,
+    std::chrono::milliseconds idleInterval,
+    ConsensusParms const& parms,
+    beast::Journal j,
+    std::unique_ptr<std::stringstream> const& clog = {});
+
 /**
  * Determines whether the current ledger should close at this time.
  *
  * This function should be called when a ledger is open and there is no close
  * in progress, or when a transaction is received and no close is in progress.
  *
+ * Equivalent to `whyCloseLedger(...) != LedgerCloseReason::KeepOpen`.
+ *
  * @param anyTransactions indicates whether any transactions have been received
  * @param prevProposers proposers in the last closing
  * @param proposersClosed proposers who have currently closed this ledger
@@ -790,17 +815,13 @@ Consensus<Adaptor>::startRoundInternal(
     if (*openSpan_)
     {
         namespace cs = telemetry::consensus::span;
-        // Which entry path created this span. A handleWrongLedger recovery
-        // re-enters startRoundInternal and emplaces a SECOND phase.open span
-        // under the same round span, so without this the two are
-        // indistinguishable in a trace.
+        // A recovery emplaces a SECOND phase.open span under the same round,
+        // so this is what tells the two apart.
         openSpan_->setAttribute(
             cs::attr::startReason,
             reason == StartRoundReason::Recovered ? std::string_view{cs::val::startRecovered}
                                                   : std::string_view{cs::val::startInitial});
-        // Read from the parameter, not previousLedger_, which is not assigned
-        // until further down this function. Feeds the sinceClose calculation
-        // in phaseOpen().
+        // From the parameter: previousLedger_ is not assigned until below.
         openSpan_->setAttribute(cs::attr::previousCloseAgree, prevLedger.closeAgree());
     }
     // On the Recovered path, fire phase.open here because startRoundTracing
@@ -840,18 +861,14 @@ Consensus<Adaptor>::startRoundInternal(
     // We may be falling behind, don't wait for the timer
     // consider closing the ledger immediately
     bool const closeImmediately = currPeerPositions_.size() > (prevProposers_ / 2);
-    // Annotate before the timerEntry() below, which can reach closeLedger()
-    // and end this span.
+    // Annotate before the timerEntry() below, which can end this span.
     if (openSpan_ && *openSpan_)
     {
         namespace cs = telemetry::consensus::span;
-        // Positions already in hand once playbackProposals() has replayed the
-        // buffered ones — the head start this round began with. Pairs with
-        // peer_positions_at_close on the same span.
+        // Head start after playbackProposals() replayed the buffered
+        // positions. Pairs with peer_positions_at_close.
         openSpan_->setAttribute(
             cs::attr::peerPositionsAtOpen, static_cast<int64_t>(currPeerPositions_.size()));
-        // Whether the round skipped waiting for the timer because enough peers
-        // had already closed.
         openSpan_->setAttribute(cs::attr::earlyCloseTriggered, closeImmediately);
     }
     if (closeImmediately)
@@ -1344,20 +1361,32 @@ Consensus<Adaptor>::phaseOpen(std::unique_ptr<std::stringstream> const& clog)
                << ", previous ledger close time resolution: "
                << previousLedger_.closeTimeResolution().count() << "ms. ";
 
-    // Decide if we should close the ledger
-    if (shouldCloseLedger(
-            anyTransactions,
-            prevProposers_,
-            proposersClosed,
-            proposersValidated,
-            prevRoundTime_,
-            sinceClose,
-            openTime_.read(),
-            idleInterval,
-            adaptor_.parms(),
-            j_,
-            clog))
+    // Decide if we should close the ledger. whyCloseLedger() so the deciding
+    // branch can be recorded; it logs the same, so only one is called.
+    LedgerCloseReason const closeReason = whyCloseLedger(
+        anyTransactions,
+        prevProposers_,
+        proposersClosed,
+        proposersValidated,
+        prevRoundTime_,
+        sinceClose,
+        openTime_.read(),
+        idleInterval,
+        adaptor_.parms(),
+        j_,
+        clog);
+    if (closeReason != LedgerCloseReason::KeepOpen)
     {
+        // Annotate before closeLedger() ends the span. Set once: the phase
+        // moves to Establish, so phaseOpen() is not entered again this round.
+        // Absent on the simulate() path, which bypasses this decision.
+        if (openSpan_ && *openSpan_)
+        {
+            namespace cs = telemetry::consensus::span;
+            openSpan_->setAttribute(cs::attr::closeReason, cs::closeReasonLabel(closeReason));
+            openSpan_->setAttribute(
+                cs::attr::proposersValidated, static_cast<int64_t>(proposersValidated));
+        }
         CLOG(clog) << "closing ledger. ";
         closeLedger(clog);
     }
@@ -1595,10 +1624,8 @@ Consensus<Adaptor>::closeLedger(std::unique_ptr<std::stringstream> const& clog)
             cs::attr::openDurationMs, static_cast<int64_t>(openTime_.read().count()));
         openSpan_->setAttribute(
             cs::attr::peerPositionsAtClose, static_cast<int64_t>(currPeerPositions_.size()));
-        // Candidate transaction sets held at close. Read before our own
-        // position is added below, so this counts only what peers shared. A
-        // low count against a high peer_positions_at_close means tx-set
-        // fetches did not land, not that peers disagreed.
+        // Read before our own position is added below, so this counts only
+        // what peers shared.
         openSpan_->setAttribute(cs::attr::txSetsAcquired, static_cast<int64_t>(acquired_.size()));
     }
     openSpan_.reset();
@@ -2153,10 +2180,8 @@ Consensus<Adaptor>::startEstablishTracing()
     if (*establishSpan_)
     {
         establishSpanContext_ = establishSpan_->spanContext();
-        // Disputes carried into the phase from the positions held at close.
-        // Set once (this function early-returns while the span is live),
-        // unlike disputes_count, which updateEstablishTracing() overwrites
-        // every iteration and so only ever reports the final value.
+        // Disputes carried in from the positions held at close. Set once:
+        // this function early-returns while the span is live.
         if (result_)
         {
             establishSpan_->setAttribute(
@@ -2188,9 +2213,7 @@ template <class Adaptor>
 void
 Consensus<Adaptor>::endEstablishTracing()
 {
-    // Terminal close-time convergence regime, recorded once before the span
-    // ends. The derived avalanche_threshold on consensus.update_positions is
-    // a weight, not the state, and cannot be inverted back to it.
+    // Terminal convergence regime, recorded once before the span ends.
     if (establishSpan_ && *establishSpan_)
     {
         establishSpan_->setAttribute(
diff --git a/include/xrpl/consensus/ConsensusSpanNames.h b/include/xrpl/consensus/ConsensusSpanNames.h
index dfc492d962..cb7d7a8c85 100644
--- a/include/xrpl/consensus/ConsensusSpanNames.h
+++ b/include/xrpl/consensus/ConsensusSpanNames.h
@@ -84,6 +84,7 @@
  */
 
 #include <xrpl/consensus/ConsensusParms.h>
+#include <xrpl/consensus/ConsensusTypes.h>
 #include <xrpl/telemetry/SpanNames.h>
 
 #include <string_view>
@@ -180,11 +181,8 @@ inline constexpr auto closeTimeResolutionMs = makeStr("close_time_resolution_ms"
 /**
  * Open-phase start metadata (set on consensus.phase.open at creation).
  *
- * `start_reason` distinguishes a fresh round from a handleWrongLedger
- * recovery, which emits a SECOND consensus.phase.open span under the same
- * round span; without it the two are indistinguishable in a trace.
- * `early_close_triggered` records that startRoundInternal itself forced an
- * immediate timerEntry because peers had already closed.
+ * A handleWrongLedger recovery emits a SECOND phase.open span under the same
+ * round, so `start_reason` is what tells the two apart.
  */
 inline constexpr auto startReason = makeStr("start_reason");
 inline constexpr auto previousCloseAgree = makeStr("previous_close_agree");
@@ -193,13 +191,16 @@ inline constexpr auto earlyCloseTriggered = makeStr("early_close_triggered");
 /**
  * Open-phase end metadata (set on consensus.phase.open before reset).
  *
- * `tx_sets_acquired` counts the candidate transaction sets held when the
- * ledger closed; a low count next to a high peer_positions_at_close points
- * at missing tx-set fetches rather than at disagreement.
+ * A low `tx_sets_acquired` next to a high peer_positions_at_close points at
+ * missing tx-set fetches rather than at disagreement. `close_reason` plus
+ * `proposers_validated` separate "the network moved on without us" from "the
+ * network was quiet".
  */
 inline constexpr auto openDurationMs = makeStr("open_duration_ms");
 inline constexpr auto peerPositionsAtClose = makeStr("peer_positions_at_close");
 inline constexpr auto txSetsAcquired = makeStr("tx_sets_acquired");
+inline constexpr auto closeReason = makeStr("close_reason");
+inline constexpr auto proposersValidated = makeStr("proposers_validated");
 /**
  * Ledger-close inputs.
  */
@@ -211,13 +212,10 @@ inline constexpr auto proposersFinished = makeStr("proposers_finished");
 /**
  * Establish-phase start/end metadata.
  *
- * `disputes_count_initial` is the dispute count carried into the establish
- * phase from the positions held at close. It is set once, unlike
- * `disputes_count`, which updateEstablishTracing() overwrites every
- * iteration and so only ever reports the final value.
- * `avalanche_state` is the terminal close-time convergence regime, set once
- * when the establish span ends; the derived `avalanche_threshold` on
- * consensus.update_positions cannot be inverted back to it.
+ * Both are set once, unlike `disputes_count`, which
+ * updateEstablishTracing() overwrites every iteration.
+ * `avalanche_state` is the terminal regime; the derived
+ * `avalanche_threshold` cannot be inverted back to it.
  */
 inline constexpr auto disputesCountInitial = makeStr("disputes_count_initial");
 inline constexpr auto avalancheState = makeStr("avalanche_state");
@@ -338,23 +336,26 @@ inline constexpr auto avalancheInit = makeStr("init");
 inline constexpr auto avalancheMid = makeStr("mid");
 inline constexpr auto avalancheLate = makeStr("late");
 inline constexpr auto avalancheStuck = makeStr("stuck");
+// close_reason values, one per LedgerCloseReason enumerator. keep_open is
+// never emitted: the attribute is only set on the path that closes.
+inline constexpr auto closeKeepOpen = makeStr("keep_open");
+inline constexpr auto closeAnomaly = makeStr("anomaly");
+inline constexpr auto closeOthersClosed = makeStr("others_closed");
+inline constexpr auto closeIdle = makeStr("idle");
+inline constexpr auto closeNormal = makeStr("normal");
 }  // namespace val
 
 /**
  * Map a close-time avalanche state to its `avalanche_state` label.
  *
- * The regime escalates Init -> Mid -> Late -> Stuck as a round takes longer
- * to converge, raising the close-time agreement threshold at each step. The
- * label is recorded once, when the establish span ends, so the terminal
- * regime of the round is queryable.
+ * The regime escalates Init -> Mid -> Late -> Stuck, raising the close-time
+ * agreement threshold at each step.
  *
  * @param state The state held by Consensus::closeTimeAvalancheState_.
  * @return The wire label; one of val::avalanche*.
  *
- * @note constexpr so the call site costs nothing. Every enumerator is
- * handled explicitly; there is no default arm, so adding an enumerator to
- * ConsensusParms::AvalancheState makes the switch fall through to the
- * unreachable return rather than silently mislabelling a new regime.
+ * @note No default arm, so a new enumerator is a compiler warning rather than
+ * a silently reused label.
  */
 [[nodiscard]] constexpr std::string_view
 avalancheStateLabel(ConsensusParms::AvalancheState const state)
@@ -373,4 +374,32 @@ avalancheStateLabel(ConsensusParms::AvalancheState const state)
     return val::avalancheInit;
 }
 
+/**
+ * Map a ledger-close decision to its `close_reason` label.
+ *
+ * @param reason The value returned by whyCloseLedger().
+ * @return The wire label; one of val::close*.
+ *
+ * @note No default arm, so a new enumerator is a compiler warning rather than
+ * a silently reused label. `keep_open` is mapped but never emitted.
+ */
+[[nodiscard]] constexpr std::string_view
+closeReasonLabel(LedgerCloseReason const reason)
+{
+    switch (reason)
+    {
+        case LedgerCloseReason::KeepOpen:
+            return val::closeKeepOpen;
+        case LedgerCloseReason::Anomaly:
+            return val::closeAnomaly;
+        case LedgerCloseReason::OthersClosed:
+            return val::closeOthersClosed;
+        case LedgerCloseReason::Idle:
+            return val::closeIdle;
+        case LedgerCloseReason::Normal:
+            return val::closeNormal;
+    }
+    return val::closeKeepOpen;
+}
+
 }  // namespace xrpl::telemetry::consensus::span
diff --git a/include/xrpl/consensus/ConsensusTypes.h b/include/xrpl/consensus/ConsensusTypes.h
index 11231de40c..74b25e4fa7 100644
--- a/include/xrpl/consensus/ConsensusTypes.h
+++ b/include/xrpl/consensus/ConsensusTypes.h
@@ -154,6 +154,41 @@ to_string(ConsensusPhase p)
     }
 }
 
+/**
+ * Why the open ledger should, or should not, close right now.
+ *
+ * Returned by whyCloseLedger(); shouldCloseLedger() reduces it to a bool.
+ *
+ * @note KeepOpen is not a close reason. Compare against it rather than
+ * treating the enum as a flag.
+ */
+enum class LedgerCloseReason : std::uint8_t {
+    /**
+     * No close condition is met yet.
+     */
+    KeepOpen,
+
+    /**
+     * Timings out of range; close defensively.
+     */
+    Anomaly,
+
+    /**
+     * More than half the network has closed or validated.
+     */
+    OthersClosed,
+
+    /**
+     * Nothing waiting and the idle interval elapsed.
+     */
+    Idle,
+
+    /**
+     * Transactions waiting and both minimum-open floors met.
+     */
+    Normal,
+};
+
 /**
  * Measures the duration of phases of consensus
  */
diff --git a/src/libxrpl/consensus/Consensus.cpp b/src/libxrpl/consensus/Consensus.cpp
index 6f398cf66c..50d9a82d5f 100644
--- a/src/libxrpl/consensus/Consensus.cpp
+++ b/src/libxrpl/consensus/Consensus.cpp
@@ -13,8 +13,8 @@
 
 namespace xrpl {
 
-bool
-shouldCloseLedger(
+LedgerCloseReason
+whyCloseLedger(
     bool anyTransactions,
     std::size_t prevProposers,
     std::size_t proposersClosed,
@@ -47,7 +47,7 @@ shouldCloseLedger(
 
         JLOG(j.warn()) << ss.str();
         CLOG(clog) << "closing ledger: " << ss.str() << ". ";
-        return true;
+        return LedgerCloseReason::Anomaly;
     }
 
     if ((proposersClosed + proposersValidated) > (prevProposers / 2))
@@ -55,14 +55,16 @@ shouldCloseLedger(
         // If more than half of the network has closed, we close
         JLOG(j.trace()) << "Others have closed";
         CLOG(clog) << "closing ledger because enough others have already. ";
-        return true;
+        return LedgerCloseReason::OthersClosed;
     }
 
     if (!anyTransactions)
     {
         // Only close at the end of the idle interval
         CLOG(clog) << "no transactions, returning. ";
-        return timeSincePrevClose >= idleInterval;  // normal idle
+        return timeSincePrevClose >= idleInterval  // normal idle
+            ? LedgerCloseReason::Idle
+            : LedgerCloseReason::KeepOpen;
     }
 
     // Preserve minimum ledger open time
@@ -70,7 +72,7 @@ shouldCloseLedger(
     {
         JLOG(j.debug()) << "Must wait minimum time before closing";
         CLOG(clog) << "not closing because under ledgerMIN_CLOSE. ";
-        return false;
+        return LedgerCloseReason::KeepOpen;
     }
 
     // Don't let this ledger close more than twice as fast as the previous
@@ -80,12 +82,40 @@ shouldCloseLedger(
     {
         JLOG(j.debug()) << "Ledger has not been open long enough";
         CLOG(clog) << "not closing because not open long enough. ";
-        return false;
+        return LedgerCloseReason::KeepOpen;
     }
 
     // Close the ledger
     CLOG(clog) << "no reason to not close. ";
-    return true;
+    return LedgerCloseReason::Normal;
+}
+
+bool
+shouldCloseLedger(
+    bool anyTransactions,
+    std::size_t prevProposers,
+    std::size_t proposersClosed,
+    std::size_t proposersValidated,
+    std::chrono::milliseconds prevRoundTime,
+    std::chrono::milliseconds timeSincePrevClose,
+    std::chrono::milliseconds openTime,
+    std::chrono::milliseconds idleInterval,
+    ConsensusParms const& parms,
+    beast::Journal j,
+    std::unique_ptr<std::stringstream> const& clog)
+{
+    return whyCloseLedger(
+               anyTransactions,
+               prevProposers,
+               proposersClosed,
+               proposersValidated,
+               prevRoundTime,
+               timeSincePrevClose,
+               openTime,
+               idleInterval,
+               parms,
+               j,
+               clog) != LedgerCloseReason::KeepOpen;
 }
 
 bool
diff --git a/src/tests/libxrpl/consensus/Consensus.cpp b/src/tests/libxrpl/consensus/Consensus.cpp
index d303d28e89..c0298b64a7 100644
--- a/src/tests/libxrpl/consensus/Consensus.cpp
+++ b/src/tests/libxrpl/consensus/Consensus.cpp
@@ -66,6 +66,33 @@ shouldCloseLedger(
         clog);
 }
 
+LedgerCloseReason
+whyCloseLedger(
+    bool anyTransactions,
+    std::size_t prevProposers,
+    std::size_t proposersClosed,
+    std::size_t proposersValidated,
+    std::chrono::milliseconds prevRoundTime,
+    std::chrono::milliseconds timeSincePrevClose,
+    std::chrono::milliseconds openTime,
+    std::chrono::milliseconds idleInterval,
+    ConsensusParms const& parms,
+    std::unique_ptr<std::stringstream> const& clog = {})
+{
+    return xrpl::whyCloseLedger(
+        anyTransactions,
+        prevProposers,
+        proposersClosed,
+        proposersValidated,
+        prevRoundTime,
+        timeSincePrevClose,
+        openTime,
+        idleInterval,
+        parms,
+        journal(),
+        clog);
+}
+
 ConsensusState
 checkConsensus(
     std::size_t prevProposers,
@@ -219,6 +246,57 @@ TEST(ConsensusTest, should_close_ledger)
     EXPECT_TRUE(shouldCloseLedger(true, 10, 0, 0, 10s, 10s, 10s, 10s, p));
 }
 
+TEST(ConsensusTest, why_close_ledger_reports_the_deciding_branch)
+{
+    using namespace std::chrono_literals;
+    SCOPED_TRACE("why close ledger");
+
+    // Same input vectors as should_close_ledger above, pinned to the reason
+    // rather than the bool, so a branch that starts returning the wrong
+    // reason is caught even though the close/no-close verdict is unchanged.
+    ConsensusParms const p{};
+
+    // Bizarre times forcibly close. These vectors ALSO satisfy the
+    // others-closed condition (8 > 10/2), so they pin the precedence: the
+    // anomaly check runs first.
+    EXPECT_EQ(whyCloseLedger(true, 10, 10, 10, -10s, 10s, 1s, 1s, p), LedgerCloseReason::Anomaly);
+    EXPECT_EQ(whyCloseLedger(true, 10, 10, 10, 100h, 10s, 1s, 1s, p), LedgerCloseReason::Anomaly);
+    EXPECT_EQ(whyCloseLedger(true, 10, 10, 10, 10s, 100h, 1s, 1s, p), LedgerCloseReason::Anomaly);
+
+    // Rest of network has closed: 3 closed + 5 validated > 10/2.
+    EXPECT_EQ(
+        whyCloseLedger(true, 10, 3, 5, 10s, 10s, 10s, 10s, p), LedgerCloseReason::OthersClosed);
+
+    // No transactions: keep open until the idle interval elapses, then close
+    // as idle rather than as a normal close.
+    EXPECT_EQ(whyCloseLedger(false, 10, 0, 0, 1s, 1s, 1s, 10s, p), LedgerCloseReason::KeepOpen);
+    EXPECT_EQ(whyCloseLedger(false, 10, 0, 0, 1s, 10s, 1s, 10s, p), LedgerCloseReason::Idle);
+
+    // Transactions present, but under ledgerMinClose (2s).
+    EXPECT_EQ(whyCloseLedger(true, 10, 0, 0, 10s, 10s, 1s, 10s, p), LedgerCloseReason::KeepOpen);
+
+    // Past ledgerMinClose but under prevRoundTime/2 (5s), so still too fast.
+    EXPECT_EQ(whyCloseLedger(true, 10, 0, 0, 10s, 10s, 3s, 10s, p), LedgerCloseReason::KeepOpen);
+
+    // Both minimum-open constraints satisfied.
+    EXPECT_EQ(whyCloseLedger(true, 10, 0, 0, 10s, 10s, 10s, 10s, p), LedgerCloseReason::Normal);
+}
+
+TEST(ConsensusTest, why_close_ledger_idle_boundary_is_inclusive)
+{
+    using namespace std::chrono_literals;
+    SCOPED_TRACE("idle boundary");
+
+    // The idle path closes on `timeSincePrevClose >= idleInterval`. One
+    // millisecond either side of the boundary, to pin the comparison as
+    // inclusive rather than strict.
+    ConsensusParms const p{};
+
+    EXPECT_EQ(whyCloseLedger(false, 10, 0, 0, 1s, 9999ms, 1s, 10s, p), LedgerCloseReason::KeepOpen);
+    EXPECT_EQ(whyCloseLedger(false, 10, 0, 0, 1s, 10s, 1s, 10s, p), LedgerCloseReason::Idle);
+    EXPECT_EQ(whyCloseLedger(false, 10, 0, 0, 1s, 10001ms, 1s, 10s, p), LedgerCloseReason::Idle);
+}
+
 TEST(ConsensusTest, check_consensus)
 {
     using namespace std::chrono_literals;
diff --git a/src/tests/libxrpl/telemetry/ConsensusSpanNames.cpp b/src/tests/libxrpl/telemetry/ConsensusSpanNames.cpp
index ea8384038b..d371b67789 100644
--- a/src/tests/libxrpl/telemetry/ConsensusSpanNames.cpp
+++ b/src/tests/libxrpl/telemetry/ConsensusSpanNames.cpp
@@ -41,6 +41,39 @@ TEST(ConsensusSpanNames, phase_open_end_attribute_keys)
     EXPECT_EQ(std::string_view(attr::openDurationMs), "open_duration_ms");
     EXPECT_EQ(std::string_view(attr::peerPositionsAtClose), "peer_positions_at_close");
     EXPECT_EQ(std::string_view(attr::txSetsAcquired), "tx_sets_acquired");
+    EXPECT_EQ(std::string_view(attr::closeReason), "close_reason");
+    EXPECT_EQ(std::string_view(attr::proposersValidated), "proposers_validated");
+}
+
+TEST(ConsensusSpanNames, close_reason_values_are_the_close_paths)
+{
+    // One per branch of whyCloseLedger() that closes the ledger. keep_open is
+    // never emitted (the attribute is only set on the closing path) but is
+    // labelled rather than left blank so the mapping is total.
+    EXPECT_EQ(std::string_view(val::closeKeepOpen), "keep_open");
+    EXPECT_EQ(std::string_view(val::closeAnomaly), "anomaly");
+    EXPECT_EQ(std::string_view(val::closeOthersClosed), "others_closed");
+    EXPECT_EQ(std::string_view(val::closeIdle), "idle");
+    EXPECT_EQ(std::string_view(val::closeNormal), "normal");
+}
+
+TEST(ConsensusSpanNames, close_reason_label_maps_every_enum_state)
+{
+    // A missed branch would attribute a close to the wrong cause, which is the
+    // whole point of the attribute, so every enumerator is asserted.
+    EXPECT_EQ(closeReasonLabel(xrpl::LedgerCloseReason::KeepOpen), "keep_open");
+    EXPECT_EQ(closeReasonLabel(xrpl::LedgerCloseReason::Anomaly), "anomaly");
+    EXPECT_EQ(closeReasonLabel(xrpl::LedgerCloseReason::OthersClosed), "others_closed");
+    EXPECT_EQ(closeReasonLabel(xrpl::LedgerCloseReason::Idle), "idle");
+    EXPECT_EQ(closeReasonLabel(xrpl::LedgerCloseReason::Normal), "normal");
+}
+
+TEST(ConsensusSpanNames, close_reason_label_is_usable_at_compile_time)
+{
+    static_assert(
+        closeReasonLabel(xrpl::LedgerCloseReason::Idle) == "idle",
+        "closeReasonLabel must be constexpr-evaluable");
+    SUCCEED();
 }
 
 TEST(ConsensusSpanNames, establish_attribute_keys)

From 779bf0ca983753abf6d535301833762415e49746 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 20:29:36 +0100
Subject: [PATCH 10/38] docs(telemetry): drop the runbook row for a panel this
 branch removes

The dashboard inventory listed "Peer TX Receive Rate", but this branch
deletes that panel from transaction-overview.json, so an operator
following the runbook finds no such chart. The row is stale from
phase-7 onward and is still present at the tip of the chain, so
nothing downstream repairs it.

Remove the row rather than retarget it: the neighbouring
"Transaction Receive vs Suppressed" row already documents the
tx.receive rate and its panel exists on every branch, so retargeting
would have produced two rows for one chart.
---
 docs/telemetry-runbook.md | 1 -
 1 file changed, 1 deletion(-)

diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md
index da509533bc..f61dd7aa38 100644
--- a/docs/telemetry-runbook.md
+++ b/docs/telemetry-runbook.md
@@ -680,7 +680,6 @@ Ten dashboards are pre-provisioned in `docker/telemetry/grafana/dashboards/`:
 | Transaction Receive vs Suppressed  | timeseries     | `rate(span_calls_total{span_name="tx.receive"}[5m])`                                         | —                                   |
 | TX Processing Duration Heatmap     | heatmap        | `tx.process` histogram buckets                                                               | `le`                                |
 | TX Apply Duration per Ledger       | timeseries     | p95/p50 of `tx.apply`                                                                        | —                                   |
-| Peer TX Receive Rate               | timeseries     | `tx.receive` rate                                                                            | —                                   |
 | TX Apply Failed Rate               | stat           | `rate(span_calls_total{span_name="tx.transactor",stage="apply",ter_result!~"tesSUCCESS\|"})` | `stage`, `ter_result`               |
 | TxQ Accept: Applied Ratio per Node | state-timeline | applied / (applied+failed) of `span_calls_total{span_name="txq.accept_tx"}` per node         | `txq_status`, `service_instance_id` |
 

From f4718cee0871774ce9bcb55702fca9c34c692939 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 20:40:00 +0100
Subject: [PATCH 11/38] fix(telemetry): correct mistimed and ambiguous
 consensus phase attributes

Three defects found in review of the two preceding commits.

Drop disputes_count_initial. It claimed to be the dispute count carried in
from the positions held at close, but startEstablishTracing() runs a full
timer tick after closeLedger(): timerEntry() dispatches
`if (phase_ == Open) phaseOpen(); else if (phase_ == Establish)
phaseEstablish();`, and phase_ was Open on the closing tick, so the else-if
cannot run. With ledgerGRANULARITY at 1s the value absorbed up to a second of
dispute growth from peer proposals and arriving tx sets. Making it honest
needs either a member captured at close or moving span creation into
closeLedger(), so it is removed rather than shipped mislabelled.

Record close_time_avalanche_state on recovered rounds. startRoundInternal()
reset establishSpan_ inline, discarding the span before the attribute was
written, so the value was present only on rounds that reached Accepted --
survivor bias in exactly the rounds worth investigating. It now calls
endEstablishTracing(). The comment claiming this avoided "reporting a stale
regime" was wrong: closeTimeAvalancheState_ is not reset until 39 lines
later, so the value was still that span's terminal regime.

Rename avalanche_state to close_time_avalanche_state. DisputedTx carries a
second, per-transaction avalanche tracker; the bare name invited reading a
close-time-only value as the transaction one, which is the tracker that
actually escalates in a stuck round.

Also: both label helpers now fall through to "unknown" instead of a
plausible-looking regime, matching to_string(ConsensusPhase); and the header
now records that the end-of-open attributes are absent on recovered and
simulated rounds, and that tx_sets_acquired can skew either way because
handleWrongLedger clears currPeerPositions_ but not acquired_.

Tests: the minimum-open-time assertion used prevRoundTime=10s, where
openTime=1s trips the too-fast branch as well, so deleting the ledgerMinClose
check entirely left it green. Replaced with prevRoundTime=2s, which isolates
the branch. Added the others-closed boundary, which is strict and was
untested in either direction, its integer truncation for odd prevProposers,
and its precedence over the no-transactions and minimum-open branches.
---
 include/xrpl/consensus/Consensus.h            | 31 ++++--------
 include/xrpl/consensus/ConsensusSpanNames.h   | 50 ++++++++++---------
 src/tests/libxrpl/consensus/Consensus.cpp     | 31 ++++++++++--
 .../libxrpl/telemetry/ConsensusSpanNames.cpp  |  5 +-
 4 files changed, 69 insertions(+), 48 deletions(-)

diff --git a/include/xrpl/consensus/Consensus.h b/include/xrpl/consensus/Consensus.h
index 5922801e1c..577b154cbe 100644
--- a/include/xrpl/consensus/Consensus.h
+++ b/include/xrpl/consensus/Consensus.h
@@ -694,8 +694,7 @@ private:
 
     /**
      * Create the establish-phase span if not yet active.
-     *  Called on each phaseEstablish() invocation; no-op while span is live,
-     *  so the entry-state attributes it sets are written exactly once.
+     *  Called on each phaseEstablish() invocation; no-op while span is live.
      */
     void
     startEstablishTracing();
@@ -708,10 +707,9 @@ private:
     updateEstablishTracing();
 
     /**
-     * End the establish span when transitioning to the accepted phase.
-     *  Records the terminal avalanche_state before ending the span. A round
-     *  that instead loses the establish span to a wrongLedger recovery omits
-     *  the attribute rather than reporting a stale regime.
+     * End the establish span, recording its terminal regime.
+     *  Also called from startRoundInternal() on a wrongLedger recovery, so a
+     *  round that never reaches Accepted still reports the regime it reached.
      */
     void
     endEstablishTracing();
@@ -799,11 +797,12 @@ Consensus<Adaptor>::startRoundInternal(
     CLOG(clog) << "startRoundInternal transitioned to ConsensusPhase::Open, "
                   "previous ledgerID: "
                << prevLedgerID << ", seq: " << prevLedger.seq() << ". ";
-    // Reset establishSpan_ so a wrongLedger recovery mid-establish doesn't
-    // leak the prior round's span into the new one (startEstablishTracing
-    // early-returns when establishSpan_ is populated).
-    establishSpan_.reset();
-    establishSpanContext_ = telemetry::SpanContext{};
+    // End establishSpan_ so a wrongLedger recovery mid-establish doesn't leak
+    // the prior round's span into the new one (startEstablishTracing
+    // early-returns when establishSpan_ is populated). Via
+    // endEstablishTracing() so the recovered round still records its terminal
+    // regime; closeTimeAvalancheState_ is not reset until further down.
+    endEstablishTracing();
     // Child of the round span via its captured context: parent phase.open
     // explicitly under roundSpanContext_. An invalid round context (round span
     // not yet created) yields a null guard. openSpan_ is a thread-free
@@ -2180,14 +2179,6 @@ Consensus<Adaptor>::startEstablishTracing()
     if (*establishSpan_)
     {
         establishSpanContext_ = establishSpan_->spanContext();
-        // Disputes carried in from the positions held at close. Set once:
-        // this function early-returns while the span is live.
-        if (result_)
-        {
-            establishSpan_->setAttribute(
-                telemetry::consensus::span::attr::disputesCountInitial,
-                static_cast<int64_t>(result_->disputes.size()));
-        }
     }
 }
 
@@ -2217,7 +2208,7 @@ Consensus<Adaptor>::endEstablishTracing()
     if (establishSpan_ && *establishSpan_)
     {
         establishSpan_->setAttribute(
-            telemetry::consensus::span::attr::avalancheState,
+            telemetry::consensus::span::attr::closeTimeAvalancheState,
             telemetry::consensus::span::avalancheStateLabel(closeTimeAvalancheState_));
     }
     establishSpan_.reset();
diff --git a/include/xrpl/consensus/ConsensusSpanNames.h b/include/xrpl/consensus/ConsensusSpanNames.h
index cb7d7a8c85..d6715f2732 100644
--- a/include/xrpl/consensus/ConsensusSpanNames.h
+++ b/include/xrpl/consensus/ConsensusSpanNames.h
@@ -22,8 +22,11 @@
  *    |     Created: Consensus::startRoundInternal()
  *    |     Ended:   Consensus::closeLedger()
  *    |     Attrs:   start_reason, previous_close_agree, peer_positions_at_open,
- *    |              early_close_triggered (all at start); open_duration_ms,
- *    |              peer_positions_at_close, tx_sets_acquired (all at end)
+ *    |              early_close_triggered (at start); open_duration_ms,
+ *    |              peer_positions_at_close, tx_sets_acquired, close_reason,
+ *    |              proposers_validated (at close; absent if the round is
+ *    |              recovered or simulated, neither of which reaches
+ *    |              closeLedger())
  *    |
  *    +-- consensus.proposal.send                 [main thread]
  *    |     Created: Adaptor::propose()
@@ -36,9 +39,9 @@
  *    +-- consensus.establish                     [main thread, child]
  *    |     Created: Consensus::startEstablishTracing()
  *    |     Ended:   Consensus::phaseEstablish() on accept
- *    |     Attrs:   disputes_count_initial (at start); converge_percent,
- *    |              establish_count, proposers, disputes_count (overwritten
- *    |              each iteration); avalanche_state (terminal, at end)
+ *    |     Attrs:   converge_percent, establish_count, proposers,
+ *    |              disputes_count (overwritten each iteration);
+ *    |              close_time_avalanche_state (terminal, at end)
  *    |
  *    +-- consensus.update_positions              [main thread]
  *    |     Created: Consensus::updateOurPositions()
@@ -191,10 +194,11 @@ inline constexpr auto earlyCloseTriggered = makeStr("early_close_triggered");
 /**
  * Open-phase end metadata (set on consensus.phase.open before reset).
  *
- * A low `tx_sets_acquired` next to a high peer_positions_at_close points at
- * missing tx-set fetches rather than at disagreement. `close_reason` plus
- * `proposers_validated` separate "the network moved on without us" from "the
- * network was quiet".
+ * A low `tx_sets_acquired` next to a high peer_positions_at_close suggests
+ * tx-set fetches did not land; the reverse skew is also possible, because
+ * handleWrongLedger clears currPeerPositions_ but not acquired_.
+ * `close_reason` plus `proposers_validated` separate "the network moved on
+ * without us" from "the network was quiet".
  */
 inline constexpr auto openDurationMs = makeStr("open_duration_ms");
 inline constexpr auto peerPositionsAtClose = makeStr("peer_positions_at_close");
@@ -210,15 +214,13 @@ inline constexpr auto txCountOpen = makeStr("tx_count_open");
  */
 inline constexpr auto proposersFinished = makeStr("proposers_finished");
 /**
- * Establish-phase start/end metadata.
+ * Establish-phase end metadata.
  *
- * Both are set once, unlike `disputes_count`, which
- * updateEstablishTracing() overwrites every iteration.
- * `avalanche_state` is the terminal regime; the derived
- * `avalanche_threshold` cannot be inverted back to it.
+ * The terminal close-time regime, set once. Qualified because DisputedTx
+ * tracks a SECOND, per-transaction avalanche; this is not that one. The
+ * derived `avalanche_threshold` cannot be inverted back to it.
  */
-inline constexpr auto disputesCountInitial = makeStr("disputes_count_initial");
-inline constexpr auto avalancheState = makeStr("avalanche_state");
+inline constexpr auto closeTimeAvalancheState = makeStr("close_time_avalanche_state");
 /**
  * Accept/apply enrichment.
  */
@@ -331,11 +333,13 @@ inline constexpr auto phaseAccepted = makeStr("accepted");
 // start_reason values (how startRoundInternal was entered).
 inline constexpr auto startInitial = makeStr("initial");
 inline constexpr auto startRecovered = makeStr("recovered");
-// avalanche_state values, one per ConsensusParms::AvalancheState enumerator.
+// close_time_avalanche_state values, one per AvalancheState enumerator.
 inline constexpr auto avalancheInit = makeStr("init");
 inline constexpr auto avalancheMid = makeStr("mid");
 inline constexpr auto avalancheLate = makeStr("late");
 inline constexpr auto avalancheStuck = makeStr("stuck");
+// Sentinel for an unmapped enumerator, matching to_string(ConsensusPhase).
+inline constexpr auto unknown = makeStr("unknown");
 // close_reason values, one per LedgerCloseReason enumerator. keep_open is
 // never emitted: the attribute is only set on the path that closes.
 inline constexpr auto closeKeepOpen = makeStr("keep_open");
@@ -354,8 +358,8 @@ inline constexpr auto closeNormal = makeStr("normal");
  * @param state The state held by Consensus::closeTimeAvalancheState_.
  * @return The wire label; one of val::avalanche*.
  *
- * @note No default arm, so a new enumerator is a compiler warning rather than
- * a silently reused label.
+ * @note No default arm, so a new enumerator is a -Wswitch warning; the
+ * fall-through returns "unknown" rather than a plausible-looking regime.
  */
 [[nodiscard]] constexpr std::string_view
 avalancheStateLabel(ConsensusParms::AvalancheState const state)
@@ -371,7 +375,7 @@ avalancheStateLabel(ConsensusParms::AvalancheState const state)
         case ConsensusParms::AvalancheState::Stuck:
             return val::avalancheStuck;
     }
-    return val::avalancheInit;
+    return val::unknown;
 }
 
 /**
@@ -380,8 +384,8 @@ avalancheStateLabel(ConsensusParms::AvalancheState const state)
  * @param reason The value returned by whyCloseLedger().
  * @return The wire label; one of val::close*.
  *
- * @note No default arm, so a new enumerator is a compiler warning rather than
- * a silently reused label. `keep_open` is mapped but never emitted.
+ * @note No default arm, so a new enumerator is a -Wswitch warning; the
+ * fall-through returns "unknown". `keep_open` is mapped but never emitted.
  */
 [[nodiscard]] constexpr std::string_view
 closeReasonLabel(LedgerCloseReason const reason)
@@ -399,7 +403,7 @@ closeReasonLabel(LedgerCloseReason const reason)
         case LedgerCloseReason::Normal:
             return val::closeNormal;
     }
-    return val::closeKeepOpen;
+    return val::unknown;
 }
 
 }  // namespace xrpl::telemetry::consensus::span
diff --git a/src/tests/libxrpl/consensus/Consensus.cpp b/src/tests/libxrpl/consensus/Consensus.cpp
index c0298b64a7..d0878252bf 100644
--- a/src/tests/libxrpl/consensus/Consensus.cpp
+++ b/src/tests/libxrpl/consensus/Consensus.cpp
@@ -257,7 +257,7 @@ TEST(ConsensusTest, why_close_ledger_reports_the_deciding_branch)
     ConsensusParms const p{};
 
     // Bizarre times forcibly close. These vectors ALSO satisfy the
-    // others-closed condition (8 > 10/2), so they pin the precedence: the
+    // others-closed condition (10+10 > 10/2), so they pin the precedence: the
     // anomaly check runs first.
     EXPECT_EQ(whyCloseLedger(true, 10, 10, 10, -10s, 10s, 1s, 1s, p), LedgerCloseReason::Anomaly);
     EXPECT_EQ(whyCloseLedger(true, 10, 10, 10, 100h, 10s, 1s, 1s, p), LedgerCloseReason::Anomaly);
@@ -272,8 +272,11 @@ TEST(ConsensusTest, why_close_ledger_reports_the_deciding_branch)
     EXPECT_EQ(whyCloseLedger(false, 10, 0, 0, 1s, 1s, 1s, 10s, p), LedgerCloseReason::KeepOpen);
     EXPECT_EQ(whyCloseLedger(false, 10, 0, 0, 1s, 10s, 1s, 10s, p), LedgerCloseReason::Idle);
 
-    // Transactions present, but under ledgerMinClose (2s).
-    EXPECT_EQ(whyCloseLedger(true, 10, 0, 0, 10s, 10s, 1s, 10s, p), LedgerCloseReason::KeepOpen);
+    // Under ledgerMinClose (2s). prevRoundTime is 2s so prevRoundTime/2 is 1s
+    // and openTime is NOT under it -- this vector isolates the min-close
+    // branch, which the 10s variant does not (there openTime < 5s trips the
+    // too-fast branch as well, so deleting min-close entirely still passes).
+    EXPECT_EQ(whyCloseLedger(true, 10, 0, 0, 2s, 10s, 1s, 10s, p), LedgerCloseReason::KeepOpen);
 
     // Past ledgerMinClose but under prevRoundTime/2 (5s), so still too fast.
     EXPECT_EQ(whyCloseLedger(true, 10, 0, 0, 10s, 10s, 3s, 10s, p), LedgerCloseReason::KeepOpen);
@@ -282,6 +285,28 @@ TEST(ConsensusTest, why_close_ledger_reports_the_deciding_branch)
     EXPECT_EQ(whyCloseLedger(true, 10, 0, 0, 10s, 10s, 10s, 10s, p), LedgerCloseReason::Normal);
 }
 
+TEST(ConsensusTest, why_close_ledger_others_closed_boundary_is_exclusive)
+{
+    using namespace std::chrono_literals;
+    SCOPED_TRACE("others-closed boundary");
+
+    // The branch is `(closed + validated) > prevProposers / 2`, strict. With
+    // prevProposers 10 the threshold is 5, so 5 must NOT close and 6 must.
+    // Flipping > to >= would otherwise go unnoticed.
+    ConsensusParms const p{};
+
+    EXPECT_EQ(whyCloseLedger(true, 10, 3, 2, 10s, 10s, 10s, 10s, p), LedgerCloseReason::Normal);
+    EXPECT_EQ(
+        whyCloseLedger(true, 10, 3, 3, 10s, 10s, 10s, 10s, p), LedgerCloseReason::OthersClosed);
+
+    // Integer truncation: 11/2 is 5, so 5 still does not close.
+    EXPECT_EQ(whyCloseLedger(true, 11, 3, 2, 10s, 10s, 10s, 10s, p), LedgerCloseReason::Normal);
+
+    // Others-closed outranks both the no-transactions and the minimum-open
+    // branches, which would otherwise return KeepOpen for these inputs.
+    EXPECT_EQ(whyCloseLedger(false, 10, 3, 5, 1s, 1s, 1s, 10s, p), LedgerCloseReason::OthersClosed);
+}
+
 TEST(ConsensusTest, why_close_ledger_idle_boundary_is_inclusive)
 {
     using namespace std::chrono_literals;
diff --git a/src/tests/libxrpl/telemetry/ConsensusSpanNames.cpp b/src/tests/libxrpl/telemetry/ConsensusSpanNames.cpp
index d371b67789..d4cd8ebcb5 100644
--- a/src/tests/libxrpl/telemetry/ConsensusSpanNames.cpp
+++ b/src/tests/libxrpl/telemetry/ConsensusSpanNames.cpp
@@ -78,8 +78,8 @@ TEST(ConsensusSpanNames, close_reason_label_is_usable_at_compile_time)
 
 TEST(ConsensusSpanNames, establish_attribute_keys)
 {
-    EXPECT_EQ(std::string_view(attr::disputesCountInitial), "disputes_count_initial");
-    EXPECT_EQ(std::string_view(attr::avalancheState), "avalanche_state");
+    // Qualified: DisputedTx tracks a second, per-transaction avalanche.
+    EXPECT_EQ(std::string_view(attr::closeTimeAvalancheState), "close_time_avalanche_state");
 }
 
 TEST(ConsensusSpanNames, start_reason_values_are_the_two_entry_paths)
@@ -97,6 +97,7 @@ TEST(ConsensusSpanNames, avalanche_state_values_match_the_parms_enum)
     EXPECT_EQ(std::string_view(val::avalancheMid), "mid");
     EXPECT_EQ(std::string_view(val::avalancheLate), "late");
     EXPECT_EQ(std::string_view(val::avalancheStuck), "stuck");
+    EXPECT_EQ(std::string_view(val::unknown), "unknown");
 }
 
 TEST(ConsensusSpanNames, avalanche_state_label_maps_every_enum_state)

From 258adf491e9acb16b789e7310535ef23b844b6a8 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 20:45:24 +0100
Subject: [PATCH 12/38] refactor(telemetry): split consensus span labels into
 their own header

Review of the preceding commits found a clang-tidy failure and a convention
break, both rooted in the same place: the enum-to-label helpers were put in
ConsensusSpanNames.h, which pulled two domain headers into it.

misc-include-cleaner rejected the new test: it used xrpl::LedgerCloseReason
without directly including ConsensusTypes.h, relying on the transitive
include. misc-* is enabled and this path is not in IgnoreHeaders, so it would
have failed CI.

ConsensusSpanNames.h had also become the only one of the eight *SpanNames.h
headers to include anything beyond SpanNames.h. That cost is paid by every
consumer: PeerImp.cpp, ConsensusReceiveTracing.h and RCLConsensus.cpp want
only name and key constants, but were newly compiling ConsensusTypes.h and
DisputedTx.h through it.

Move both helpers to a new ConsensusSpanLabels.h, which owns the domain
includes. ConsensusSpanNames.h is dependency-free again like its siblings, and
the labels reach their only production caller, Consensus.h, directly.

Also from the review:

- phaseOpen() had grown to 81 lines, over the 80-line limit. Extract
  annotateOpenStart() and annotateOpenClose(), which also removes the repeated
  span guards. phaseOpen is 72 lines; startRoundInternal drops 103 to 93,
  still over the limit but it was 99 before this work began.
- Note at the CLOG why the log text keeps the shouldCloseLedger name: existing
  consumers match on it.
- whyCloseLedger's doc claimed "both log identically", implying the wrapper
  logs too. It delegates, so the logging happens once either way.
- Cross-reference proposers_validated and proposers_finished, which sit eight
  lines apart and count different things: validators of the previous ledger
  versus those already past it.
- The two static_asserts no longer sit inside TEST bodies with SUCCEED(); they
  fire at compile time regardless. Also "consteval-safe" was wrong; they are
  constexpr.
- SpanGuardFactory.cpp claimed a libxrpl test cannot include the consensus
  span-name header. The new test in the same directory does exactly that, so
  the claim is corrected to name the real constraint: the rpc_* constants it
  needs live in an xrpld-level header.
---
 include/xrpl/consensus/Consensus.h            | 83 ++++++++++++------
 include/xrpl/consensus/ConsensusSpanLabels.h  | 85 +++++++++++++++++++
 include/xrpl/consensus/ConsensusSpanNames.h   | 66 +-------------
 src/libxrpl/consensus/Consensus.cpp           |  1 +
 .../libxrpl/telemetry/ConsensusSpanNames.cpp  | 27 +++---
 .../libxrpl/telemetry/SpanGuardFactory.cpp    |  7 +-
 6 files changed, 158 insertions(+), 111 deletions(-)
 create mode 100644 include/xrpl/consensus/ConsensusSpanLabels.h

diff --git a/include/xrpl/consensus/Consensus.h b/include/xrpl/consensus/Consensus.h
index 577b154cbe..f299ce2b9a 100644
--- a/include/xrpl/consensus/Consensus.h
+++ b/include/xrpl/consensus/Consensus.h
@@ -8,6 +8,7 @@
 #include <xrpl/beast/utility/instrumentation.h>
 #include <xrpl/consensus/ConsensusParms.h>
 #include <xrpl/consensus/ConsensusProposal.h>
+#include <xrpl/consensus/ConsensusSpanLabels.h>
 #include <xrpl/consensus/ConsensusSpanNames.h>
 #include <xrpl/consensus/ConsensusTypes.h>
 #include <xrpl/json/json_value.h>
@@ -34,9 +35,9 @@ namespace xrpl {
 /**
  * Determines why the current ledger should close at this time.
  *
- * Holds the close decision that shouldCloseLedger() reduces to a bool. Call
- * this one when the deciding branch matters. Both log identically, so call
- * one or the other, never both. Parameters match shouldCloseLedger().
+ * Holds the close decision. shouldCloseLedger() delegates here and adds only
+ * a comparison, so the logging happens once either way; call whichever suits.
+ * Parameters match shouldCloseLedger().
  *
  * @return The deciding branch, or KeepOpen if no close condition is met.
  */
@@ -692,6 +693,24 @@ private:
      */
     std::optional<xrpl::telemetry::SpanGuard> openSpan_;
 
+    /**
+     * Record how the open-phase span began.
+     *
+     * @param reason      Which startRoundInternal() entry path created it.
+     * @param prevLedger  The prior ledger, read before previousLedger_ is set.
+     */
+    void
+    annotateOpenStart(StartRoundReason reason, Ledger_t const& prevLedger);
+
+    /**
+     * Record what ended the open phase.
+     *
+     * @param closeReason         The deciding whyCloseLedger() branch.
+     * @param proposersValidated  Trusted peers already past the prior ledger.
+     */
+    void
+    annotateOpenClose(LedgerCloseReason closeReason, std::size_t proposersValidated);
+
     /**
      * Create the establish-phase span if not yet active.
      *  Called on each phaseEstablish() invocation; no-op while span is live.
@@ -811,18 +830,7 @@ Consensus<Adaptor>::startRoundInternal(
     openSpan_.emplace(
         telemetry::SpanGuard::childSpan(
             telemetry::consensus::span::phaseOpen, adaptor_.roundSpanContext()));
-    if (*openSpan_)
-    {
-        namespace cs = telemetry::consensus::span;
-        // A recovery emplaces a SECOND phase.open span under the same round,
-        // so this is what tells the two apart.
-        openSpan_->setAttribute(
-            cs::attr::startReason,
-            reason == StartRoundReason::Recovered ? std::string_view{cs::val::startRecovered}
-                                                  : std::string_view{cs::val::startInitial});
-        // From the parameter: previousLedger_ is not assigned until below.
-        openSpan_->setAttribute(cs::attr::previousCloseAgree, prevLedger.closeAgree());
-    }
+    annotateOpenStart(reason, prevLedger);
     // On the Recovered path, fire phase.open here because startRoundTracing
     // (which fires it for the Initial path) is not called on re-entry. On
     // the Initial path this is a no-op because the round span hasn't been
@@ -1376,16 +1384,7 @@ Consensus<Adaptor>::phaseOpen(std::unique_ptr<std::stringstream> const& clog)
         clog);
     if (closeReason != LedgerCloseReason::KeepOpen)
     {
-        // Annotate before closeLedger() ends the span. Set once: the phase
-        // moves to Establish, so phaseOpen() is not entered again this round.
-        // Absent on the simulate() path, which bypasses this decision.
-        if (openSpan_ && *openSpan_)
-        {
-            namespace cs = telemetry::consensus::span;
-            openSpan_->setAttribute(cs::attr::closeReason, cs::closeReasonLabel(closeReason));
-            openSpan_->setAttribute(
-                cs::attr::proposersValidated, static_cast<int64_t>(proposersValidated));
-        }
+        annotateOpenClose(closeReason, proposersValidated);
         CLOG(clog) << "closing ledger. ";
         closeLedger(clog);
     }
@@ -2160,6 +2159,36 @@ Consensus<Adaptor>::asCloseTime(NetClock::time_point raw) const
     return roundCloseTime(raw, closeResolution_);
 }
 
+template <class Adaptor>
+void
+Consensus<Adaptor>::annotateOpenStart(StartRoundReason const reason, Ledger_t const& prevLedger)
+{
+    if (!openSpan_ || !*openSpan_)
+        return;
+    namespace cs = telemetry::consensus::span;
+    openSpan_->setAttribute(
+        cs::attr::startReason,
+        reason == StartRoundReason::Recovered ? std::string_view{cs::val::startRecovered}
+                                              : std::string_view{cs::val::startInitial});
+    // From the parameter: previousLedger_ is not assigned until later.
+    openSpan_->setAttribute(cs::attr::previousCloseAgree, prevLedger.closeAgree());
+}
+
+template <class Adaptor>
+void
+Consensus<Adaptor>::annotateOpenClose(
+    LedgerCloseReason const closeReason,
+    std::size_t const proposersValidated)
+{
+    // Called before closeLedger() ends the span, and only on the closing tick,
+    // so each attribute is written once per round.
+    if (!openSpan_ || !*openSpan_)
+        return;
+    namespace cs = telemetry::consensus::span;
+    openSpan_->setAttribute(cs::attr::closeReason, cs::closeReasonLabel(closeReason));
+    openSpan_->setAttribute(cs::attr::proposersValidated, static_cast<int64_t>(proposersValidated));
+}
+
 template <class Adaptor>
 void
 Consensus<Adaptor>::startEstablishTracing()
@@ -2207,9 +2236,9 @@ Consensus<Adaptor>::endEstablishTracing()
     // Terminal convergence regime, recorded once before the span ends.
     if (establishSpan_ && *establishSpan_)
     {
+        namespace cs = telemetry::consensus::span;
         establishSpan_->setAttribute(
-            telemetry::consensus::span::attr::closeTimeAvalancheState,
-            telemetry::consensus::span::avalancheStateLabel(closeTimeAvalancheState_));
+            cs::attr::closeTimeAvalancheState, cs::avalancheStateLabel(closeTimeAvalancheState_));
     }
     establishSpan_.reset();
     establishSpanContext_ = telemetry::SpanContext{};
diff --git a/include/xrpl/consensus/ConsensusSpanLabels.h b/include/xrpl/consensus/ConsensusSpanLabels.h
new file mode 100644
index 0000000000..cc66639c41
--- /dev/null
+++ b/include/xrpl/consensus/ConsensusSpanLabels.h
@@ -0,0 +1,85 @@
+#pragma once
+
+/**
+ * Enum-to-label mappings for consensus span attribute values.
+ *
+ *  Split from ConsensusSpanNames.h so that header stays dependency-free like
+ *  its siblings: the span-name and attribute-key constants are included by
+ *  overlay and app translation units that have no use for the consensus
+ *  enums, while these mappings are needed only by Consensus.h.
+ *
+ *      ConsensusSpanNames.h  (constants only, no domain deps)
+ *              ^
+ *              |  includes
+ *      ConsensusSpanLabels.h  --includes--> ConsensusParms.h, ConsensusTypes.h
+ *              ^
+ *              |  includes
+ *          Consensus.h
+ */
+
+#include <xrpl/consensus/ConsensusParms.h>
+#include <xrpl/consensus/ConsensusSpanNames.h>
+#include <xrpl/consensus/ConsensusTypes.h>
+
+#include <string_view>
+
+namespace xrpl::telemetry::consensus::span {
+
+/**
+ * Map a close-time avalanche state to its `avalanche_state` label.
+ *
+ * The regime escalates Init -> Mid -> Late -> Stuck, raising the close-time
+ * agreement threshold at each step.
+ *
+ * @param state The state held by Consensus::closeTimeAvalancheState_.
+ * @return The wire label; one of val::avalanche*.
+ *
+ * @note No default arm, so a new enumerator is a -Wswitch warning; the
+ * fall-through returns "unknown" rather than a plausible-looking regime.
+ */
+[[nodiscard]] constexpr std::string_view
+avalancheStateLabel(ConsensusParms::AvalancheState const state)
+{
+    switch (state)
+    {
+        case ConsensusParms::AvalancheState::Init:
+            return val::avalancheInit;
+        case ConsensusParms::AvalancheState::Mid:
+            return val::avalancheMid;
+        case ConsensusParms::AvalancheState::Late:
+            return val::avalancheLate;
+        case ConsensusParms::AvalancheState::Stuck:
+            return val::avalancheStuck;
+    }
+    return val::unknown;
+}
+
+/**
+ * Map a ledger-close decision to its `close_reason` label.
+ *
+ * @param reason The value returned by whyCloseLedger().
+ * @return The wire label; one of val::close*.
+ *
+ * @note No default arm, so a new enumerator is a -Wswitch warning; the
+ * fall-through returns "unknown". `keep_open` is mapped but never emitted.
+ */
+[[nodiscard]] constexpr std::string_view
+closeReasonLabel(LedgerCloseReason const reason)
+{
+    switch (reason)
+    {
+        case LedgerCloseReason::KeepOpen:
+            return val::closeKeepOpen;
+        case LedgerCloseReason::Anomaly:
+            return val::closeAnomaly;
+        case LedgerCloseReason::OthersClosed:
+            return val::closeOthersClosed;
+        case LedgerCloseReason::Idle:
+            return val::closeIdle;
+        case LedgerCloseReason::Normal:
+            return val::closeNormal;
+    }
+    return val::unknown;
+}
+
+}  // namespace xrpl::telemetry::consensus::span
diff --git a/include/xrpl/consensus/ConsensusSpanNames.h b/include/xrpl/consensus/ConsensusSpanNames.h
index d6715f2732..4945db30d0 100644
--- a/include/xrpl/consensus/ConsensusSpanNames.h
+++ b/include/xrpl/consensus/ConsensusSpanNames.h
@@ -86,8 +86,6 @@
  *    +~~~ follows-from link (separate sub-tree, causal link)
  */
 
-#include <xrpl/consensus/ConsensusParms.h>
-#include <xrpl/consensus/ConsensusTypes.h>
 #include <xrpl/telemetry/SpanNames.h>
 
 #include <string_view>
@@ -194,11 +192,8 @@ inline constexpr auto earlyCloseTriggered = makeStr("early_close_triggered");
 /**
  * Open-phase end metadata (set on consensus.phase.open before reset).
  *
- * A low `tx_sets_acquired` next to a high peer_positions_at_close suggests
- * tx-set fetches did not land; the reverse skew is also possible, because
- * handleWrongLedger clears currPeerPositions_ but not acquired_.
- * `close_reason` plus `proposers_validated` separate "the network moved on
- * without us" from "the network was quiet".
+ * `proposers_validated` counts validators of the previous ledger, unlike
+ * `proposers_finished` below, which counts those already past it.
  */
 inline constexpr auto openDurationMs = makeStr("open_duration_ms");
 inline constexpr auto peerPositionsAtClose = makeStr("peer_positions_at_close");
@@ -349,61 +344,4 @@ inline constexpr auto closeIdle = makeStr("idle");
 inline constexpr auto closeNormal = makeStr("normal");
 }  // namespace val
 
-/**
- * Map a close-time avalanche state to its `avalanche_state` label.
- *
- * The regime escalates Init -> Mid -> Late -> Stuck, raising the close-time
- * agreement threshold at each step.
- *
- * @param state The state held by Consensus::closeTimeAvalancheState_.
- * @return The wire label; one of val::avalanche*.
- *
- * @note No default arm, so a new enumerator is a -Wswitch warning; the
- * fall-through returns "unknown" rather than a plausible-looking regime.
- */
-[[nodiscard]] constexpr std::string_view
-avalancheStateLabel(ConsensusParms::AvalancheState const state)
-{
-    switch (state)
-    {
-        case ConsensusParms::AvalancheState::Init:
-            return val::avalancheInit;
-        case ConsensusParms::AvalancheState::Mid:
-            return val::avalancheMid;
-        case ConsensusParms::AvalancheState::Late:
-            return val::avalancheLate;
-        case ConsensusParms::AvalancheState::Stuck:
-            return val::avalancheStuck;
-    }
-    return val::unknown;
-}
-
-/**
- * Map a ledger-close decision to its `close_reason` label.
- *
- * @param reason The value returned by whyCloseLedger().
- * @return The wire label; one of val::close*.
- *
- * @note No default arm, so a new enumerator is a -Wswitch warning; the
- * fall-through returns "unknown". `keep_open` is mapped but never emitted.
- */
-[[nodiscard]] constexpr std::string_view
-closeReasonLabel(LedgerCloseReason const reason)
-{
-    switch (reason)
-    {
-        case LedgerCloseReason::KeepOpen:
-            return val::closeKeepOpen;
-        case LedgerCloseReason::Anomaly:
-            return val::closeAnomaly;
-        case LedgerCloseReason::OthersClosed:
-            return val::closeOthersClosed;
-        case LedgerCloseReason::Idle:
-            return val::closeIdle;
-        case LedgerCloseReason::Normal:
-            return val::closeNormal;
-    }
-    return val::unknown;
-}
-
 }  // namespace xrpl::telemetry::consensus::span
diff --git a/src/libxrpl/consensus/Consensus.cpp b/src/libxrpl/consensus/Consensus.cpp
index 50d9a82d5f..6f3e36fe92 100644
--- a/src/libxrpl/consensus/Consensus.cpp
+++ b/src/libxrpl/consensus/Consensus.cpp
@@ -27,6 +27,7 @@ whyCloseLedger(
     beast::Journal j,
     std::unique_ptr<std::stringstream> const& clog)
 {
+    // Log text keeps the shouldCloseLedger name: consumers match on it.
     CLOG(clog) << "shouldCloseLedger params anyTransactions: " << anyTransactions
                << ", prevProposers: " << prevProposers << ", proposersClosed: " << proposersClosed
                << ", proposersValidated: " << proposersValidated
diff --git a/src/tests/libxrpl/telemetry/ConsensusSpanNames.cpp b/src/tests/libxrpl/telemetry/ConsensusSpanNames.cpp
index d4cd8ebcb5..708a94f05f 100644
--- a/src/tests/libxrpl/telemetry/ConsensusSpanNames.cpp
+++ b/src/tests/libxrpl/telemetry/ConsensusSpanNames.cpp
@@ -1,6 +1,8 @@
 #include <xrpl/consensus/ConsensusSpanNames.h>
 
 #include <xrpl/consensus/ConsensusParms.h>
+#include <xrpl/consensus/ConsensusSpanLabels.h>
+#include <xrpl/consensus/ConsensusTypes.h>
 
 #include <gtest/gtest.h>
 
@@ -68,14 +70,6 @@ TEST(ConsensusSpanNames, close_reason_label_maps_every_enum_state)
     EXPECT_EQ(closeReasonLabel(xrpl::LedgerCloseReason::Normal), "normal");
 }
 
-TEST(ConsensusSpanNames, close_reason_label_is_usable_at_compile_time)
-{
-    static_assert(
-        closeReasonLabel(xrpl::LedgerCloseReason::Idle) == "idle",
-        "closeReasonLabel must be constexpr-evaluable");
-    SUCCEED();
-}
-
 TEST(ConsensusSpanNames, establish_attribute_keys)
 {
     // Qualified: DisputedTx tracks a second, per-transaction avalanche.
@@ -113,12 +107,11 @@ TEST(ConsensusSpanNames, avalanche_state_label_maps_every_enum_state)
     EXPECT_EQ(avalancheStateLabel(AvalancheState::Stuck), "stuck");
 }
 
-TEST(ConsensusSpanNames, avalanche_state_label_is_usable_at_compile_time)
-{
-    // The mapping is consteval-safe so the label costs nothing at the call
-    // site in endEstablishTracing().
-    static_assert(
-        avalancheStateLabel(xrpl::ConsensusParms::AvalancheState::Stuck) == "stuck",
-        "avalancheStateLabel must be constexpr-evaluable");
-    SUCCEED();
-}
+// Both mappings are constexpr, so the labels cost nothing at their call sites.
+// These fire at compile time and need no test body.
+static_assert(
+    closeReasonLabel(xrpl::LedgerCloseReason::Idle) == "idle",
+    "closeReasonLabel must be constexpr-evaluable");
+static_assert(
+    avalancheStateLabel(xrpl::ConsensusParms::AvalancheState::Stuck) == "stuck",
+    "avalancheStateLabel must be constexpr-evaluable");
diff --git a/src/tests/libxrpl/telemetry/SpanGuardFactory.cpp b/src/tests/libxrpl/telemetry/SpanGuardFactory.cpp
index 75e5804bbb..36ab40b1b6 100644
--- a/src/tests/libxrpl/telemetry/SpanGuardFactory.cpp
+++ b/src/tests/libxrpl/telemetry/SpanGuardFactory.cpp
@@ -31,9 +31,10 @@ TEST(SpanGuardFactory, category_span_returns_null_when_disabled)
     EXPECT_FALSE(span);
 
     // Attribute keys use the underscore convention for span attributes (the
-    // dotted xrpl.<domain>. form is reserved for resource attributes). The
-    // canonical constants live in the xrpld-level *SpanNames.h headers, which a
-    // libxrpl test cannot include, so the keys are written as literals here.
+    // dotted xrpl.<domain>. form is reserved for resource attributes). These
+    // rpc_* constants live in an xrpld-level header, so they are literals here;
+    // libxrpl-level headers such as ConsensusSpanNames.h can be included
+    // directly, as ConsensusSpanNames.cpp does.
     span.setAttribute("command", "test");
     span.setAttribute("rpc_status", "success");
 }

From 2097293e8f3f9f04312f3a5e98e840cb521a939f Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 20:46:53 +0100
Subject: [PATCH 13/38] fix(telemetry): run the workload at info so log-trace
 correlation is testable

The two log.trace_id_* checks have failed on every run -- they were the only
failures in the 2026-08-20 run (158/160). The workload never satisfied their
precondition, because warning suppressed the one line that is correlated by
construction.

trace_id is injected in Log.cpp from RuntimeContext::GetCurrent(). Severity
does not affect injection, but JLOG filters on severity before format() runs,
so what matters is which severity emits a line while a span is current.

A span becomes current in either of two ways: as a ScopedSpanGuard, or by
activating a plain SpanGuard via activate() / activateIfLive(). activate()
returns a ScopedActivation holding an otel_trace::Scope built from the span,
which pushes onto the same RuntimeContext store Log.cpp reads. A plain
SpanGuard that is never activated makes no span current.

The guaranteed correlated line at info is the consensus accept pair at
RCLConsensus.cpp:736/740 -- an if/else, so exactly one fires on every accepted
round. doAccept activates the accept span as ambient over its whole body at
:565 via activateIfLive(acceptSpan), and that activation lives to the end of
the function, so both branches are inside it. At roughly one round every 4 s
this gives dozens of correlated lines per run, well inside the validator's 4 h
window. LOG_QUERY_WINDOW_SECONDS stays at 4 h deliberately -- a wider window
would let the check pass on logs from a previous run.

info is the minimum that works, which is what the task asked for. debug would
correlate strictly more, additionally covering BuildLedger.cpp:81 and
RPCHandler.cpp:188, but it is the wrong default: it puts synchronous log I/O
inside ledger.build, consensus.accept (RCLConsensus.cpp:663 logs per
transaction) and tx.apply, which are exactly the spans whose latency
regression-metrics.json gates. The next run reprints the voided baseline, so
capturing at debug would bake log I/O into the latency numbers permanently --
the same class of defect this plan exists to remove. The runbook records how to
get the broader coverage per partition, after a baseline exists.
---
 .../telemetry/workload/run-full-validation.sh | 18 ++++++++++-
 .../workload/xrpld-validator.cfg.template     | 11 ++++++-
 docs/telemetry-runbook.md                     | 32 +++++++++++++++++++
 3 files changed, 59 insertions(+), 2 deletions(-)

diff --git a/docker/telemetry/workload/run-full-validation.sh b/docker/telemetry/workload/run-full-validation.sh
index 0cc546ad38..ef3cc6cb45 100755
--- a/docker/telemetry/workload/run-full-validation.sh
+++ b/docker/telemetry/workload/run-full-validation.sh
@@ -337,7 +337,23 @@ endpoint=http://localhost:4318/v1/metrics
 prefix=xrpld
 
 [rpc_startup]
-{ "command": "log_level", "severity": "warning" }
+# info is required here, not cosmetic, and is the minimum that works. The
+# log-trace correlation checks need a log line emitted while a span is current
+# on the thread. A span becomes current either as a ScopedSpanGuard or by
+# activating a plain SpanGuard via activate()/activateIfLive(); a plain
+# SpanGuard that is never activated yields no trace_id.
+#
+# The guaranteed correlated line at info is the consensus accept pair in
+# RCLConsensus.cpp (an if/else, so exactly one fires every accepted round).
+# doAccept activates the accept span as ambient over its whole body, so both
+# branches carry a trace_id. At ~4s per round that is dozens of correlated
+# lines per run.
+#
+# Do NOT raise this to debug to get more coverage. debug puts synchronous log
+# I/O inside ledger.build, consensus.accept and tx.apply -- the same spans
+# whose p50/p95/p99 latencies regression-metrics.json gates -- so a baseline
+# captured at debug would bake log I/O into the numbers permanently.
+{ "command": "log_level", "severity": "info" }
 
 [signing_support]
 true
diff --git a/docker/telemetry/workload/xrpld-validator.cfg.template b/docker/telemetry/workload/xrpld-validator.cfg.template
index 083c2eaf25..875abd63ab 100644
--- a/docker/telemetry/workload/xrpld-validator.cfg.template
+++ b/docker/telemetry/workload/xrpld-validator.cfg.template
@@ -15,7 +15,16 @@
 #   {{IPS_FIXED}}         — Peer addresses (one per line)
 #   {{OTEL_ENDPOINT}}     — OTel Collector OTLP/HTTP traces endpoint
 #   {{OTEL_METRICS_ENDPOINT}} — OTel Collector OTLP/HTTP metrics endpoint
-#   {{LOG_LEVEL}}         — Log level (debug, info, warning, error)
+#   {{LOG_LEVEL}}         — Log level. Must be at least `info` for the
+#                           log-trace correlation checks to pass: they need a
+#                           log line emitted while a span is current on the
+#                           thread. The guaranteed such line is the consensus
+#                           accept pair in RCLConsensus.cpp, which is at info
+#                           and sits inside the activation doAccept makes over
+#                           its whole body. `warning` and above suppress it and
+#                           leave the checks with nothing to find. Do not use
+#                           `debug`: it adds log I/O inside the spans whose
+#                           latency regression-metrics.json gates.
 
 [server]
 port_rpc
diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md
index fe3f5e1ce2..37fe574984 100644
--- a/docs/telemetry-runbook.md
+++ b/docs/telemetry-runbook.md
@@ -2747,6 +2747,36 @@ This enables bidirectional navigation between logs and traces in Grafana:
 - **Tempo -> Loki**: Click "Logs for this trace" on any trace in Grafana Tempo to see all log lines from that trace.
 - **Loki -> Tempo**: Click the `TraceID` derived field link on any log line containing `trace_id=` to jump to the full trace in Tempo.
 
+### Which Log Lines Carry Trace Context
+
+Correlation requires a log line emitted **while a span is current on the emitting thread**. `Log.cpp` injects `trace_id`/`span_id` by reading `RuntimeContext::GetCurrent()`, so what matters is whether anything has pushed a span onto that thread's context store.
+
+A span becomes current in **either** of two ways:
+
+- As a [`ScopedSpanGuard`](../include/xrpl/telemetry/SpanGuard.h), which activates on construction.
+- By activating a plain `SpanGuard` through `activate()` or the `activateIfLive()` wrapper. `activate()` returns a `ScopedActivation` whose `Impl` holds an `otel_trace::Scope` constructed from the span, and that `Scope` pushes onto the same `RuntimeContext` store `Log.cpp` reads.
+
+A plain `SpanGuard` that is **never activated** makes no span current — it "never pushes the span onto the thread-local context stack" ([SpanGuard.h](../include/xrpl/telemetry/SpanGuard.h#L274)) — so lines inside such a region carry no `trace_id` regardless of severity.
+
+Severity does not affect injection, but `JLOG` filters on severity **before** `format()` runs, so the configured log level decides whether a qualifying line is emitted at all.
+
+**The guaranteed correlated line at `info`** is the consensus accept pair at [RCLConsensus.cpp:736/740](../src/xrpld/app/consensus/RCLConsensus.cpp#L736) — an `if`/`else`, so exactly one of the two fires on every accepted round. `doAccept` activates the accept span as ambient over its whole body at [:565](../src/xrpld/app/consensus/RCLConsensus.cpp#L565) (`activateIfLive(acceptSpan)`, commented "Make the accept span ambient for the whole accept so doAccept's log lines ... correlate to it"), and the activation lives to the end of the function, so both branches are inside it. At roughly one round every 4 s this yields dozens of correlated lines per run.
+
+`info` is therefore the minimum level at which the `log.trace_id_present` and `log.trace_id_cross_reference` checks are satisfied by construction, and it is what the workload harness generates. At `warning` and above that pair is suppressed and correlation becomes incidental — dependent on a `warn`-or-worse line happening to fire inside some active span.
+
+#### Why not `debug`
+
+`debug` does correlate strictly more: it additionally brings in [`BuildLedger.cpp:81`](../src/xrpld/app/ledger/detail/BuildLedger.cpp#L81) (inside the `ledger.build` `ScopedSpanGuard` at [:55](../src/xrpld/app/ledger/detail/BuildLedger.cpp#L55), once per ledger close) and [`RPCHandler.cpp:188`](../src/xrpld/rpc/detail/RPCHandler.cpp#L188) (inside the `rpc.command.*` `ScopedSpanGuard` at [:168](../src/xrpld/rpc/detail/RPCHandler.cpp#L168), once per RPC command), giving broader multi-subsystem coverage.
+
+But raising the **base** level to `debug` puts synchronous log I/O inside `ledger.build`, `consensus.accept` (including [RCLConsensus.cpp:663](../src/xrpld/app/consensus/RCLConsensus.cpp#L663), which logs **per transaction**) and `tx.apply` — precisely the spans whose p50/p95/p99 latencies `regression-metrics.json` gates. A baseline captured at `debug` bakes that log I/O into the latency numbers permanently, turning the regression gate into a measurement of its own configuration.
+
+So if you need the broader coverage, enable it **per partition** rather than globally, and only **after** a baseline has been captured at the harness's normal level:
+
+```
+log_level LedgerConsensus debug
+log_level RPCHandler debug
+```
+
 ### Log Ingestion Pipeline
 
 Log files are ingested by the OTel Collector's `filelog` receiver, which tails `debug.log` files and parses them with a regex that extracts `timestamp`, `partition`, `severity`, `trace_id`, `span_id`, and `message` fields. Parsed entries are exported to Grafana Loki.
@@ -3485,6 +3515,8 @@ not a sign the cache is working.
 - Verify xrpld was built with `telemetry=ON` (the `XRPL_ENABLE_TELEMETRY` preprocessor flag)
 - Verify `enabled=1` in the `[telemetry]` config section
 - Log lines only contain `trace_id`/`span_id` when emitted inside an active span — background logs outside of RPC/consensus/transaction processing will not have trace context
+- Check `log_level` is at least `info`. The guaranteed correlated line is the consensus accept pair, which is at info severity, so at `warning` or above correlation becomes incidental — see [Which Log Lines Carry Trace Context](#which-log-lines-carry-trace-context)
+- A plain `SpanGuard` that is never activated does not make its span current, so lines inside one are never correlated regardless of severity. Activated guards (`activate()` / `activateIfLive()`) and `ScopedSpanGuard` both do make their span current
 - Check that the specific trace category is enabled (e.g., `trace_rpc=1`)
 
 ### No logs in Loki

From 3559968177ac99ec1193355aa37143c0a7d7c79f Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 20:47:37 +0100
Subject: [PATCH 14/38] docs(telemetry): correct counter names and rate windows
 in the runbook

The runbook's metric tables are headed "Prometheus Metric", so they
should carry the series names an operator can paste into Grafana. Five
counter rows carried the code-side name instead: the OTel Prometheus
exporter appends _total, so rpc_requests, ledger_fetches,
ledger_history_mismatch, warn and drop name no series at all. Verified
two ways: every dashboard queries the _total form and never the bare
one, and phase-10's expected_metrics.json lists the _total names. The
same file already used warn_total further down, so it contradicted
itself.

Nine panel rows documented a hardcoded [5m] window where the panel
actually uses $__rate_interval. The RPC Request Rate row had both
faults.

Left alone deliberately:
- four histogram _bucket rows keep [5m], because the dashboards
  hardcode the window for bucket queries and the dashboard linter only
  flags hardcoded ranges on counter-rate shapes;
- three example queries inside fenced code blocks, which are standalone
  snippets rather than panel documentation;
- the All Traffic Categories (Detail) row, which phase-10 has already
  rewritten to the label_replace shape.

Prettier re-padded two table separator rows because the longer names
widen the columns.
---
 docs/telemetry-runbook.md | 50 +++++++++++++++++++--------------------
 1 file changed, 25 insertions(+), 25 deletions(-)

diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md
index ef1656c650..b3fa323a7d 100644
--- a/docs/telemetry-runbook.md
+++ b/docs/telemetry-runbook.md
@@ -551,13 +551,13 @@ The `OTelCollector` implementation exports metrics via OTLP/HTTP to the same OTe
 
 #### Counters
 
-| Prometheus Metric         | Source                | Description                    |
-| ------------------------- | --------------------- | ------------------------------ |
-| `rpc_requests`            | ServerHandler.cpp:108 | Total RPC request count        |
-| `ledger_fetches`          | InboundLedgers.cpp:44 | Ledger fetch request count     |
-| `ledger_history_mismatch` | LedgerHistory.cpp:16  | Ledger hash mismatch count     |
-| `warn`                    | Logic.h:33            | Resource manager warning count |
-| `drop`                    | Logic.h:34            | Resource manager drop count    |
+| Prometheus Metric               | Source                | Description                    |
+| ------------------------------- | --------------------- | ------------------------------ |
+| `rpc_requests_total`            | ServerHandler.cpp:108 | Total RPC request count        |
+| `ledger_fetches_total`          | InboundLedgers.cpp:44 | Ledger fetch request count     |
+| `ledger_history_mismatch_total` | LedgerHistory.cpp:16  | Ledger hash mismatch count     |
+| `warn_total`                    | Logic.h:33            | Resource manager warning count |
+| `drop_total`                    | Logic.h:34            | Resource manager drop count    |
 
 #### Histograms
 
@@ -661,7 +661,7 @@ Ten dashboards are pre-provisioned in `docker/telemetry/grafana/dashboards/`:
 
 | Panel                       | Type       | PromQL                                                                                                                     | Labels Used              |
 | --------------------------- | ---------- | -------------------------------------------------------------------------------------------------------------------------- | ------------------------ |
-| RPC Request Rate by Command | timeseries | `sum by (command) (rate(span_calls_total{span_name=~"rpc.command.*"}[5m]))`                                                | `command`                |
+| RPC Request Rate by Command | timeseries | `sum by (command) (rate(span_calls_total{span_name=~"rpc.command.*"}[$__rate_interval]))`                                  | `command`                |
 | RPC Latency p95 by Command  | timeseries | `histogram_quantile(0.95, sum by (le, command) (rate(span_duration_milliseconds_bucket{span_name=~"rpc.command.*"}[5m])))` | `command`                |
 | RPC Error Rate              | bargauge   | Error spans / total spans × 100, grouped by `command`                                                                      | `command`, `status_code` |
 | RPC Latency Heatmap         | heatmap    | `sum(increase(span_duration_milliseconds_bucket{span_name=~"rpc.command.*"}[5m])) by (le)`                                 | `le` (bucket boundaries) |
@@ -674,10 +674,10 @@ Ten dashboards are pre-provisioned in `docker/telemetry/grafana/dashboards/`:
 
 | Panel                              | Type           | PromQL                                                                                       | Labels Used                         |
 | ---------------------------------- | -------------- | -------------------------------------------------------------------------------------------- | ----------------------------------- |
-| Transaction Processing Rate        | timeseries     | `rate(span_calls_total{span_name="tx.process"}[5m])` and `tx.receive`                        | `span_name`                         |
+| Transaction Processing Rate        | timeseries     | `rate(span_calls_total{span_name="tx.process"}[$__rate_interval])` and `tx.receive`          | `span_name`                         |
 | Transaction Processing Latency     | timeseries     | `histogram_quantile(0.95 / 0.50, ... {span_name="tx.process"})`                              | —                                   |
-| Transaction Path Distribution      | piechart       | `sum by (local) (increase(span_calls_total{span_name="tx.process"}[5m]))`                    | `local`                             |
-| Transaction Receive vs Suppressed  | timeseries     | `rate(span_calls_total{span_name="tx.receive"}[5m])`                                         | —                                   |
+| Transaction Path Distribution      | piechart       | `sum by (local) (increase(span_calls_total{span_name="tx.process"}[$__rate_interval]))`      | `local`                             |
+| Transaction Receive vs Suppressed  | timeseries     | `rate(span_calls_total{span_name="tx.receive"}[$__rate_interval])`                           | —                                   |
 | TX Processing Duration Heatmap     | heatmap        | `tx.process` histogram buckets                                                               | `le`                                |
 | TX Apply Duration per Ledger       | timeseries     | p95/p50 of `tx.apply`                                                                        | —                                   |
 | TX Apply Failed Rate               | stat           | `rate(span_calls_total{span_name="tx.transactor",stage="apply",ter_result!~"tesSUCCESS\|"})` | `stage`, `ter_result`               |
@@ -685,18 +685,18 @@ Ten dashboards are pre-provisioned in `docker/telemetry/grafana/dashboards/`:
 
 ### Consensus Health (`consensus-health`)
 
-| Panel                         | Type       | PromQL                                                                      | Labels Used      |
-| ----------------------------- | ---------- | --------------------------------------------------------------------------- | ---------------- |
-| Consensus Round Duration      | timeseries | `histogram_quantile(0.95 / 0.50, ... {span_name="consensus.accept"})`       | —                |
-| Consensus Proposals Sent Rate | timeseries | `rate(span_calls_total{span_name="consensus.proposal.send"}[5m])`           | —                |
-| Ledger Close Duration         | timeseries | `histogram_quantile(0.95, ... {span_name="consensus.ledger_close"})`        | —                |
-| Validation Send Rate          | stat       | `rate(span_calls_total{span_name="consensus.validation.send"}[5m])`         | —                |
-| Ledger Apply Duration         | timeseries | `histogram_quantile(0.95 / 0.50, ... {span_name="consensus.accept.apply"})` | —                |
-| Close Time Agreement          | timeseries | `rate(span_calls_total{span_name="consensus.accept.apply"}[5m])`            | —                |
-| Consensus Mode Over Time      | timeseries | `consensus.ledger_close` by `consensus_mode`                                | `consensus_mode` |
-| Accept vs Close Rate          | timeseries | `consensus.accept` vs `consensus.ledger_close` rate                         | —                |
-| Validation vs Close Rate      | timeseries | `consensus.validation.send` vs `consensus.ledger_close`                     | —                |
-| Accept Duration Heatmap       | heatmap    | `consensus.accept` histogram buckets                                        | `le`             |
+| Panel                         | Type       | PromQL                                                                            | Labels Used      |
+| ----------------------------- | ---------- | --------------------------------------------------------------------------------- | ---------------- |
+| Consensus Round Duration      | timeseries | `histogram_quantile(0.95 / 0.50, ... {span_name="consensus.accept"})`             | —                |
+| Consensus Proposals Sent Rate | timeseries | `rate(span_calls_total{span_name="consensus.proposal.send"}[$__rate_interval])`   | —                |
+| Ledger Close Duration         | timeseries | `histogram_quantile(0.95, ... {span_name="consensus.ledger_close"})`              | —                |
+| Validation Send Rate          | stat       | `rate(span_calls_total{span_name="consensus.validation.send"}[$__rate_interval])` | —                |
+| Ledger Apply Duration         | timeseries | `histogram_quantile(0.95 / 0.50, ... {span_name="consensus.accept.apply"})`       | —                |
+| Close Time Agreement          | timeseries | `rate(span_calls_total{span_name="consensus.accept.apply"}[$__rate_interval])`    | —                |
+| Consensus Mode Over Time      | timeseries | `consensus.ledger_close` by `consensus_mode`                                      | `consensus_mode` |
+| Accept vs Close Rate          | timeseries | `consensus.accept` vs `consensus.ledger_close` rate                               | —                |
+| Validation vs Close Rate      | timeseries | `consensus.validation.send` vs `consensus.ledger_close`                           | —                |
+| Accept Duration Heatmap       | heatmap    | `consensus.accept` histogram buckets                                              | `le`             |
 
 ### Ledger Operations (`ledger-operations`)
 
@@ -739,7 +739,7 @@ Requires `trace_peer=1` in the `[telemetry]` config section.
 | FullBelowCache Size                    | timeseries | `node_family_full_below_cache_size`                                                | —           |
 | FullBelowCache Hit Rate                | gauge      | `node_family_full_below_cache_hit_rate`                                            | —           |
 | Ledger Publish Gap                     | stat       | `Published_Ledger_Age - Validated_Ledger_Age`                                      | —           |
-| State Duration Rate (Full vs Tracking) | timeseries | `rate(state_accounting_full_duration[5m]) / 1000000`                               | —           |
+| State Duration Rate (Full vs Tracking) | timeseries | `rate(state_accounting_full_duration[$__rate_interval]) / 1000000`                 | —           |
 | All Jobs Execution Time (Detail)       | timeseries | `histogram_quantile($quantile, rate(job_running_us_bucket[5m])) by job_type` — µs  | `quantile`  |
 | All Jobs Dequeue Wait (Detail)         | timeseries | `histogram_quantile($quantile, rate(job_queued_us_bucket[5m])) by job_type` — µs   | `quantile`  |
 
@@ -762,7 +762,7 @@ Requires `trace_peer=1` in the `[telemetry]` config section.
 
 | Panel                     | Type       | PromQL                                           | Labels Used |
 | ------------------------- | ---------- | ------------------------------------------------ | ----------- |
-| RPC Request Rate          | stat       | `rate(rpc_requests[5m])`                         | —           |
+| RPC Request Rate          | stat       | `rate(rpc_requests_total[$__rate_interval])`     | —           |
 | RPC Response Time         | timeseries | `histogram_quantile(0.95, rpc_time_bucket)`      | —           |
 | RPC Response Size         | timeseries | `histogram_quantile(0.95, rpc_size_bucket)`      | —           |
 | RPC Response Time Heatmap | heatmap    | `rpc_time_bucket`                                | —           |

From 44a9f7776ccfb17e1efa36b994a19c3e6623d478 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 20:49:03 +0100
Subject: [PATCH 15/38] docs(telemetry): record the new consensus phase-span
 attributes

The Phase 4a span table had no consensus.phase.open row at all and listed
only three attributes for consensus.establish. Add the row, the eight
attributes added on this branch, and the two label-valued ones to the
attribute inventory.

Leaves the pre-existing dotted names in the consensus.round row alone; they
predate the underscore convention and are not part of this change.
---
 OpenTelemetryPlan/Phase4_taskList.md | 17 ++++++++++-------
 1 file changed, 10 insertions(+), 7 deletions(-)

diff --git a/OpenTelemetryPlan/Phase4_taskList.md b/OpenTelemetryPlan/Phase4_taskList.md
index 028a7a7e85..2629ae82fd 100644
--- a/OpenTelemetryPlan/Phase4_taskList.md
+++ b/OpenTelemetryPlan/Phase4_taskList.md
@@ -162,6 +162,8 @@
 - `converge_percent` — on `consensus.establish`, `consensus.update_positions`, `consensus.check`
 - `tx_count` — on `consensus.accept.apply` span (in `doAccept()`)
 - `disputes_count` — on `consensus.update_positions` span (in `updateOurPositions()`)
+- `close_reason` — on `consensus.phase.open`, from `whyCloseLedger()` (`anomaly`, `others_closed`, `idle`, `normal`)
+- `close_time_avalanche_state` — on `consensus.establish`, terminal regime (`init`, `mid`, `late`, `stuck`)
 
 **Design notes**:
 
@@ -809,13 +811,14 @@ and OFF, and don't affect consensus timing.
 
 ### New Spans (Phase 4a)
 
-| Span Name                    | Location           | Key Attributes (actually set)                                                                                                 |
-| ---------------------------- | ------------------ | ----------------------------------------------------------------------------------------------------------------------------- |
-| `consensus.round`            | `RCLConsensus.cpp` | `xrpl.consensus.round_id`, `xrpl.consensus.ledger_id`, `xrpl.ledger.seq`, `xrpl.consensus.mode`, `trace_strategy`             |
-| `consensus.establish`        | `Consensus.h`      | `converge_percent`, `establish_count`, `proposers`                                                                            |
-| `consensus.update_positions` | `Consensus.h`      | `converge_percent`, `proposers`, `have_close_time_consensus`, `close_time_threshold`, `disputes_count`, `avalanche_threshold` |
-| `consensus.check`            | `Consensus.h`      | `agree_count`, `disagree_count`, `converge_percent`, `have_close_time_consensus`, `threshold_percent`, `consensus_result`     |
-| `consensus.mode_change`      | `RCLConsensus.cpp` | `mode_old`, `mode_new`                                                                                                        |
+| Span Name                    | Location           | Key Attributes (actually set)                                                                                                                                                                                             |
+| ---------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `consensus.round`            | `RCLConsensus.cpp` | `xrpl.consensus.round_id`, `xrpl.consensus.ledger_id`, `xrpl.ledger.seq`, `xrpl.consensus.mode`, `trace_strategy`                                                                                                         |
+| `consensus.phase.open`       | `Consensus.h`      | `start_reason`, `previous_close_agree`, `peer_positions_at_open`, `early_close_triggered` (at start); `open_duration_ms`, `peer_positions_at_close`, `tx_sets_acquired`, `close_reason`, `proposers_validated` (at close) |
+| `consensus.establish`        | `Consensus.h`      | `converge_percent`, `establish_count`, `proposers`, `disputes_count`, `close_time_avalanche_state`                                                                                                                        |
+| `consensus.update_positions` | `Consensus.h`      | `converge_percent`, `proposers`, `have_close_time_consensus`, `close_time_threshold`, `disputes_count`, `avalanche_threshold`                                                                                             |
+| `consensus.check`            | `Consensus.h`      | `agree_count`, `disagree_count`, `converge_percent`, `have_close_time_consensus`, `threshold_percent`, `consensus_result`                                                                                                 |
+| `consensus.mode_change`      | `RCLConsensus.cpp` | `mode_old`, `mode_new`                                                                                                                                                                                                    |
 
 ### New Events (Phase 4a)
 

From fd873b01897f5a495736bcbd5f7b14cea274acd5 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 20:49:18 +0100
Subject: [PATCH 16/38] docs(telemetry): document the establish-span attributes
 in the runbook

close_time_avalanche_state is new; the row also did not say that the other
three are rewritten on every convergence iteration, so a reader could not tell
that the exported value is the last one rather than a series.

This row is byte-identical on phases 5 through 10, so it is edited here and
merges forward. The consensus.phase.open row is empty until phase 9 and is
updated there instead.
---
 docs/telemetry-runbook.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md
index eb3a6e9c28..0a42eb3b96 100644
--- a/docs/telemetry-runbook.md
+++ b/docs/telemetry-runbook.md
@@ -117,7 +117,7 @@ lifecycle spans be joined to the ledger trace it targeted (`span.current_ledger_
 | `consensus.phase.open`         | Consensus.h      | --                                                                                                                                                                                                                           | Open phase duration (child of round)                                                                                                  |
 | `consensus.proposal.send`      | RCLConsensus.cpp | `consensus_round`, `is_bow_out`                                                                                                                                                                                              | Consensus proposal broadcast                                                                                                          |
 | `consensus.ledger_close`       | RCLConsensus.cpp | `ledger_seq`, `consensus_mode`                                                                                                                                                                                               | Ledger close event                                                                                                                    |
-| `consensus.establish`          | Consensus.h      | `converge_percent`, `establish_count`, `proposers`                                                                                                                                                                           | Establish phase duration (child of round)                                                                                             |
+| `consensus.establish`          | Consensus.h      | `converge_percent`, `establish_count`, `proposers`, `disputes_count` (all overwritten each iteration); `close_time_avalanche_state` (terminal regime, set once when the span ends)                                           | Establish phase duration (child of round)                                                                                             |
 | `consensus.update_positions`   | Consensus.h      | `converge_percent`, `proposers`, `disputes_count`                                                                                                                                                                            | Position update and dispute resolution (see Events below)                                                                             |
 | `consensus.check`              | Consensus.h      | `agree_count`, `disagree_count`, `converge_percent`, `have_close_time_consensus`, `threshold_percent`, `proposers_finished`, `consensus_stalled`, `establish_count`, `consensus_result`                                      | Consensus threshold check                                                                                                             |
 | `consensus.accept`             | RCLConsensus.cpp | `proposers`, `round_time_ms`, `quorum`, `disputes_count`, `consensus_state`                                                                                                                                                  | Ledger accepted by consensus                                                                                                          |

From a10a4e98dbb98c362c8f6f4af775741f03a1bac0 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 20:49:37 +0100
Subject: [PATCH 17/38] docs(telemetry): add the consensus phase-span
 attributes to the reference

Eight attributes added on phase 4 were missing from the attribute catalogue.
Seven are on consensus.phase.open, which had no entries at all; the eighth is
the terminal regime on consensus.establish.

Does not touch the neighbouring proposers_agreed row, which names an attribute
the code never sets -- pre-existing and outside this change.
---
 .../09-data-collection-reference.md           | 82 ++++++++++---------
 1 file changed, 45 insertions(+), 37 deletions(-)

diff --git a/OpenTelemetryPlan/09-data-collection-reference.md b/OpenTelemetryPlan/09-data-collection-reference.md
index 8bce3eca19..eb907904d5 100644
--- a/OpenTelemetryPlan/09-data-collection-reference.md
+++ b/OpenTelemetryPlan/09-data-collection-reference.md
@@ -346,43 +346,51 @@ Join a transaction's work to its ledger with `{span.current_ledger_seq=<N>}`.
 
 #### Consensus Attributes
 
-| Attribute                   | Type    | Set On                                                                                                                 | Description                                                           |
-| --------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
-| `consensus_ledger_id`       | string  | `consensus.round`                                                                                                      | Previous ledger hash (used for deterministic trace ID)                |
-| `ledger_seq`                | int64   | `consensus.round`, `consensus.ledger_close`, `consensus.accept`, `consensus.validation.send`, `consensus.accept.apply` | Ledger sequence number                                                |
-| `consensus_mode`            | string  | `consensus.round`, `consensus.proposal.send`, `consensus.ledger_close`                                                 | Node mode via `toDisplayString()`: `"Proposing"`, `"Observing"`, etc. |
-| `consensus_round`           | int64   | `consensus.proposal.send`                                                                                              | Consensus round number                                                |
-| `proposers`                 | int64   | `consensus.proposal.send`, `consensus.accept`                                                                          | Number of proposers in the round                                      |
-| `round_time_ms`             | int64   | `consensus.accept`, `consensus.accept.apply`                                                                           | Total consensus round duration in milliseconds                        |
-| `proposing`                 | boolean | `consensus.validation.send`                                                                                            | Whether this node was a proposer                                      |
-| `consensus_state`           | string  | `consensus.accept.apply`                                                                                               | Consensus outcome: `"finished"` or `"moved_on"`                       |
-| `close_time`                | int64   | `consensus.accept.apply`                                                                                               | Agreed-upon ledger close time (epoch seconds)                         |
-| `close_time_correct`        | boolean | `consensus.accept.apply`                                                                                               | Whether validators reached agreement on close time                    |
-| `close_resolution_ms`       | int64   | `consensus.accept.apply`                                                                                               | Close time rounding granularity in milliseconds                       |
-| `parent_close_time`         | int64   | `consensus.accept.apply`                                                                                               | Parent ledger's close time (epoch seconds)                            |
-| `close_time_self`           | int64   | `consensus.accept.apply`                                                                                               | This node's proposed close time                                       |
-| `close_time_vote_bins`      | string  | `consensus.accept.apply`                                                                                               | Histogram of close time votes from validators                         |
-| `resolution_direction`      | string  | `consensus.accept.apply`                                                                                               | Resolution change: `"increased"`, `"decreased"`, or `"unchanged"`     |
-| `converge_percent`          | int64   | `consensus.establish`                                                                                                  | Convergence percentage threshold                                      |
-| `establish_count`           | int64   | `consensus.establish`                                                                                                  | Number of establish iterations completed                              |
-| `proposers_agreed`          | int64   | `consensus.establish`                                                                                                  | Number of proposers that agreed on this round                         |
-| `avalanche_threshold`       | int64   | `consensus.update_positions`                                                                                           | Avalanche threshold for dispute resolution                            |
-| `close_time_threshold`      | int64   | `consensus.update_positions`                                                                                           | Close time agreement threshold                                        |
-| `have_close_time_consensus` | boolean | `consensus.update_positions`                                                                                           | Whether close time consensus has been reached                         |
-| `agree_count`               | int64   | `consensus.check`                                                                                                      | Number of proposers that agree with our position                      |
-| `disagree_count`            | int64   | `consensus.check`                                                                                                      | Number of proposers that disagree with our position                   |
-| `threshold_percent`         | int64   | `consensus.check`                                                                                                      | Required agreement threshold percentage                               |
-| `consensus_result`          | string  | `consensus.check`                                                                                                      | Check result: `"yes"`, `"no"`, or `"expired"`                         |
-| `quorum`                    | int64   | `consensus.check`                                                                                                      | Required quorum for validation                                        |
-| `validation_count`          | int64   | `consensus.check`                                                                                                      | Number of validations received                                        |
-| `trace_strategy`            | string  | `consensus.round`                                                                                                      | Trace sampling strategy used for this round                           |
-| `consensus_round_id`        | int64   | `consensus.round`                                                                                                      | Deterministic round identifier (previous ledger seq + 1)              |
-| `mode_old`                  | string  | `consensus.mode_change`                                                                                                | Previous consensus mode                                               |
-| `mode_new`                  | string  | `consensus.mode_change`                                                                                                | New consensus mode                                                    |
-| `tx_id`                     | string  | `consensus.update_positions`                                                                                           | Disputed transaction ID                                               |
-| `dispute_our_vote`          | boolean | `consensus.update_positions`                                                                                           | Our vote on the disputed transaction                                  |
-| `dispute_yays`              | int64   | `consensus.update_positions`                                                                                           | Number of proposers voting to include                                 |
-| `dispute_nays`              | int64   | `consensus.update_positions`                                                                                           | Number of proposers voting to exclude                                 |
+| Attribute                    | Type    | Set On                                                                                                                 | Description                                                           |
+| ---------------------------- | ------- | ---------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------- |
+| `consensus_ledger_id`        | string  | `consensus.round`                                                                                                      | Previous ledger hash (used for deterministic trace ID)                |
+| `ledger_seq`                 | int64   | `consensus.round`, `consensus.ledger_close`, `consensus.accept`, `consensus.validation.send`, `consensus.accept.apply` | Ledger sequence number                                                |
+| `consensus_mode`             | string  | `consensus.round`, `consensus.proposal.send`, `consensus.ledger_close`                                                 | Node mode via `toDisplayString()`: `"Proposing"`, `"Observing"`, etc. |
+| `consensus_round`            | int64   | `consensus.proposal.send`                                                                                              | Consensus round number                                                |
+| `proposers`                  | int64   | `consensus.proposal.send`, `consensus.accept`                                                                          | Number of proposers in the round                                      |
+| `round_time_ms`              | int64   | `consensus.accept`, `consensus.accept.apply`                                                                           | Total consensus round duration in milliseconds                        |
+| `proposing`                  | boolean | `consensus.validation.send`                                                                                            | Whether this node was a proposer                                      |
+| `consensus_state`            | string  | `consensus.accept.apply`                                                                                               | Consensus outcome: `"finished"` or `"moved_on"`                       |
+| `close_time`                 | int64   | `consensus.accept.apply`                                                                                               | Agreed-upon ledger close time (epoch seconds)                         |
+| `close_time_correct`         | boolean | `consensus.accept.apply`                                                                                               | Whether validators reached agreement on close time                    |
+| `close_resolution_ms`        | int64   | `consensus.accept.apply`                                                                                               | Close time rounding granularity in milliseconds                       |
+| `parent_close_time`          | int64   | `consensus.accept.apply`                                                                                               | Parent ledger's close time (epoch seconds)                            |
+| `close_time_self`            | int64   | `consensus.accept.apply`                                                                                               | This node's proposed close time                                       |
+| `close_time_vote_bins`       | string  | `consensus.accept.apply`                                                                                               | Histogram of close time votes from validators                         |
+| `resolution_direction`       | string  | `consensus.accept.apply`                                                                                               | Resolution change: `"increased"`, `"decreased"`, or `"unchanged"`     |
+| `converge_percent`           | int64   | `consensus.establish`                                                                                                  | Convergence percentage threshold                                      |
+| `establish_count`            | int64   | `consensus.establish`                                                                                                  | Number of establish iterations completed                              |
+| `proposers_agreed`           | int64   | `consensus.establish`                                                                                                  | Number of proposers that agreed on this round                         |
+| `avalanche_threshold`        | int64   | `consensus.update_positions`                                                                                           | Avalanche threshold for dispute resolution                            |
+| `start_reason`               | string  | `consensus.phase.open`                                                                                                 | Entry path: `"initial"` or `"recovered"`                              |
+| `previous_close_agree`       | bool    | `consensus.phase.open`                                                                                                 | Whether the prior ledger's close time was agreed                      |
+| `peer_positions_at_open`     | int64   | `consensus.phase.open`                                                                                                 | Positions held after buffered proposals are replayed                  |
+| `early_close_triggered`      | bool    | `consensus.phase.open`                                                                                                 | Round skipped the timer because peers had already closed              |
+| `tx_sets_acquired`           | int64   | `consensus.phase.open`                                                                                                 | Peer transaction sets held at close, excluding our own                |
+| `close_reason`               | string  | `consensus.phase.open`                                                                                                 | `"anomaly"`, `"others_closed"`, `"idle"`, or `"normal"`               |
+| `proposers_validated`        | int64   | `consensus.phase.open`                                                                                                 | Trusted validators of the previous ledger, at close                   |
+| `close_time_avalanche_state` | string  | `consensus.establish`                                                                                                  | Terminal regime: `"init"`, `"mid"`, `"late"`, or `"stuck"`            |
+| `close_time_threshold`       | int64   | `consensus.update_positions`                                                                                           | Close time agreement threshold                                        |
+| `have_close_time_consensus`  | boolean | `consensus.update_positions`                                                                                           | Whether close time consensus has been reached                         |
+| `agree_count`                | int64   | `consensus.check`                                                                                                      | Number of proposers that agree with our position                      |
+| `disagree_count`             | int64   | `consensus.check`                                                                                                      | Number of proposers that disagree with our position                   |
+| `threshold_percent`          | int64   | `consensus.check`                                                                                                      | Required agreement threshold percentage                               |
+| `consensus_result`           | string  | `consensus.check`                                                                                                      | Check result: `"yes"`, `"no"`, or `"expired"`                         |
+| `quorum`                     | int64   | `consensus.check`                                                                                                      | Required quorum for validation                                        |
+| `validation_count`           | int64   | `consensus.check`                                                                                                      | Number of validations received                                        |
+| `trace_strategy`             | string  | `consensus.round`                                                                                                      | Trace sampling strategy used for this round                           |
+| `consensus_round_id`         | int64   | `consensus.round`                                                                                                      | Deterministic round identifier (previous ledger seq + 1)              |
+| `mode_old`                   | string  | `consensus.mode_change`                                                                                                | Previous consensus mode                                               |
+| `mode_new`                   | string  | `consensus.mode_change`                                                                                                | New consensus mode                                                    |
+| `tx_id`                      | string  | `consensus.update_positions`                                                                                           | Disputed transaction ID                                               |
+| `dispute_our_vote`           | boolean | `consensus.update_positions`                                                                                           | Our vote on the disputed transaction                                  |
+| `dispute_yays`               | int64   | `consensus.update_positions`                                                                                           | Number of proposers voting to include                                 |
+| `dispute_nays`               | int64   | `consensus.update_positions`                                                                                           | Number of proposers voting to exclude                                 |
 
 **Tempo query**: `{span.consensus_mode="Proposing"}` to find rounds where node was proposing.
 

From 5506575d5cd2318250d3665383dde83118701f5a Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 20:49:53 +0100
Subject: [PATCH 18/38] docs(telemetry): document the open-phase span
 attributes in the runbook

The row listed only the two end-of-phase attributes. Add the four set at span
creation and the three set at the close decision.

Records what the previous wording implied but did not state: the end-of-phase
attributes are absent when the round is recovered by handleWrongLedger or
driven by simulate(), because neither reaches closeLedger(). Any average over
open_duration_ms silently excludes those rounds.

Edited here rather than on phase 5, where this cell is empty and would conflict
on the way up.
---
 docs/telemetry-runbook.md | 30 +++++++++++++++---------------
 1 file changed, 15 insertions(+), 15 deletions(-)

diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md
index efd6a84277..49d0aabbf6 100644
--- a/docs/telemetry-runbook.md
+++ b/docs/telemetry-runbook.md
@@ -324,21 +324,21 @@ read it from the parent rather than filtering `tx.apply` on it.
 
 ### Consensus Spans
 
-| Span Name                      | Source File      | Attributes                                                                                                                                                                                                                                              | Description                                                                                                                           |
-| ------------------------------ | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
-| `consensus.round`              | RCLConsensus.cpp | `consensus_ledger_id`, `ledger_seq`, `consensus_mode`, `trace_strategy`, `consensus_round_id`                                                                                                                                                           | Root span for a consensus round (deterministic or random trace ID)                                                                    |
-| `consensus.phase.open`         | Consensus.h      | `open_duration_ms`, `peer_positions_at_close` (both only if the span is still live at `closeLedger()`)                                                                                                                                                  | Open phase duration (child of round)                                                                                                  |
-| `consensus.proposal.send`      | RCLConsensus.cpp | `consensus_round`, `is_bow_out`                                                                                                                                                                                                                         | Consensus proposal broadcast                                                                                                          |
-| `consensus.ledger_close`       | RCLConsensus.cpp | `ledger_seq`, `consensus_mode`                                                                                                                                                                                                                          | Ledger close event                                                                                                                    |
-| `consensus.establish`          | Consensus.h      | `converge_percent`, `establish_count`, `proposers`                                                                                                                                                                                                      | Establish phase duration (child of round)                                                                                             |
-| `consensus.update_positions`   | Consensus.h      | `converge_percent`, `proposers`, `disputes_count`, `avalanche_threshold` (only when peer positions exist), `have_close_time_consensus`, `close_time_threshold`                                                                                          | Position update and dispute resolution (see Events below)                                                                             |
-| `consensus.check`              | Consensus.h      | `agree_count`, `disagree_count`, `converge_percent`, `have_close_time_consensus`, `threshold_percent`, `proposers_finished`, `consensus_stalled`, `establish_count`, `consensus_result`                                                                 | Consensus threshold check                                                                                                             |
-| `consensus.accept`             | RCLConsensus.cpp | `proposers`, `round_time_ms`, `quorum`, `disputes_count`, `consensus_state`                                                                                                                                                                             | Ledger accepted by consensus                                                                                                          |
-| `consensus.accept.apply`       | RCLConsensus.cpp | `ledger_seq`, `close_time`, `close_time_correct`, `close_resolution_ms`, `consensus_state`, `proposing`, `round_time_ms`, `parent_close_time`, `close_time_self`, `close_time_vote_bins`, `resolution_direction`, `tx_count`, `disputes_resolved_count` | Ledger application with close time details (see Events below)                                                                         |
-| `consensus.validation.send`    | RCLConsensus.cpp | `ledger_seq`, `proposing`, `ledger_hash`, `full_validation`, `validation_sign_time`                                                                                                                                                                     | Validation sent after accept (follows-from link)                                                                                      |
-| `consensus.mode_change`        | RCLConsensus.cpp | `mode_old`, `mode_new`                                                                                                                                                                                                                                  | Consensus mode transition                                                                                                             |
-| `consensus.proposal.receive`   | PeerImp.cpp      | `proposal_trusted`, `consensus_round`, `prev_ledger_prefix`, `position_hash_prefix`                                                                                                                                                                     | Proposal received from peer (extracts parent context from TraceContext when present; falls back to standalone span for older peers)   |
-| `consensus.validation.receive` | PeerImp.cpp      | `validation_trusted`, `ledger_seq` (only when the validation carries `sfLedgerSequence`), `full_validation`, `validation_sign_time`                                                                                                                     | Validation received from peer (extracts parent context from TraceContext when present; falls back to standalone span for older peers) |
+| Span Name                      | Source File      | Attributes                                                                                                                                                                                                                                                                                                               | Description                                                                                                                           |
+| ------------------------------ | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------- |
+| `consensus.round`              | RCLConsensus.cpp | `consensus_ledger_id`, `ledger_seq`, `consensus_mode`, `trace_strategy`, `consensus_round_id`                                                                                                                                                                                                                            | Root span for a consensus round (deterministic or random trace ID)                                                                    |
+| `consensus.phase.open`         | Consensus.h      | `start_reason`, `previous_close_agree`, `peer_positions_at_open`, `early_close_triggered` (at start); `open_duration_ms`, `peer_positions_at_close`, `tx_sets_acquired`, `close_reason`, `proposers_validated` (at close, so all five are absent on a recovered or simulated round, which never reaches `closeLedger()`) | Open phase duration (child of round)                                                                                                  |
+| `consensus.proposal.send`      | RCLConsensus.cpp | `consensus_round`, `is_bow_out`                                                                                                                                                                                                                                                                                          | Consensus proposal broadcast                                                                                                          |
+| `consensus.ledger_close`       | RCLConsensus.cpp | `ledger_seq`, `consensus_mode`                                                                                                                                                                                                                                                                                           | Ledger close event                                                                                                                    |
+| `consensus.establish`          | Consensus.h      | `converge_percent`, `establish_count`, `proposers`                                                                                                                                                                                                                                                                       | Establish phase duration (child of round)                                                                                             |
+| `consensus.update_positions`   | Consensus.h      | `converge_percent`, `proposers`, `disputes_count`, `avalanche_threshold` (only when peer positions exist), `have_close_time_consensus`, `close_time_threshold`                                                                                                                                                           | Position update and dispute resolution (see Events below)                                                                             |
+| `consensus.check`              | Consensus.h      | `agree_count`, `disagree_count`, `converge_percent`, `have_close_time_consensus`, `threshold_percent`, `proposers_finished`, `consensus_stalled`, `establish_count`, `consensus_result`                                                                                                                                  | Consensus threshold check                                                                                                             |
+| `consensus.accept`             | RCLConsensus.cpp | `proposers`, `round_time_ms`, `quorum`, `disputes_count`, `consensus_state`                                                                                                                                                                                                                                              | Ledger accepted by consensus                                                                                                          |
+| `consensus.accept.apply`       | RCLConsensus.cpp | `ledger_seq`, `close_time`, `close_time_correct`, `close_resolution_ms`, `consensus_state`, `proposing`, `round_time_ms`, `parent_close_time`, `close_time_self`, `close_time_vote_bins`, `resolution_direction`, `tx_count`, `disputes_resolved_count`                                                                  | Ledger application with close time details (see Events below)                                                                         |
+| `consensus.validation.send`    | RCLConsensus.cpp | `ledger_seq`, `proposing`, `ledger_hash`, `full_validation`, `validation_sign_time`                                                                                                                                                                                                                                      | Validation sent after accept (follows-from link)                                                                                      |
+| `consensus.mode_change`        | RCLConsensus.cpp | `mode_old`, `mode_new`                                                                                                                                                                                                                                                                                                   | Consensus mode transition                                                                                                             |
+| `consensus.proposal.receive`   | PeerImp.cpp      | `proposal_trusted`, `consensus_round`, `prev_ledger_prefix`, `position_hash_prefix`                                                                                                                                                                                                                                      | Proposal received from peer (extracts parent context from TraceContext when present; falls back to standalone span for older peers)   |
+| `consensus.validation.receive` | PeerImp.cpp      | `validation_trusted`, `ledger_seq` (only when the validation carries `sfLedgerSequence`), `full_validation`, `validation_sign_time`                                                                                                                                                                                      | Validation received from peer (extracts parent context from TraceContext when present; falls back to standalone span for older peers) |
 
 #### Consensus Span Events
 

From ff8629bb116a64ccd748ec3a97513a2a5022733d Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 20:50:21 +0100
Subject: [PATCH 19/38] fix(telemetry): drop the inert insight prefix and its
 false comment

Every generated node cfg carried prefix=xrpld under a comment claiming it
"matches the OTel resource service name and the metric names the dashboards
query". Both halves are false.

Verified inert before removing: CollectorManager.cpp reads the key on the OTel
path and passes it to OTelCollector::New, but the only use of prefix_ anywhere
in OTelCollector.cpp is the startup log line. formatName() -- the single funnel
for every instrument name -- only lowercases the name and turns dots and
spaces into underscores; it never reads prefix_. So exported names carry no
prefix at all. expected_metrics.json's own description records this ("Metric
names have no prefix (the xrpld_ prefix was removed)") and 488 live metric
names confirmed it: jobq_job_count, rpc_requests_total, total_bytes_in.

A reader trusting the comment would look for xrpld_jobq_job_count and find
nothing.

The replacement comment states what is true and checkable: the collector
declares no statsd receiver (its metrics pipeline is [otlp, spanmetrics],
confirmed in otel-collector-config.yaml), so beast::insight must export over
OTLP for system metrics to reach Prometheus at all; server=otel is the only
load-bearing key; exported names carry no prefix.

Metric names, series and dashboards are unchanged. The one observable
difference is the OTelCollector startup log line, which now prints an empty
prefix.

Also updated workload/README.md, which repeated the same prefix=xrpld claim
and would have been left describing a cfg key that no longer exists, and made
the template header state the sync obligation explicitly -- nothing reads that
file, so nothing catches it drifting from the cfg the runner generates.
---
 docker/telemetry/workload/README.md            |  2 +-
 .../telemetry/workload/run-full-validation.sh  | 13 +++++++++----
 .../workload/xrpld-validator.cfg.template      | 18 ++++++++++++++----
 3 files changed, 24 insertions(+), 9 deletions(-)

diff --git a/docker/telemetry/workload/README.md b/docker/telemetry/workload/README.md
index 9d96499051..4c0ce078ca 100644
--- a/docker/telemetry/workload/README.md
+++ b/docker/telemetry/workload/README.md
@@ -469,7 +469,7 @@ absence is a skip rather than a failure:
 The orchestrator (`run-full-validation.sh`) generates node configs with:
 
 - `[telemetry] enabled=1` with all five trace categories: `trace_rpc`, `trace_transactions`, `trace_consensus`, `trace_peer`, `trace_ledger`
-- `[insight] server=otel` with `endpoint=http://localhost:4318/v1/metrics` and `prefix=xrpld` — `beast::insight` metrics reach Prometheus over OTLP, because the collector declares no `statsd` receiver
+- `[insight] server=otel` with `endpoint=http://localhost:4318/v1/metrics` — `beast::insight` metrics reach Prometheus over OTLP, because the collector declares no `statsd` receiver. No `prefix` is set: it would be inert, since `OTelCollector` applies no prefix to instrument names and exported names are the lowercased raw names (`jobq_job_count`, not `xrpld_jobq_job_count`)
 - `[signing_support] true` — required for `tx_submitter.py` to submit signed transactions via WebSocket
 - `[ips]` (not `[ips_fixed]`) — ensures peer connections are counted in the PeerFinder active-peer gauges, exported as `peer_finder_active_inbound_peers` / `peer_finder_active_outbound_peers` (fixed peers are excluded from these counters by design). The `beast::insight` group/name pair is `Peer_Finder` / `Active_Inbound_Peers`; `formatName()` lowercases it for export.
 
diff --git a/docker/telemetry/workload/run-full-validation.sh b/docker/telemetry/workload/run-full-validation.sh
index ef3cc6cb45..5acbbfbfc5 100755
--- a/docker/telemetry/workload/run-full-validation.sh
+++ b/docker/telemetry/workload/run-full-validation.sh
@@ -329,12 +329,17 @@ trace_ledger=1
 
 [insight]
 # Native OTel metrics via OTLP/HTTP. The collector has no StatsD receiver
-# (metrics pipeline is [otlp, spanmetrics]), so beast::insight must export
-# over OTLP for system metrics to reach Prometheus. prefix=xrpld matches the
-# OTel resource service name and the metric names the dashboards query.
+# (its metrics pipeline is [otlp, spanmetrics]), so beast::insight must export
+# over OTLP for system metrics to reach Prometheus at all. server=otel is the
+# only load-bearing key here -- it selects the OTel collector; endpoint is
+# read but already matches the built-in default.
+#
+# No prefix is set on purpose. It would be inert: OTelCollector applies no
+# prefix to instrument names, so exported names are the lowercased raw names
+# (jobq_job_count, rpc_requests_total, total_bytes_in). The service is
+# identified by the OTel resource service.name, not by a name prefix.
 server=otel
 endpoint=http://localhost:4318/v1/metrics
-prefix=xrpld
 
 [rpc_startup]
 # info is required here, not cosmetic, and is the minimum that works. The
diff --git a/docker/telemetry/workload/xrpld-validator.cfg.template b/docker/telemetry/workload/xrpld-validator.cfg.template
index 875abd63ab..623b781707 100644
--- a/docker/telemetry/workload/xrpld-validator.cfg.template
+++ b/docker/telemetry/workload/xrpld-validator.cfg.template
@@ -4,6 +4,11 @@
 # cfg inline. Kept as the reference layout for a validator run as a container,
 # whose entrypoint would substitute the placeholders below.
 #
+# Because nothing reads this file, nothing catches it drifting from the cfg the
+# runner actually generates. Any change to the [telemetry], [insight] or
+# [rpc_startup] stanzas here must be mirrored in run-full-validation.sh, and
+# vice versa.
+#
 # Placeholders:
 #   {{NODE_INDEX}}        — Node number (1-based)
 #   {{RPC_PORT}}          — HTTP RPC port
@@ -90,13 +95,18 @@ trace_peer=1
 trace_ledger=1
 
 # --- Native OTel metrics (beast::insight over OTLP/HTTP) ---
-# The collector has no StatsD receiver (metrics pipeline is [otlp, spanmetrics]),
-# so beast::insight exports natively over OTLP. prefix=xrpld matches the OTel
-# resource service name and the metric names the dashboards query.
+# The collector has no StatsD receiver (its metrics pipeline is
+# [otlp, spanmetrics]), so beast::insight must export natively over OTLP for
+# system metrics to reach Prometheus at all. `server=otel` is the only
+# load-bearing key here -- it selects the OTel collector.
+#
+# No `prefix` is set on purpose. It would be inert: OTelCollector applies no
+# prefix to instrument names, so exported names are the lowercased raw names
+# (`jobq_job_count`, `rpc_requests_total`, `total_bytes_in`). The service is
+# identified by the OTel resource `service.name`, not by a name prefix.
 [insight]
 server=otel
 endpoint={{OTEL_METRICS_ENDPOINT}}
-prefix=xrpld
 
 [rpc_startup]
 { "command": "log_level", "severity": "{{LOG_LEVEL}}" }

From ea4282f6e00ae5cccc9c9f37b030bab523f1c071 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 20:24:27 +0100
Subject: [PATCH 20/38] docs(telemetry): correct the stale metric-gate counts
 in the runbook

The "What Gets Validated" table claimed "58 metrics in 23 categories". Both
numbers were stale: generalising required_labels and adding the asserted
io_latency group changed the gate to 66 checks across 24 asserting categories
(61 metric names plus 5 required_labels checks).

Derived from the inventory rather than counted by hand -- ran the validator's
own _metric_check_targets() against expected_metrics.json:
  metric names = 61 | label checks = 5 | TOTAL = 66
  asserting groups = 24
  label checks per group = {'spanmetrics': 4, 'job_queue': 1}

Also records that labels are gated at all. They were declared in the contract
but nothing read the key until the check was generalised, so four spanmetrics
labels were documented as required while going unverified -- the table
described existence checks only and gave no hint that a label regression was
now catchable.

The other rows in the same table were re-derived and are still accurate:
spans 41 total / 26 required / 15 optional, parity 6 span attributes + 4 value
bounds = 10, dashboards 15 uids against 15 provisioned files, logs 2.
---
 docs/telemetry-runbook.md | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md
index 37fe574984..ce1089b57f 100644
--- a/docs/telemetry-runbook.md
+++ b/docs/telemetry-runbook.md
@@ -3598,13 +3598,13 @@ The counts are not hard-coded in the validator — it iterates the inventory fil
 so those files are authoritative. The figures below are the inventory as it
 stands today.
 
-| Category   | Checks                                                                                                                              | Description                                                                                                                                                                                                                                                                                                                                               |
-| ---------- | ----------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| Spans      | Every **required** entry in `expected_spans.json` — 41 span types at the time of writing: 26 required, 15 marked `"optional": true` | Span name found in Tempo carrying its `required_attributes`, plus the declared parent-child relationships. An `"optional": true` entry that does not fire is recorded as a skip, not a failure — it needs traffic the harness may not generate (HTTP/JSON-RPC client, gRPC client, missing-ledger fetch, mode transitions).                               |
-| Metrics    | Every entry in every asserted category of `expected_metrics.json` — 58 metrics in 23 categories at the time of writing              | SpanMetrics, `beast::insight` gauges/counters exported over OTLP, and the `MetricsRegistry` OTLP metrics. Each must have > 0 Prometheus series; none are optional. The separate `not_asserted` group lists metrics deliberately left out of the gate because they are workload-gated or defect-gated; it has no `metrics` key, so the validator skips it. |
-| Logs       | 2 checks                                                                                                                            | `trace_id`/`span_id` present in Loki, and a Tempo trace id resolves in Loki. Skipped in CI, which runs `--skip-loki`.                                                                                                                                                                                                                                     |
-| Parity     | 10 checks                                                                                                                           | 6 span attributes the external-parity dashboard panels read, plus 4 metric value-sanity bounds.                                                                                                                                                                                                                                                           |
-| Dashboards | Every uid in `expected_metrics.json` under `grafana_dashboards.uids` — currently all 15 provisioned dashboards                      | Each listed dashboard loads and reports a panel count. This is a provisioning check only: it does **not** execute the panels' queries, so a dashboard can pass while individual panels render empty. `log-derived-insights` is Loki-backed, so under `--skip-loki` only its provisioning is meaningfully covered.                                         |
+| Category   | Checks                                                                                                                                                                               | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
+| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
+| Spans      | Every **required** entry in `expected_spans.json` — 41 span types at the time of writing: 26 required, 15 marked `"optional": true`                                                  | Span name found in Tempo carrying its `required_attributes`, plus the declared parent-child relationships. An `"optional": true` entry that does not fire is recorded as a skip, not a failure — it needs traffic the harness may not generate (HTTP/JSON-RPC client, gRPC client, missing-ledger fetch, mode transitions).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
+| Metrics    | Every entry in every asserted category of `expected_metrics.json` — 66 checks across 24 asserting categories at the time of writing: 61 metric names plus 5 `required_labels` checks | SpanMetrics, `beast::insight` gauges/counters exported over OTLP, and the `MetricsRegistry` OTLP metrics. Each must have > 0 Prometheus series; none are optional. A category may also declare `required_labels`, and each label there becomes one additional check that at least one of that category's series carries it with a non-empty value (matched as `<label>!=""`, because Prometheus cannot distinguish an absent label from an empty one). Those labels were declared but never actually read until the check was generalised, so they were documented as required while going unverified; `spanmetrics` contributes 4 and `job_queue` 1. The separate `not_asserted` group lists metrics deliberately left out of the gate because they are workload-gated or defect-gated; it has neither a `metrics` nor a `required_labels` key, so the validator skips it entirely. |
+| Logs       | 2 checks                                                                                                                                                                             | `trace_id`/`span_id` present in Loki, and a Tempo trace id resolves in Loki. Skipped in CI, which runs `--skip-loki`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
+| Parity     | 10 checks                                                                                                                                                                            | 6 span attributes the external-parity dashboard panels read, plus 4 metric value-sanity bounds.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
+| Dashboards | Every uid in `expected_metrics.json` under `grafana_dashboards.uids` — currently all 15 provisioned dashboards                                                                       | Each listed dashboard loads and reports a panel count. This is a provisioning check only: it does **not** execute the panels' queries, so a dashboard can pass while individual panels render empty. `log-derived-insights` is Loki-backed, so under `--skip-loki` only its provisioning is meaningfully covered.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
 
 ### Running Individual Tools
 

From 98ba282854725c545dfc192b2fa99a8f4a961c8d Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 20:49:07 +0100
Subject: [PATCH 21/38] fix(telemetry): make the integration test correlate by
 construction, record the baseline log-level coupling

Three related follow-ups to running the workload at info.

integration-test.sh has its own log-trace correlation check that the workload
validator knows nothing about: check_log_correlation() greps each node's
debug.log for "trace_id=<hex> span_id=<hex>" and fails when it finds none, then
cross-checks a sample id against Tempo. At warning it had no guaranteed source.
The only warn-or-worse statement inside the activated accept scope is
RCLConsensus.cpp:671, which fires solely when a transaction throws, so the
check was passing incidentally -- helped by scanning whole files with no time
window. Raising it to info gives it the same guarantee the workload now has:
the consensus accept pair, one branch of which fires every accepted round.
Safe here because this script captures no latency baseline, so there is nothing
for the extra log I/O to contaminate.

baselines/README.md now records that the committed baseline is only valid at
the log level the harness generates. Logging is synchronous and several gated
spans contain log statements -- ledger.build has BuildLedger.cpp:81, and
consensus.accept has RCLConsensus.cpp:655/663/686 with :663 logging once per
transaction -- so the configured level is part of the measurement. Moving it
inflates or deflates the quantiles the gate reads without ever reporting a
regression, because the baseline moves with it. Changing the level therefore
requires re-capturing the baseline.

benchmark.sh keeps warning and keeps prefix=xrpld, and now says why. It
measures telemetry overhead as a delta between a telemetry-off and a
telemetry-on arm, so extra synchronous log I/O would inflate both arms and the
thresholds gate the result. The comment exists to stop a future reader
"aligning" it with the workload harness and quietly degrading the measurement.
---
 docker/telemetry/integration-test.sh          | 11 +++++++++-
 docker/telemetry/workload/baselines/README.md | 21 +++++++++++++++++++
 docker/telemetry/workload/benchmark.sh        |  7 +++++++
 3 files changed, 38 insertions(+), 1 deletion(-)

diff --git a/docker/telemetry/integration-test.sh b/docker/telemetry/integration-test.sh
index 2254ac4399..090ac28624 100755
--- a/docker/telemetry/integration-test.sh
+++ b/docker/telemetry/integration-test.sh
@@ -403,7 +403,16 @@ prefix=rippled
 service_instance_id=Node-${i}
 
 [rpc_startup]
-{ "command": "log_level", "severity": "warning" }
+# info, not warning: check_log_correlation() below greps these nodes' debug.log
+# for "trace_id=<hex> span_id=<hex>" and fails if it finds none. A line only
+# carries trace context when it is emitted while a span is current, and the one
+# line guaranteed to be is the consensus accept pair in RCLConsensus.cpp, which
+# logs at info inside the activation doAccept makes over its whole body. At
+# warning that pair is suppressed, leaving only whatever warn-or-worse line
+# happens to fire inside some active span -- so the check passed incidentally
+# rather than by construction. debug is deliberately not used here either, to
+# stay consistent with the workload harness.
+{ "command": "log_level", "severity": "info" }
 
 [ssl_verify]
 0
diff --git a/docker/telemetry/workload/baselines/README.md b/docker/telemetry/workload/baselines/README.md
index 4adfbe9b18..c80e1bf9dc 100644
--- a/docker/telemetry/workload/baselines/README.md
+++ b/docker/telemetry/workload/baselines/README.md
@@ -70,6 +70,27 @@ new numbers should become the norm, open a PR pasting the fresh timings into
 Do **not** edit `baseline-timings.json` by hand outside of this process — every entry
 should trace back to a real CI run so variance characteristics are preserved.
 
+## The baseline is only valid at the log level it was captured at
+
+Every timing here is coupled to the `log_level` that `run-full-validation.sh` writes into
+each node's `[rpc_startup]` stanza. Logging is **synchronous**, and several of the gated
+spans contain log statements, so the configured level is part of the measurement:
+
+- `ledger.build` contains [`BuildLedger.cpp:81`](../../../../src/xrpld/app/ledger/detail/BuildLedger.cpp#L81) (debug).
+- `consensus.accept` contains [RCLConsensus.cpp:655/663/686](../../../../src/xrpld/app/consensus/RCLConsensus.cpp#L663) (debug) — `:663` logs **once per transaction** in the canonical set.
+- `tx.apply` and the other `spans.names` entries in [`../regression-metrics.json`](../regression-metrics.json) are affected the same way.
+
+Raising the level admits more of those statements and inflates the p50/p95/p99 of the very
+spans the gate measures; lowering it deflates them. Neither shows up as a regression, because
+the baseline moves with it — the gate simply starts measuring a different configuration.
+
+**Changing the workload log level therefore invalidates this baseline and requires
+re-capturing it.** Treat it exactly like a deliberate performance change: follow
+[Refreshing the baseline](#refreshing-the-baseline), and note the level change in the PR so
+the reviewer knows why the numbers moved. In particular, do not capture a baseline while the
+harness is running at `debug` — see the runbook's "Why not `debug`" note; if you need
+debug-level detail, enable it per partition **after** the baseline exists.
+
 ## Schema
 
 ```json
diff --git a/docker/telemetry/workload/benchmark.sh b/docker/telemetry/workload/benchmark.sh
index 6c87a6b121..7518e52485 100755
--- a/docker/telemetry/workload/benchmark.sh
+++ b/docker/telemetry/workload/benchmark.sh
@@ -238,6 +238,13 @@ ${ips_fixed}
 ${telemetry_section}
 
 [rpc_startup]
+# Deliberately diverges from run-full-validation.sh (which uses info) and keeps
+# prefix=xrpld above. This script measures telemetry OVERHEAD as the delta
+# between a telemetry-off and a telemetry-on arm, so it needs a quiet, identical
+# baseline in both arms rather than good observability. Logging is synchronous, so
+# admitting more log lines would add I/O to both arms and inflate the CPU,
+# memory and latency figures the thresholds gate. Do not "align" this with the
+# workload harness -- that would silently degrade the measurement.
 { "command": "log_level", "severity": "warning" }
 
 [ssl_verify]

From 4c33ffb9caa78498a619cbb564b2bf7ad1096736 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 21:08:43 +0100
Subject: [PATCH 22/38] docs(telemetry): retire the rpc_size
 instrument-mismatch warning

---
 OpenTelemetryPlan/09-data-collection-reference.md      | 10 +++-------
 .../telemetry/grafana/dashboards/rpc-pathfinding.json  |  2 +-
 docker/telemetry/workload/benchmark.sh                 | 10 ++++++----
 3 files changed, 10 insertions(+), 12 deletions(-)

diff --git a/OpenTelemetryPlan/09-data-collection-reference.md b/OpenTelemetryPlan/09-data-collection-reference.md
index 004e6dd7d2..03b66bb9c9 100644
--- a/OpenTelemetryPlan/09-data-collection-reference.md
+++ b/OpenTelemetryPlan/09-data-collection-reference.md
@@ -677,21 +677,17 @@ prefix=xrpld
 | Prometheus Metric | Source File       | Unit | Description                    |
 | ----------------- | ----------------- | ---- | ------------------------------ |
 | `rpc_time`        | ServerHandler.cpp | ms   | RPC response time distribution |
-| `rpc_size`        | ServerHandler.cpp | ms\* | RPC response size (see note)   |
+| `rpc_size`        | ServerHandler.cpp | By\* | RPC response size (see note)   |
 | `ios_latency`     | Application.cpp   | ms   | I/O service loop latency       |
 | `pathfind_fast`   | PathRequests.h    | ms   | Fast pathfinding duration      |
 | `pathfind_full`   | PathRequests.h    | ms   | Full pathfinding duration      |
 
 Quantiles collected: 0th, 50th, 90th, 95th, 99th, 100th percentile.
 
-\* **`rpc_size` now records bytes as bytes (fixed).** It used to go through the
-millisecond-scaled event histogram and export as `rpc_size_milliseconds_bucket`
-on a ladder topping out at 5000, so the 24.9% of responses larger than 5 kB all
-landed in the last bucket and every percentile read back as a flat 5000 — a
-plausible-looking constant rather than a byte size. `beast::insight::Event` now
+\* **`rpc_size` records bytes, not a duration.** `beast::insight::Event`
 declares a `Unit`, so this instrument is created with unit `By` and exports as
 **`rpc_size_bytes_bucket`** on `kByteBuckets` (512 B to 1 MiB, placed from the
-measured distribution). Queries and panels must use the new name.
+measured distribution). Queries and panels must use that name.
 
 **Grafana dashboards**: _Node Health_ (`ios_latency`), _RPC & Pathfinding_ (`rpc_time`, `rpc_size`, `pathfind_*`)
 
diff --git a/docker/telemetry/grafana/dashboards/rpc-pathfinding.json b/docker/telemetry/grafana/dashboards/rpc-pathfinding.json
index 8feac10762..c0cb25c1f1 100644
--- a/docker/telemetry/grafana/dashboards/rpc-pathfinding.json
+++ b/docker/telemetry/grafana/dashboards/rpc-pathfinding.json
@@ -154,7 +154,7 @@
     },
     {
       "title": "RPC Response Size",
-      "description": "\u26a0 Instrument mismatch \u2014 values unreliable. Response size is recorded through the millisecond-scaled event histogram (rpc_size_bytes_bucket), so byte values saturate at the top time bucket (5000) and the percentiles are not true byte sizes. A dedicated byte-unit histogram is needed to fix this; tracked separately. Treat this panel as indicative only until then.\n\n###### What this is:\n*The 95th-percentile size of RPC response payloads in bytes.*\n\n###### How it's computed:\n*95th-percentile of response payload sizes over the dashboard rate interval, per node.*\n\n###### Reading it:\n*Smaller is cheaper; large responses cost bandwidth and memory.*\n\n###### Healthy range:\n*Workload-dependent; small for status queries, large for bulk data queries.*\n\n###### Watch for:\n*Growth in large responses, consistent with expensive queries or API misuse.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[ServerHandler.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/rpc/detail/ServerHandler.cpp)\n\n###### Function:\n`ServerHandler ctor`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)",
+      "description": "###### What this is:\n*The 95th-percentile size of RPC response payloads in bytes.*\n\n###### How it's computed:\n*95th-percentile of response payload sizes over the dashboard rate interval, per node. The instrument declares a byte unit, so it exports as `rpc_size_bytes_bucket` and is recorded on the byte bucket ladder (512 B to 1 MiB); the panel unit `decbytes` therefore shows a true size.*\n\n###### Reading it:\n*Smaller is cheaper; large responses cost bandwidth and memory.*\n\n###### Healthy range:\n*Workload-dependent; small for status queries, large for bulk data queries.*\n\n###### Watch for:\n*Growth in large responses, consistent with expensive queries or API misuse.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[ServerHandler.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/rpc/detail/ServerHandler.cpp)\n\n###### Function:\n`ServerHandler ctor`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)",
       "type": "timeseries",
       "gridPos": {
         "h": 10,
diff --git a/docker/telemetry/workload/benchmark.sh b/docker/telemetry/workload/benchmark.sh
index 7518e52485..4c0ea99167 100755
--- a/docker/telemetry/workload/benchmark.sh
+++ b/docker/telemetry/workload/benchmark.sh
@@ -189,9 +189,11 @@ trace_peer=1
 trace_ledger=1
 
 [insight]
+# No prefix is set on purpose: OTelCollector applies no prefix to instrument
+# names, so setting one here would be inert. Same reasoning as
+# run-full-validation.sh.
 server=otel
-endpoint=http://localhost:4318/v1/metrics
-prefix=xrpld"
+endpoint=http://localhost:4318/v1/metrics"
         else
             telemetry_section="
 [telemetry]
@@ -238,8 +240,8 @@ ${ips_fixed}
 ${telemetry_section}
 
 [rpc_startup]
-# Deliberately diverges from run-full-validation.sh (which uses info) and keeps
-# prefix=xrpld above. This script measures telemetry OVERHEAD as the delta
+# The log level deliberately diverges from run-full-validation.sh, which uses
+# info. This script measures telemetry OVERHEAD as the delta
 # between a telemetry-off and a telemetry-on arm, so it needs a quiet, identical
 # baseline in both arms rather than good observability. Logging is synchronous, so
 # admitting more log lines would add I/O to both arms and inflate the CPU,

From 2f8d8eabc74ae12a28cf3d679c969e5011629710 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 21:09:58 +0100
Subject: [PATCH 23/38] fix(insight): include <cstdint> for the type
 Event::notify takes

Event::notify takes std::uint64_t but the header never included <cstdint>,
relying on it arriving through another include. clang-tidy's include-cleaner
flags it, which fails CI on any branch where this file lands in the
changed-file set.

Fixed here, on the branch that introduced the std::uint64_t parameter, rather
than only downstream where it happened to surface.
---
 include/xrpl/beast/insight/Event.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/xrpl/beast/insight/Event.h b/include/xrpl/beast/insight/Event.h
index 9640e2c1b4..9782ceb496 100644
--- a/include/xrpl/beast/insight/Event.h
+++ b/include/xrpl/beast/insight/Event.h
@@ -3,6 +3,7 @@
 #include <xrpl/beast/insight/EventImpl.h>
 
 #include <chrono>
+#include <cstdint>
 #include <memory>
 #include <utility>
 

From cc5e09892813cff2295bc7bba00cde956b419413 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 21:10:28 +0100
Subject: [PATCH 24/38] fix(telemetry): rate the overlay traffic bargauge
 instead of ranking totals

The panel described itself as showing which categories consume the most
receive bandwidth, and told operators to watch for a category dominating
during ongoing sync, but it ranked raw cumulative counters. That ranks by
lifetime total, so a sync burst days earlier still dominates the bars and
the stated guidance cannot work.

Wrapping the existing selector in rate() does not work: rate() drops
__name__, so the many *_bytes_in series collapse to one labelset and the
query fails with "vector cannot contain metrics with the same labelset".
Grouping with sum by (__name__) does not help either, because Prometheus
strips __name__ from aggregation output regardless of the grouping. Both
were tried against a local Prometheus with synthetic multi-category
counters and both error.

Instead recover the category into a normal `series` label before rating,
and use a subquery so the renamed series can be rated. Verified against
the same synthetic data: rates match a plain single-metric rate control
to within subquery resolution, ranking order is correct, and both the
category and node labels survive.

Also set unit to Bps, matching the ten sibling rate panels in this
dashboard (this was the only byte panel still on decbytes and the only
one plotting raw), and drop "Heatmap" from the title since the panel is
a bargauge.

Note the subquery evaluates the selector at several steps, so it costs
more than the previous instant query across the traffic categories.
---
 docker/telemetry/grafana/dashboards/ledger-data-sync.json | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/docker/telemetry/grafana/dashboards/ledger-data-sync.json b/docker/telemetry/grafana/dashboards/ledger-data-sync.json
index 3650fb9d35..598830b3fa 100644
--- a/docker/telemetry/grafana/dashboards/ledger-data-sync.json
+++ b/docker/telemetry/grafana/dashboards/ledger-data-sync.json
@@ -10,8 +10,8 @@
   "links": [],
   "panels": [
     {
-      "title": "Overlay Traffic Heatmap (All Categories, Bytes In) [$xrpl_network_type]",
-      "description": "###### What this is:\n*All overlay traffic categories ranked by inbound bytes, giving an at-a-glance view of which message types consume the most receive bandwidth.*\n\n###### How it's computed:\n*Top categories by latest inbound byte value across all traffic categories. Each bar is labelled with its traffic category followed by the node identity; the shared `_bytes_in` suffix is dropped from the category name because the panel already reports inbound bytes.*\n\n###### Reading it:\n*The longest bars are the biggest bandwidth consumers; on a synced node transactions, proposals, and validations usually lead.*\n\n###### Healthy range:\n*workload-dependent.*\n\n###### Watch for:\n*A single ledger-data or fetch category dominating (ongoing sync) or an unexpected category topping the list.*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`",
+      "title": "Overlay Traffic by Category (Bytes In Rate) [$xrpl_network_type]",
+      "description": "###### What this is:\n*All overlay traffic categories ranked by inbound bytes, giving an at-a-glance view of which message types consume the most receive bandwidth.*\n\n###### How it's computed:\n*Top categories by inbound byte rate across all traffic categories. The category name is recovered from the metric name before `rate()` is applied, because `rate()` drops `__name__`; a subquery is used so the renamed series can then be rated. Each bar is labelled with its traffic category followed by the node identity; the shared `_bytes_in` suffix is dropped from the category name because the panel already reports inbound bytes.*\n\n###### Reading it:\n*The longest bars are the biggest bandwidth consumers; on a synced node transactions, proposals, and validations usually lead.*\n\n###### Healthy range:\n*workload-dependent.*\n\n###### Watch for:\n*A single ledger-data or fetch category dominating (ongoing sync) or an unexpected category topping the list.*\n\n###### Source:\n[OverlayImpl.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/overlay/detail/OverlayImpl.cpp)\n\n###### Function:\n`OverlayImpl ctor (TrafficGauges)`",
       "type": "bargauge",
       "gridPos": {
         "h": 10,
@@ -38,13 +38,13 @@
           "datasource": {
             "type": "prometheus"
           },
-          "expr": "label_replace(topk(20, {service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", __name__=~\".*_bytes_in\", __name__!~\"total_.*\"}), \"series\", \"$1\", \"__name__\", \"(.*)_bytes_in\")"
+          "expr": "topk(20, rate(label_replace({service_instance_id=~\"$node\", deployment_environment=~\"$deployment_environment\", xrpl_network_type=~\"$xrpl_network_type\", service_name=~\"$service_name\", xrpl_work_item=~\"$xrpl_work_item\", xrpl_branch=~\"$xrpl_branch\", xrpl_node_role=~\"$xrpl_node_role\", __name__=~\".*_bytes_in\", __name__!~\"total_.*\"}, \"series\", \"$1\", \"__name__\", \"(.*)_bytes_in\")[$__rate_interval:]))"
         }
       ],
       "fieldConfig": {
         "defaults": {
           "displayName": "${__field.labels.series} [${__field.labels.service_instance_id} ${__field.labels.xrpl_branch} ${__field.labels.xrpl_node_role} ${__field.labels.xrpl_work_item}]",
-          "unit": "decbytes",
+          "unit": "Bps",
           "thresholds": {
             "mode": "absolute",
             "steps": [

From bca2b35bf01ada6dc161a9dcc803d076fa9687d4 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 21:45:35 +0100
Subject: [PATCH 25/38] docs(telemetry): stop showing inert [insight] prefix on
 the OTel path

ff8629bb11 dropped prefix=xrpld as inert and misleading, but four OTel-path
sites still set it, so the branch contradicted itself.

OTelCollector routes every instrument name through a static formatName()
that only lowercases and maps '.'/space to '_'; the sole read of prefix_ is
the startup log line at OTelCollector.cpp:810. All four instrument factories
funnel through formatName(), so no prefix can reach an exported name.
StatsDCollector does prepend it (StatsDCollector.cpp:551/592/640/715), so the
StatsD example legitimately keeps it.

Removed from the 09 reference's OTel config block and from both
quick-reference setups, and from the cfg integration-test.sh generates. The
StatsD example is unchanged and now states why it keeps the key.

Also corrected run-full-validation.sh: [insight] endpoint was described as
"already matches the built-in default", implying it would matter if it
differed. CollectorManager reads it and hands it to OTelCollector, which also
only logs it; the exporter URL is built in Telemetry::initMetrics() from
[telemetry] endpoint. It is as inert as prefix was.
---
 .../09-data-collection-reference.md           | 23 +++++++++++++++----
 docker/telemetry/integration-test.sh          |  6 ++++-
 .../telemetry/workload/run-full-validation.sh | 10 ++++++--
 3 files changed, 31 insertions(+), 8 deletions(-)

diff --git a/OpenTelemetryPlan/09-data-collection-reference.md b/OpenTelemetryPlan/09-data-collection-reference.md
index 03b66bb9c9..61eb2e1fc4 100644
--- a/OpenTelemetryPlan/09-data-collection-reference.md
+++ b/OpenTelemetryPlan/09-data-collection-reference.md
@@ -598,15 +598,26 @@ These are system-level metrics emitted by xrpld's `beast::insight` framework via
 [insight]
 server=otel
 endpoint=http://localhost:4318/v1/metrics
-prefix=xrpld
 ```
 
+`server=otel` is the only key here that changes what gets exported. No `prefix` is
+shown because it would do nothing on this path: `OTelCollector` routes every
+instrument name through its `static formatName()`, which only lowercases the raw
+name and maps `.` and space to `_`, and the only place the class reads `prefix_`
+is its startup log line. Exported names are therefore the lowercased raw names
+(`jobq_job_count`, `rpc_requests_total`) and the service is identified by the OTel
+resource `service.name`, not by a name prefix. `endpoint` is read from this section
+but likewise reaches only that log line — the real exporter URL is derived inside
+`Telemetry::initMetrics()` from `[telemetry] endpoint`, by swapping the trailing
+`/v1/traces` for `/v1/metrics`.
+
 Fallback (StatsD). `StatsDCollector` is still selected by this value, but the
 stack in `docker/telemetry/` no longer receives it: using this path also requires
 re-adding the `statsd` receiver to `otel-collector-config.yaml` and uncommenting
 port 8125 in `docker-compose.yml`, otherwise the metrics go to a port nothing
-listens on. Note also that `StatsDCollector` applies `prefix` to the metric name
-while `OTelCollector` does not, so switching transports renames every series.
+listens on. `prefix` does appear below, because `StatsDCollector` really does
+prepend it to every metric name it serializes — so switching transports renames
+every series.
 
 ```ini
 [insight]
@@ -2168,7 +2179,6 @@ enabled=1
 [insight]
 server=otel
 endpoint=http://localhost:4318/v1/metrics
-prefix=xrpld
 ```
 
 ### Production Setup
@@ -2184,9 +2194,12 @@ max_queue_size=4096
 [insight]
 server=otel
 endpoint=http://otel-collector:4318/v1/metrics
-prefix=xrpld
 ```
 
+Neither block sets `[insight] prefix`: on the `server=otel` path it is inert and
+would only mislead — see [§2 Configuration](#configuration). It applies solely to
+`server=statsd`.
+
 ### Trace Category Toggle
 
 | Config Key           | Default | Controls                     |
diff --git a/docker/telemetry/integration-test.sh b/docker/telemetry/integration-test.sh
index 090ac28624..b24d90015a 100755
--- a/docker/telemetry/integration-test.sh
+++ b/docker/telemetry/integration-test.sh
@@ -397,9 +397,13 @@ trace_ledger=1
 metrics_endpoint=http://localhost:4318/v1/metrics
 
 [insight]
+# server=otel is the only load-bearing key here -- it selects OTelCollector so
+# beast::insight metrics leave over OTLP. No prefix is set on purpose: on this
+# path it is inert, because OTelCollector's formatName() only lowercases the raw
+# instrument name and the one place the class reads prefix_ is its startup log
+# line. The service is identified by the OTel resource service.name.
 server=otel
 endpoint=http://localhost:4318/v1/metrics
-prefix=rippled
 service_instance_id=Node-${i}
 
 [rpc_startup]
diff --git a/docker/telemetry/workload/run-full-validation.sh b/docker/telemetry/workload/run-full-validation.sh
index 5acbbfbfc5..d5295c9b3e 100755
--- a/docker/telemetry/workload/run-full-validation.sh
+++ b/docker/telemetry/workload/run-full-validation.sh
@@ -331,8 +331,14 @@ trace_ledger=1
 # Native OTel metrics via OTLP/HTTP. The collector has no StatsD receiver
 # (its metrics pipeline is [otlp, spanmetrics]), so beast::insight must export
 # over OTLP for system metrics to reach Prometheus at all. server=otel is the
-# only load-bearing key here -- it selects the OTel collector; endpoint is
-# read but already matches the built-in default.
+# only load-bearing key here -- it selects the OTel collector.
+#
+# endpoint below is parsed but inert, exactly like prefix: CollectorManager
+# reads it and hands it to OTelCollector, which uses it only in its startup log
+# line. The URL the metric exporter actually posts to is derived in
+# Telemetry::initMetrics() from [telemetry] endpoint, by swapping the trailing
+# /v1/traces for /v1/metrics. It is kept here so the log line names the right
+# URL; changing it would not redirect a single metric.
 #
 # No prefix is set on purpose. It would be inert: OTelCollector applies no
 # prefix to instrument names, so exported names are the lowercased raw names

From 9c89419929120dd7d4ad17f0d5e3433cbb4198b4 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 21:46:04 +0100
Subject: [PATCH 26/38] docs(telemetry): correct two miscounted facts in
 dashboard and metric docs

RPC Response Size on rpc-pathfinding said its p95 was computed "over the
dashboard rate interval", but the query hardcodes [5m]. Every other panel on
that dashboard with a hardcoded [5m] -- RPC Response Time, RPC Response Time
Distribution, both Pathfinding duration panels, the gRPC latency panel and
Pathfinding Compute Duration -- says "over 5 minutes". Matched the clause to
the query. The same edit was applied to the local grafanacloud copy so the
two stay identical; that tree is gitignored, so it is not in this commit.

The io_latency group in expected_metrics.json claimed "all 6 panels that
query it". That 6 was a raw string-occurrence count over the dashboards and
included two panel descriptions. Verified truth: two distinct panels query
the metric, ledger-data-sync "I/O Scheduler Latency p95" and node-health
"I/O Latency", each mirrored in a grafanacloud copy, plus one alert rule in
grafana/provisioning/alerting/rules.yaml. Stated that instead of a count.
---
 docker/telemetry/grafana/dashboards/rpc-pathfinding.json | 2 +-
 docker/telemetry/workload/expected_metrics.json          | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/docker/telemetry/grafana/dashboards/rpc-pathfinding.json b/docker/telemetry/grafana/dashboards/rpc-pathfinding.json
index c0cb25c1f1..7da901c382 100644
--- a/docker/telemetry/grafana/dashboards/rpc-pathfinding.json
+++ b/docker/telemetry/grafana/dashboards/rpc-pathfinding.json
@@ -154,7 +154,7 @@
     },
     {
       "title": "RPC Response Size",
-      "description": "###### What this is:\n*The 95th-percentile size of RPC response payloads in bytes.*\n\n###### How it's computed:\n*95th-percentile of response payload sizes over the dashboard rate interval, per node. The instrument declares a byte unit, so it exports as `rpc_size_bytes_bucket` and is recorded on the byte bucket ladder (512 B to 1 MiB); the panel unit `decbytes` therefore shows a true size.*\n\n###### Reading it:\n*Smaller is cheaper; large responses cost bandwidth and memory.*\n\n###### Healthy range:\n*Workload-dependent; small for status queries, large for bulk data queries.*\n\n###### Watch for:\n*Growth in large responses, consistent with expensive queries or API misuse.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[ServerHandler.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/rpc/detail/ServerHandler.cpp)\n\n###### Function:\n`ServerHandler ctor`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)",
+      "description": "###### What this is:\n*The 95th-percentile size of RPC response payloads in bytes.*\n\n###### How it's computed:\n*95th-percentile of response payload sizes over 5 minutes, per node. The instrument declares a byte unit, so it exports as `rpc_size_bytes_bucket` and is recorded on the byte bucket ladder (512 B to 1 MiB); the panel unit `decbytes` therefore shows a true size.*\n\n###### Reading it:\n*Smaller is cheaper; large responses cost bandwidth and memory.*\n\n###### Healthy range:\n*Workload-dependent; small for status queries, large for bulk data queries.*\n\n###### Watch for:\n*Growth in large responses, consistent with expensive queries or API misuse.*\n\n###### Keywords:\n- **RPC command / method** *(per node)* \u2014 a named API request served by the node (e.g. account_info, ledger, submit), the unit RPC panels break down by.\n\n###### Computation boundary:\n*Result: Per node \u2014 each series is one server's own value.*\n*Recorded in xrpld code as a native metric (beast::insight); the collector only forwards it; the Grafana query selects and aggregates it.*\n\n###### Source:\n[ServerHandler.cpp](https://github.com/XRPLF/rippled/blob/develop/src/xrpld/rpc/detail/ServerHandler.cpp)\n\n###### Function:\n`ServerHandler ctor`\n\n###### References:\n[RPC command / method](https://xrpl.org/docs/references/http-websocket-apis/public-api-methods) \u00b7 [Telemetry glossary](https://github.com/XRPLF/rippled/blob/develop/docs/telemetry-glossary.md#rpc-command-method)",
       "type": "timeseries",
       "gridPos": {
         "h": 10,
diff --git a/docker/telemetry/workload/expected_metrics.json b/docker/telemetry/workload/expected_metrics.json
index 3731a1f5c6..d07da27b4d 100644
--- a/docker/telemetry/workload/expected_metrics.json
+++ b/docker/telemetry/workload/expected_metrics.json
@@ -47,7 +47,7 @@
     "metrics": ["rpc_requests_total", "ledger_fetches_total"]
   },
   "io_latency": {
-    "description": "io_context scheduling-latency histogram — a beast::insight Event exported over OTLP (Application.cpp:515, makeEvent('ios_latency') with the default millisecond unit). It is the one beast Event the node itself guarantees, which is why it is asserted while rpc_time_milliseconds and the jobq_* pairs sit in not_asserted: the sampler starts on the unconditional startup path (Application.cpp:1697, outside the 'if (withTimers)' guard), and its handler always emits the first sample whatever its value (Application.cpp:185-189, 'firstSample_.exchange(false) || lastSample >= 10ms', with a comment stating the point is to register the metric downstream). The OTel histogram is cumulative, so that one notify creates series that persist for the rest of the run. Named with the _bucket/_count/_sum suffixes the Prometheus exporter emits for a histogram — no bare ios_latency_milliseconds series exists, the same convention rpc_method_us and span_duration_milliseconds follow, and all 6 panels that query it use ios_latency_milliseconds_bucket. Only existence is asserted: past the first sample the handler reports only latencies >= 10 ms, so neither the sample count nor the value is predictable.",
+    "description": "io_context scheduling-latency histogram — a beast::insight Event exported over OTLP (Application.cpp:515, makeEvent('ios_latency') with the default millisecond unit). It is the one beast Event the node itself guarantees, which is why it is asserted while rpc_time_milliseconds and the jobq_* pairs sit in not_asserted: the sampler starts on the unconditional startup path (Application.cpp:1697, outside the 'if (withTimers)' guard), and its handler always emits the first sample whatever its value (Application.cpp:185-189, 'firstSample_.exchange(false) || lastSample >= 10ms', with a comment stating the point is to register the metric downstream). The OTel histogram is cumulative, so that one notify creates series that persist for the rest of the run. Named with the _bucket/_count/_sum suffixes the Prometheus exporter emits for a histogram — no bare ios_latency_milliseconds series exists, the same convention rpc_method_us and span_duration_milliseconds follow, and every consumer queries ios_latency_milliseconds_bucket: 2 distinct panels (ledger-data-sync 'I/O Scheduler Latency p95' and node-health 'I/O Latency', each mirrored in a grafanacloud copy) plus one alert rule (grafana/provisioning/alerting/rules.yaml:586). Only existence is asserted: past the first sample the handler reports only latencies >= 10 ms, so neither the sample count nor the value is predictable.",
     "metrics": [
       "ios_latency_milliseconds_bucket",
       "ios_latency_milliseconds_count",

From 6a82fc6f37c9e6cb747722906e83aaf949d100b8 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 21:46:30 +0100
Subject: [PATCH 27/38] docs(telemetry): qualify the log-correlation guarantee
 and record the CI gap

The runbook said a correlated log line was "guaranteed" at info severity.
Info is necessary but not sufficient. Replaced the flat claim with the four
real preconditions, each with the code that enforces it and the failure mode
it produces: telemetry enabled, trace_consensus=1, a valid roundSpanContext_
(SpanGuard::childSpan returns a null guard on an invalid parent), and a valid
plus sampled span context (Log.cpp gates injection on IsValid and IsSampled).
Also noted which harness cfgs satisfy them -- run-full-validation.sh and
integration-test.sh set all three config keys; benchmark.sh deliberately
stays at warning and runs no correlation check.

Second, the two checks this work exists to make pass are not exercised by
CI. The workflow hardcodes --skip-loki, and validate_telemetry.py builds
log.trace_id_present and log.trace_id_cross_reference only inside an
"if not skip_loki" branch, so they are never constructed rather than merely
skipped, and never appear in the report. No workflow runs integration-test.sh
either, so its own check_log_correlation() never runs in CI. Recorded that in
the runbook's CI workflow section and in the workload README, with the local
command that does cover it: run-full-validation.sh without --skip-loki.

The workflow itself is unchanged on purpose. Dropping the flag would make CI
exercise Loki ingestion and filelog mounting for the first time on the same
run that must produce a clean regression baseline, so a red result would not
be attributable.
---
 docker/telemetry/workload/README.md | 24 +++++++++++++++++++-
 docs/telemetry-runbook.md           | 35 ++++++++++++++++++++++++++---
 2 files changed, 55 insertions(+), 4 deletions(-)

diff --git a/docker/telemetry/workload/README.md b/docker/telemetry/workload/README.md
index 4c0ce078ca..adba3a7864 100644
--- a/docker/telemetry/workload/README.md
+++ b/docker/telemetry/workload/README.md
@@ -215,7 +215,7 @@ Automated validation that all expected telemetry data exists. Every metric in `e
 
 - **Span validation**: All span types from `expected_spans.json` with required attributes and parent-child hierarchies. Entries marked `"optional": true` only fire under traffic the harness may not produce (HTTP/JSON-RPC client, gRPC client, missing-ledger fetch, mode transitions); their absence is recorded as a passing skip, not a failure.
 - **Metric validation**: All metrics from `expected_metrics.json` — SpanMetrics, `beast::insight` gauges/counters/histograms, `MetricsRegistry` OTLP metrics. Every listed metric must have > 0 series. Uses the Prometheus `/api/v1/series` endpoint (not instant queries), polled until the metric appears or the poll window elapses, so a late-populating or quiet series is not a false negative.
-- **Log-trace correlation**: trace_id/span_id in Loki logs (requires Loki)
+- **Log-trace correlation**: trace_id/span_id in Loki logs (requires Loki). The two checks are `log.trace_id_present` and `log.trace_id_cross_reference`, and they exist only when `--skip-loki` is **not** passed — `run_validation()` builds them inside an `if not skip_loki` branch, so with the flag they are absent from the report rather than reported as skipped. **CI always passes `--skip-loki`, so these two are never exercised there** — see [CI Integration](#ci-integration).
 - **Dashboard validation**: Every dashboard uid listed under `grafana_dashboards.uids` in `expected_metrics.json` loads with panels. That list currently covers **all 15** dashboards provisioned in `docker/telemetry/grafana/dashboards/`. Note the scope of this check: it asks the Grafana API whether the dashboard exists and returns a panel count — it does **not** run the panels' queries, so a dashboard can pass here while individual panels render empty.
 
 ```bash
@@ -396,6 +396,28 @@ Of the five `workflow_dispatch` inputs, only `run_benchmark` changes behaviour.
 them again — load shape comes entirely from `--profile` and
 `workload-profiles.json`. Their `description:` fields say so.
 
+### What CI does not cover
+
+The workflow invokes the harness with a hardcoded `--skip-loki`
+(`telemetry-validation.yml:237`), and `validate_telemetry.py` creates the
+`log.trace_id_present` and `log.trace_id_cross_reference` checks only inside an
+`if not skip_loki` branch (`validate_telemetry.py:1657`). Those two checks are
+therefore **never constructed on a CI run** — a green `Telemetry Validation`
+carries no evidence that log lines carry trace context or that a Tempo trace id
+resolves in Loki. `integration-test.sh` has its own `check_log_correlation()`, but
+no workflow runs that script either.
+
+Cover the gap locally by omitting the flag:
+
+```bash
+docker/telemetry/workload/run-full-validation.sh --xrpld .build/xrpld
+```
+
+Do that after any change to log formatting, span activation, the collector's
+`filelog` receiver, or the Loki exporter. Removing `--skip-loki` from the workflow
+would make CI exercise Loki ingestion and filelog mounting for the first time, so
+it is held back as its own change rather than folded into an unrelated push.
+
 ## Configuration Files
 
 | File                              | Purpose                                                                                                                       |
diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md
index a4970e07c9..95a1369593 100644
--- a/docs/telemetry-runbook.md
+++ b/docs/telemetry-runbook.md
@@ -2759,9 +2759,26 @@ A plain `SpanGuard` that is **never activated** makes no span current — it "ne
 
 Severity does not affect injection, but `JLOG` filters on severity **before** `format()` runs, so the configured log level decides whether a qualifying line is emitted at all.
 
-**The guaranteed correlated line at `info`** is the consensus accept pair at [RCLConsensus.cpp:736/740](../src/xrpld/app/consensus/RCLConsensus.cpp#L736) — an `if`/`else`, so exactly one of the two fires on every accepted round. `doAccept` activates the accept span as ambient over its whole body at [:565](../src/xrpld/app/consensus/RCLConsensus.cpp#L565) (`activateIfLive(acceptSpan)`, commented "Make the accept span ambient for the whole accept so doAccept's log lines ... correlate to it"), and the activation lives to the end of the function, so both branches are inside it. At roughly one round every 4 s this yields dozens of correlated lines per run.
+**The dependably correlated line at `info`** is the consensus accept pair at [RCLConsensus.cpp:736/740](../src/xrpld/app/consensus/RCLConsensus.cpp#L736) — an `if`/`else`, so exactly one of the two fires on every accepted round. `doAccept` activates the accept span as ambient over its whole body at [:565](../src/xrpld/app/consensus/RCLConsensus.cpp#L565) (`activateIfLive(acceptSpan)`, commented "Make the accept span ambient for the whole accept so doAccept's log lines ... correlate to it"), and the activation lives to the end of the function, so both branches are inside it. At roughly one round every 4 s this yields dozens of correlated lines per run.
 
-`info` is therefore the minimum level at which the `log.trace_id_present` and `log.trace_id_cross_reference` checks are satisfied by construction, and it is what the workload harness generates. At `warning` and above that pair is suppressed and correlation becomes incidental — dependent on a `warn`-or-worse line happening to fire inside some active span.
+That is a dependable pair rather than an unconditional one: `info` severity is necessary but not sufficient. Four preconditions must all hold, and each has its own bail-out that silently yields an uncorrelated line rather than an error:
+
+| Precondition                                                                  | Where it is enforced                                                                                                                                                                                                                                                                        | What happens if it fails                                                                        |
+| ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
+| Telemetry enabled — built with `telemetry=ON` **and** `[telemetry] enabled=1` | The `SpanGuard` factories return early on `tel == nullptr \|\| !tel->isEnabled()`, e.g. [SpanGuard.cpp:278-280](../src/libxrpl/telemetry/SpanGuard.cpp#L278)                                                                                                                                | Null guard, no span, no `trace_id`                                                              |
+| `trace_consensus=1`                                                           | The round span is built by `hashSpan(TraceCategory::Consensus, …)`, which checks `isCategoryEnabled()` ([SpanGuard.cpp:354](../src/libxrpl/telemetry/SpanGuard.cpp#L354))                                                                                                                   | No round span, so `roundSpanContext_` never becomes valid                                       |
+| A valid `roundSpanContext_` at accept time                                    | `makeAcceptSpan()` calls `SpanGuard::childSpan(cs::accept, roundSpanContext_)` ([RCLConsensus.cpp:512](../src/xrpld/app/consensus/RCLConsensus.cpp#L512)), and `childSpan` returns a null guard on an invalid parent ([SpanGuard.cpp:274-281](../src/libxrpl/telemetry/SpanGuard.cpp#L274)) | Null accept span, so `activateIfLive` activates nothing and the pair logs without trace context |
+| The current span context is valid **and sampled**                             | Injection is gated on `spanCtx.IsValid() && spanCtx.IsSampled()` ([Log.cpp:328](../src/libxrpl/basics/Log.cpp#L328))                                                                                                                                                                        | Ids are withheld deliberately, so the log never advertises a trace that was not exported        |
+
+The sampled check is normally satisfied on a self-rooted consensus round — head sampling is a fixed ratio of 1.0 wrapped in a `ParentBasedSampler` ([Telemetry.cpp:412-414](../src/libxrpl/telemetry/Telemetry.cpp#L412)) — but a span inheriting an unsampled **remote** parent is dropped while still carrying valid ids, which is exactly the case the gate exists for.
+
+With all four satisfied, `info` is the minimum level at which the `log.trace_id_present` and `log.trace_id_cross_reference` checks pass by construction, and it is what the correlation-checking harnesses generate: the cfgs written by [run-full-validation.sh](../docker/telemetry/workload/run-full-validation.sh) and [integration-test.sh](../docker/telemetry/integration-test.sh) each set `enabled=1`, `trace_consensus=1` and `log_level info` together. `benchmark.sh` deliberately does not — it stays at `warning` to keep log I/O out of the overhead measurement, and it runs no correlation check. At `warning` and above that pair is suppressed and correlation becomes incidental — dependent on a `warn`-or-worse line happening to fire inside some active span.
+
+> **CI does not exercise either check.** `log.trace_id_present` and `log.trace_id_cross_reference` never run in CI — see [Coverage gap](#ci-workflow) for why. Cover them locally, without `--skip-loki`, after any change to log formatting, span activation, the `filelog` receiver or the Loki exporter:
+>
+> ```bash
+> docker/telemetry/workload/run-full-validation.sh --xrpld .build/xrpld
+> ```
 
 #### Why not `debug`
 
@@ -3521,7 +3538,7 @@ not a sign the cache is working.
 - Verify xrpld was built with `telemetry=ON` (the `XRPL_ENABLE_TELEMETRY` preprocessor flag)
 - Verify `enabled=1` in the `[telemetry]` config section
 - Log lines only contain `trace_id`/`span_id` when emitted inside an active span — background logs outside of RPC/consensus/transaction processing will not have trace context
-- Check `log_level` is at least `info`. The guaranteed correlated line is the consensus accept pair, which is at info severity, so at `warning` or above correlation becomes incidental — see [Which Log Lines Carry Trace Context](#which-log-lines-carry-trace-context)
+- Check `log_level` is at least `info`. The dependably correlated line is the consensus accept pair, which is at info severity, so at `warning` or above correlation becomes incidental. `info` is necessary but not sufficient: the accept pair also needs telemetry enabled, `trace_consensus=1`, a valid round span context and a sampled span context — see [Which Log Lines Carry Trace Context](#which-log-lines-carry-trace-context) for the full precondition table
 - A plain `SpanGuard` that is never activated does not make its span current, so lines inside one are never correlated regardless of severity. Activated guards (`activate()` / `activateIfLive()`) and `ScopedSpanGuard` both do make their span current
 - Check that the specific trace category is enabled (e.g., `trace_rpc=1`)
 
@@ -3754,6 +3771,18 @@ container as the main CI, so Conan and ccache hit the shared caches), and
   `src/xrpld/telemetry/**`. There is no cron schedule.
 - **Invocation**: `run-full-validation.sh --xrpld <binary> --skip-loki`, so the
   default `full-validation` profile is used and the Loki checks are skipped.
+- **Coverage gap — log-trace correlation is never checked in CI**: the
+  `--skip-loki` above is hardcoded at
+  [telemetry-validation.yml:237](../.github/workflows/telemetry-validation.yml#L237),
+  and `validate_telemetry.py` creates `log.trace_id_present` and
+  `log.trace_id_cross_reference` only inside an `if not skip_loki` branch
+  ([:1657](../docker/telemetry/workload/validate_telemetry.py#L1657)), so those
+  two checks are not merely skipped — they are never constructed and never appear
+  in the report. `docker/telemetry/integration-test.sh` (which has its own
+  `check_log_correlation()`) is run by no workflow at all. A green
+  `Telemetry Validation` therefore carries no evidence about correlation; run it
+  locally without the flag to cover it:
+  `docker/telemetry/workload/run-full-validation.sh --xrpld .build/xrpld`.
 - **Inputs**: only `run_benchmark` changes behaviour. `rpc_rate`, `rpc_duration`,
   `tx_tps` and `tx_duration` are inert, as noted in their descriptions.
 - **Results**: reports are uploaded as the `telemetry-validation-reports`

From c4b8df9de189fbc82e6c9ef8c9866ef6ad2e4902 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 22:33:22 +0100
Subject: [PATCH 28/38] test(telemetry): recapture span baselines on the
 current ladder

Copied verbatim from the timings.json produced by the telemetry-validation
run at 6a82fc6f37 (166/166 checks passed), which is the hand-off the
workflow prints for a placeholder baseline.

The numbers confirm why the previous baseline had to be voided. It was
captured 2026-06-05, before the collector's spanmetrics ladder gained
sub-millisecond edges, and its sub-1ms entries were arithmetic on the old
1ms first edge rather than latencies:

  span.ledger.store  p50/p95/p99  0.5 / 0.95 / 0.99   ->  0.005 / 0.0095 / 0.0099

Exactly 100x, because the old values were quantile x 1ms and the real ones
are quantile x 0.01ms. Since the gate only trips on increases, every
sub-millisecond span was unguarded against a 100x regression.

The job.* pair is back too, recaptured on the re-cut microsecond ladder
(floor 1us): job.acceptLedger.queued.p95 now reads 91.1us as a measurement,
where the voided value of 96.79us was 0.95/0.9926 x 100.
---
 .../workload/baselines/baseline-timings.json  | 121 +++++++++++++++++-
 1 file changed, 116 insertions(+), 5 deletions(-)

diff --git a/docker/telemetry/workload/baselines/baseline-timings.json b/docker/telemetry/workload/baselines/baseline-timings.json
index 0f9b858e5e..9a1b452478 100644
--- a/docker/telemetry/workload/baselines/baseline-timings.json
+++ b/docker/telemetry/workload/baselines/baseline-timings.json
@@ -1,9 +1,120 @@
 {
-  "_note": "PLACEHOLDER -- every entry was voided, and the next telemetry-validation run reprints a fresh block to paste back (see baselines/README.md). Span entries were captured 2026-06-05 against a spanmetrics ladder whose first edge was 1ms. That ladder had no edge below 1ms at all, so for the sub-millisecond spans nearly every sample landed in bucket 0 and their quantiles are interpolation across that single edge, not latencies: span.ledger.store read p50/p95/p99 = 0.5/0.95/0.99, exactly quantile x 1ms, and span.tx.process read 0.50428/0.95813/0.99847 for the same reason. Sub-millisecond edges (floor 0.01ms) landed 2026-08-04 in 3860c93db2, so the ladder those numbers came from no longer exists. The gate is one-sided -- compare_to_baseline.py flags a regression only when current exceeds baseline -- so a stale-high baseline passes everything in silence, including a real 10x regression from 0.05ms to 0.5ms hiding under the 0.95ms entry. The 11 entries at or above 1ms (consensus.accept p50/p95/p99, consensus.ledger_close p95/p99, ledger.build p95/p99, ledger.validate p95/p99, tx.apply p95/p99) were undistorted -- but only because every edge from 1ms to 1s is byte-identical between the two ladders and all 11 happen to fall in that band, NOT because the re-cut spared everything above 1ms. It did not: it also added 2s/3s/4s/10s/30s edges above 1s, so a span whose quantiles reach the second scale is distorted just as much. Those 11 are voided with the rest anyway: a half-filled metrics object is not a placeholder, so the gate would stay half-dead with nothing to signal it. Removed span values, for reference (ms, p50/p95/p99): consensus.accept = 1.059405940594059/9.749999999999996/23.704545454545432; consensus.ledger_close = 0.5284697508896797/1.511111111111103/7.878571428571429; ledger.build = 0.7412060301507538/4.611111111111112/7.541666666666674; ledger.store = 0.5/0.95/0.9900000000000001; ledger.validate = 0.5283687943262412/1.3666666666666627/6.699999999999978; rpc.ws_message = 0.5026522773001647/0.9550393268703128/0.9952515090543261; tx.apply = 0.6330472103004292/4.203389830508474/5.083333333333319; tx.process = 0.5042801992591597/0.9581323781882418/0.998474791584883. job.* entries were removed earlier, on 2026-08-21, for the same reason on the native microsecond ladder: its first edge was 100us with 99.3% of job_queued_us samples beneath it, so job.acceptLedger.queued.p95 = 96.79us was 0.95/0.9926 x 100. That earlier note also claimed the gate then \"gates only the span metrics, which are unaffected\" -- that was wrong, for the reasons recorded above. Removed job values, for reference (us): job.acceptLedger.queued.p95=96.79, job.acceptLedger.running.p95=10562.50, job.transaction.queued.p95=478.97, job.transaction.running.p95=494.14. Both groups need recapture on a node running the re-cut ladders. captured_at, git_sha, profile and window below describe the voided run, not a live baseline.",
-  "captured_at": "2026-06-05T18:41:52Z",
-  "git_sha": "fd1c8c6060f7a15cc9e65b16f99629d9ab7ac7dc",
-  "metrics": {},
-  "placeholder": true,
+  "captured_at": "2026-08-24T21:22:37Z",
+  "git_sha": "6a82fc6f37c9e6cb747722906e83aaf949d100b8",
+  "metrics": {
+    "job.acceptLedger.queued.p95": {
+      "unit": "us",
+      "value": 91.10576923076925
+    },
+    "job.acceptLedger.running.p95": {
+      "unit": "us",
+      "value": 17428.571428571428
+    },
+    "job.transaction.queued.p95": {
+      "unit": "us",
+      "value": 476.6129032258061
+    },
+    "job.transaction.running.p95": {
+      "unit": "us",
+      "value": 427.1084337349388
+    },
+    "span.consensus.accept.p50": {
+      "unit": "ms",
+      "value": 1.7435897435897438
+    },
+    "span.consensus.accept.p95": {
+      "unit": "ms",
+      "value": 8.9296875
+    },
+    "span.consensus.accept.p99": {
+      "unit": "ms",
+      "value": 15.150000000000082
+    },
+    "span.consensus.ledger_close.p50": {
+      "unit": "ms",
+      "value": 0.15142857142857144
+    },
+    "span.consensus.ledger_close.p95": {
+      "unit": "ms",
+      "value": 0.49328358208955236
+    },
+    "span.consensus.ledger_close.p99": {
+      "unit": "ms",
+      "value": 0.9314285714285726
+    },
+    "span.ledger.build.p50": {
+      "unit": "ms",
+      "value": 1.0612244897959187
+    },
+    "span.ledger.build.p95": {
+      "unit": "ms",
+      "value": 4.679591836734694
+    },
+    "span.ledger.build.p99": {
+      "unit": "ms",
+      "value": 5.075000000000041
+    },
+    "span.ledger.store.p50": {
+      "unit": "ms",
+      "value": 0.005
+    },
+    "span.ledger.store.p95": {
+      "unit": "ms",
+      "value": 0.0095
+    },
+    "span.ledger.store.p99": {
+      "unit": "ms",
+      "value": 0.0099
+    },
+    "span.ledger.validate.p50": {
+      "unit": "ms",
+      "value": 0.07787769784172663
+    },
+    "span.ledger.validate.p95": {
+      "unit": "ms",
+      "value": 0.2404310344827586
+    },
+    "span.ledger.validate.p99": {
+      "unit": "ms",
+      "value": 1.060000000000033
+    },
+    "span.rpc.ws_message.p50": {
+      "unit": "ms",
+      "value": 0.15990942879094971
+    },
+    "span.rpc.ws_message.p95": {
+      "unit": "ms",
+      "value": 0.8443074003795064
+    },
+    "span.rpc.ws_message.p99": {
+      "unit": "ms",
+      "value": 0.9878368121442125
+    },
+    "span.tx.apply.p50": {
+      "unit": "ms",
+      "value": 0.791666666666667
+    },
+    "span.tx.apply.p95": {
+      "unit": "ms",
+      "value": 4.627368421052632
+    },
+    "span.tx.apply.p99": {
+      "unit": "ms",
+      "value": 4.959157894736842
+    },
+    "span.tx.process.p50": {
+      "unit": "ms",
+      "value": 0.34281067382135394
+    },
+    "span.tx.process.p95": {
+      "unit": "ms",
+      "value": 0.7239865112994349
+    },
+    "span.tx.process.p99": {
+      "unit": "ms",
+      "value": 0.9945148151511076
+    }
+  },
   "profile": "full-validation",
   "schema_version": 1,
   "window": "3m"

From e4926f55bed55846c6e2a9c2d15c1104e9eae8b1 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Mon, 24 Aug 2026 23:29:30 +0100
Subject: [PATCH 29/38] fix(telemetry): derive workload gate bounds from the
 bucket above the baseline

The gate could not catch a regression on any sub-millisecond span.
compare_to_baseline.py requires both the percentage and the absolute bound to
breach, and every span shared one flat absolute bound of 10 ms (15 ms for p99)
calibrated for a 5-25 ms band the spans do not occupy. Against the baseline
captured on 2026-08-24, where 18 of the 28 quantiles gated at the time sat
below 1 ms, that bound sat 1.15x to 2000x above the metric it guarded, so the
AND never fired: a 100x regression injected into span.ledger.store.p95 reported
0 regressions and exit 0. Injecting a 10x regression into each key in turn was
caught on only 5 of 28.

Give every gated key its own absolute bound, equal to the distance from its
baseline to hi_next, the edge above the top of the bucket the baseline sits in.
The trip point is then exactly hi_next, so the gate fires only once the reading
clears the bucket above the baseline's own. That is the property a multiple of
the enclosing bucket width cannot provide: after the quantile crosses hi, the
interpolation happens across the next bucket, which on this ladder is up to
eight times wider, so no multiple of the enclosing width bounds the excursion.
Measured with a model-free reachability test, a single bucket crossing can
produce a false regression on 2 of 25 keys under the old flat bound and 0 of 25
under this rule. The smallest catchable regression is 2.02x to 9.43x per key.

The job queue bound had the same shape of problem on three of its four keys
(42x, 47x, 220x before). Defaults now sit at each ladder floor, leaving the
percentage bound operative for a metric that somehow reaches them.

Drop span.ledger.store from the gated surface. Its captured quantiles were
0.005, 0.0095 and 0.0099 ms, which is the ladder's 0.01 ms floor times the
quantile: every sample lands under 10 us, so the reported value does not move
even if each store slows from 2 us to 9 us. No bound can gate it. Presence is
still asserted by expected_spans.json and the integration test, and the rate is
still on the ledger-operations dashboard.

Add check_regression_bounds.py, wired into the same workflow step as the bucket
parity check. It fails when a bound is not the one its own baseline implies,
when a gated key has no override, when the baseline and metric surface disagree,
when the percentage bound would become operative, and when a baseline carries
the ladder floor signature. This gate has now broken three times through the
same drift between ladder, baseline and bounds, so documentation alone is not
enough.

compare_to_baseline.py is unchanged: its existing per-metric override mechanism
already expresses all of this.

A missing, unreadable or malformed input makes that check exit 1 naming the
input, rather than reporting success without having checked anything; only a
placeholder baseline, the documented bootstrap state, still exits 0. Its own
tests cover both halves of that contract plus one case per rule, and run in the
workflow before the check so a broken rule reads as a broken rule.
---
 .../telemetry/check_regression_bounds.py      | 284 ++++++++++++++++++
 .../telemetry/test_check_regression_bounds.py | 226 ++++++++++++++
 .../workflows/reusable-check-otel-naming.yml  |  17 ++
 docker/telemetry/workload/README.md           |   6 +-
 docker/telemetry/workload/baselines/README.md | 118 +++++++-
 .../workload/baselines/baseline-timings.json  |  12 -
 .../workload/regression-metrics.json          |   2 +-
 .../workload/regression-thresholds.json       | 167 +++++++++-
 docs/telemetry-runbook.md                     |  46 ++-
 9 files changed, 832 insertions(+), 46 deletions(-)
 create mode 100644 .github/scripts/telemetry/check_regression_bounds.py
 create mode 100644 .github/scripts/telemetry/test_check_regression_bounds.py

diff --git a/.github/scripts/telemetry/check_regression_bounds.py b/.github/scripts/telemetry/check_regression_bounds.py
new file mode 100644
index 0000000000..94189fc04d
--- /dev/null
+++ b/.github/scripts/telemetry/check_regression_bounds.py
@@ -0,0 +1,284 @@
+#!/usr/bin/env python3
+"""Assert every workload-gate absolute bound is the one its own baseline implies.
+
+The regression gate in ``docker/telemetry/workload`` fails CI when a span or
+job-queue quantile grows. Whether it *can* fail is decided by
+``regression-thresholds.json``, and that file's numbers are derived from
+``baselines/baseline-timings.json`` plus the two histogram ladders. Nothing
+tied the three together, and the gate has now been broken three times by the
+same class of drift:
+
+  1. the microsecond ladder's floor moved 100us -> 1us, voiding every
+     job_queue baseline captured before it;
+  2. the spanmetrics ladder's floor moved 1ms -> 0.01ms, voiding every
+     sub-millisecond span baseline captured before it;
+  3. the absolute bounds stayed calibrated for a 5-25ms band the spans had
+     left, so a 100x regression on ``span.ledger.store.p95`` reported zero
+     regressions and exit 0.
+
+Each time the gate stayed green, which is indistinguishable from a passing
+build. Documentation did not prevent recurrence, so this is a check.
+
+The rule it enforces is the one recorded in ``regression-thresholds.json``
+under ``_absolute_bound_derivation``: for a baseline sitting in the half-open
+bucket ``(lo, hi]`` of its ladder, with ``hi_next`` the next edge above ``hi``,
+
+    max_abs_increase_* == hi_next - baseline
+
+so the gate trips only when the reading clears the bucket *above* the
+baseline's own. Five rules are checked:
+
+  A  the baseline's key set equals the surface ``regression-metrics.json``
+     declares (a stale key left behind reads as covered but never gates);
+  B  every gated key has a per-metric override, not a fallback default;
+  C  each absolute bound equals ``hi_next - baseline``;
+  D  each percentage bound stays below ``100 * bound / baseline``, so the
+     absolute bound remains the operative half of the ``AND`` -- the span
+     ladder's 2s/3s/4s edges are only 1.25x-1.5x apart, where this silently
+     stops being true;
+  E  no baseline carries the ladder-floor signature ``quantile x first_edge``,
+     which means every sample landed in the first bucket and the number is
+     interpolation arithmetic rather than a latency.
+
+A PLACEHOLDER baseline -- ``"placeholder": true`` or an empty ``metrics``
+object -- exits 0, because that is the documented bootstrap state and CI has to
+stay green while a baseline is being recaptured. A missing, unreadable or
+malformed input is a different thing and exits 1: a check that reports success
+without having checked anything is the same green-build-that-is-not failure this
+script exists to prevent, so renaming or deleting one of its inputs must not
+silence it.
+
+Exit 0 when every rule holds, 1 with per-key detail otherwise.
+"""
+
+import json
+import re
+import sys
+from pathlib import Path
+
+WORKLOAD = Path("docker/telemetry/workload")
+BASELINE = WORKLOAD / "baselines/baseline-timings.json"
+THRESHOLDS = WORKLOAD / "regression-thresholds.json"
+METRICS = WORKLOAD / "regression-metrics.json"
+COLLECTOR = Path("docker/telemetry/otel-collector-config.yaml")
+HEADER = Path("include/xrpl/telemetry/HistogramBuckets.h")
+
+UNIT_TO_MS = {"ms": 1.0, "s": 1000.0}
+# A bound may differ from the derived value only by double round-tripping.
+REL_TOLERANCE = 1e-12
+
+
+def read_text_or_exit(path):
+    """Read a required text input, or exit 1 naming the input that failed."""
+    try:
+        return path.read_text()
+    except OSError as exc:
+        sys.exit(f"{path}: required input could not be read -- {exc}")
+
+
+def read_json_or_exit(path):
+    """Read and parse a required JSON input, or exit 1 naming what failed."""
+    try:
+        return json.loads(read_text_or_exit(path))
+    except json.JSONDecodeError as exc:
+        sys.exit(f"{path}: required input is not valid JSON -- {exc}")
+
+
+def span_edges_ms():
+    """Parse the spanmetrics bucket list, normalising each edge to milliseconds."""
+    match = re.search(r"buckets:\s*\[(.*?)\]", read_text_or_exit(COLLECTOR), re.S)
+    if not match:
+        sys.exit(f"{COLLECTOR}: no 'buckets:' list found")
+    edges = []
+    for raw in match.group(1).split(","):
+        token = raw.strip()
+        if not token:
+            continue
+        parsed = re.fullmatch(r"([0-9.]+)(ms|s)", token)
+        if not parsed:
+            sys.exit(f"{COLLECTOR}: cannot parse bucket edge {token!r}")
+        edges.append(float(parsed.group(1)) * UNIT_TO_MS[parsed.group(2)])
+    return edges
+
+
+def microsecond_edges():
+    """Parse kMicrosecondBuckets out of the header that owns every ladder."""
+    match = re.search(r"kMicrosecondBuckets\{(.*?)\};", read_text_or_exit(HEADER), re.S)
+    if not match:
+        sys.exit(f"{HEADER}: kMicrosecondBuckets not found")
+    return [
+        float(token.strip().replace("'", ""))
+        for token in match.group(1).split(",")
+        if token.strip()
+    ]
+
+
+def declared_keys(metrics_cfg):
+    """Rebuild the flat key set regression-metrics.json declares.
+
+    Deliberately reimplemented rather than imported from ``prom_queries.py``,
+    which pulls in aiohttp; CI telemetry checks stay dependency-free. The key
+    format is fixed by that file's own ``_key_format`` field.
+    """
+    keys = set()
+    spans = metrics_cfg.get("spans", {})
+    for name in spans.get("names", []):
+        for quantile in spans.get("_quantiles", []):
+            keys.add(f"span.{name}.p{_quantile_label(quantile)}")
+    jobs = metrics_cfg.get("job_queue", {})
+    for name in jobs.get("names", []):
+        for phase in jobs.get("_phases", []):
+            for quantile in jobs.get("_quantiles", []):
+                keys.add(f"job.{name}.{phase}.p{_quantile_label(quantile)}")
+    return keys
+
+
+def _quantile_label(quantile):
+    """0.95 -> '95', 0.5 -> '50', matching capture_timings.py's key format."""
+    return f"{quantile * 100:g}".replace(".", "")
+
+
+def brackets(value, edges):
+    """Return ``(lo, hi, hi_next)`` for the bucket ``(lo, hi]`` holding value."""
+    padded = [0.0] + list(edges)
+    for i in range(1, len(padded)):
+        if value <= padded[i]:
+            hi_next = padded[i + 1] if i + 1 < len(padded) else None
+            return padded[i - 1], padded[i], hi_next
+    return None, None, None
+
+
+def resolve_override(key, thresholds):
+    """Return the override rule for a key, or None if it falls back to defaults."""
+    group, quantile = key.rsplit(".", 1)
+    return thresholds.get("overrides", {}).get(group, {}).get(quantile)
+
+
+def check_key(key, entry, thresholds, ladders):
+    """Apply rules B, C, D and E to one gated key. Returns a list of failures."""
+    value, unit = entry.get("value"), entry.get("unit", "")
+    edges = ladders.get(unit)
+    if value is None or edges is None:
+        return [f"{key}: baseline has no value, or unknown unit {unit!r}"]
+
+    failures = []
+    first_edge = edges[0]
+    quantile = int(key.rsplit(".p", 1)[1]) / 100.0
+    if abs(value - quantile * first_edge) <= 1e-9 * first_edge:
+        failures.append(
+            f"{key}: baseline {value!r} equals quantile {quantile:g} x the ladder "
+            f"floor {first_edge:g}{unit}, so every sample landed in the first "
+            f"bucket and this is bucket arithmetic, not a latency. No absolute "
+            f"bound can gate it -- add a finer ladder edge or drop the metric "
+            f"from {METRICS} (rule E)"
+        )
+        return failures
+
+    _, _, hi_next = brackets(value, edges)
+    if hi_next is None:
+        return [
+            f"{key}: baseline {value!r}{unit} sits in or above the ladder's top "
+            f"bucket, so there is no hi_next to derive a bound from -- extend the "
+            f"ladder (rule C)"
+        ]
+
+    rule = resolve_override(key, thresholds)
+    if rule is None:
+        failures.append(
+            f"{key}: no per-metric override, so it falls back to the defaults and "
+            f"gates on the percentage bound alone. Add an override with "
+            f"max_abs_increase = {hi_next - value!r} (rule B)"
+        )
+        return failures
+
+    bound = rule.get("max_abs_increase_ms", rule.get("max_abs_increase_us"))
+    expected = hi_next - value
+    if bound is None or abs(bound - expected) > REL_TOLERANCE * expected:
+        failures.append(
+            f"{key}: absolute bound is {bound!r}, expected {expected!r} "
+            f"(hi_next {hi_next:g} - baseline {value!r}) (rule C)"
+        )
+
+    pct = rule.get("max_pct_increase")
+    if pct is None:
+        failures.append(f"{key}: no max_pct_increase, so the metric never gates")
+    elif bound is not None and pct >= 100.0 * bound / value:
+        failures.append(
+            f"{key}: max_pct_increase {pct:g}% is at or above the absolute bound's "
+            f"{100.0 * bound / value:.1f}% of baseline, so the percentage bound "
+            f"becomes the operative one and the bucket guarantee is lost. Lower it "
+            f"or document the metric as percentage-gated (rule D)"
+        )
+    return failures
+
+
+def main():
+    missing = [
+        p for p in (BASELINE, THRESHOLDS, METRICS, COLLECTOR, HEADER) if not p.exists()
+    ]
+    if missing:
+        print("Cannot check workload regression bounds.", file=sys.stderr)
+        for path in missing:
+            print(f"  {path}: required input is absent", file=sys.stderr)
+        print(
+            "\nA missing input is not a reason to pass. Deleting or renaming one of\n"
+            "these would otherwise leave the gate reporting success without having\n"
+            "checked a single bound -- the failure this script exists to prevent. If\n"
+            "the workload harness has genuinely moved, update the paths here.",
+            file=sys.stderr,
+        )
+        return 1
+
+    baseline = read_json_or_exit(BASELINE)
+    thresholds = read_json_or_exit(THRESHOLDS)
+    metrics_cfg = read_json_or_exit(METRICS)
+
+    if baseline.get("placeholder") is True or not baseline.get("metrics"):
+        print("OK: baseline is a placeholder, bounds cannot be derived yet")
+        return 0
+
+    ladders = {"ms": span_edges_ms(), "us": microsecond_edges()}
+    gated = baseline["metrics"]
+    failures = []
+
+    declared = declared_keys(metrics_cfg)
+    for key in sorted(set(gated) - declared):
+        failures.append(
+            f"{key}: in the baseline but not declared by {METRICS}, so it is "
+            f"reported every run and can never gate -- remove it (rule A)"
+        )
+    for key in sorted(declared - set(gated)):
+        failures.append(
+            f"{key}: declared by {METRICS} but absent from the baseline, so it "
+            f"never gates -- capture a baseline for it (rule A)"
+        )
+
+    for key in sorted(gated):
+        if key in declared:
+            failures.extend(check_key(key, gated[key], thresholds, ladders))
+
+    if not failures:
+        print(
+            f"OK: {len(gated)} gated key(s); every absolute bound equals "
+            f"hi_next - baseline, every key has an override, and the absolute "
+            f"bound is the operative half of the AND for all of them"
+        )
+        return 0
+
+    print(
+        "Workload regression bounds are not derived from the baseline.", file=sys.stderr
+    )
+    for failure in failures:
+        print(f"  {failure}", file=sys.stderr)
+    print(
+        f"\nThe rule is recorded in {THRESHOLDS} under _absolute_bound_derivation:\n"
+        "a bound is hi_next - baseline, where hi_next is the edge above the top of\n"
+        "the bucket holding the baseline. Refreshing a baseline therefore obliges\n"
+        "you to re-derive its bound; see baselines/README.md.",
+        file=sys.stderr,
+    )
+    return 1
+
+
+if __name__ == "__main__":
+    sys.exit(main())
diff --git a/.github/scripts/telemetry/test_check_regression_bounds.py b/.github/scripts/telemetry/test_check_regression_bounds.py
new file mode 100644
index 0000000000..4c7a4000cb
--- /dev/null
+++ b/.github/scripts/telemetry/test_check_regression_bounds.py
@@ -0,0 +1,226 @@
+#!/usr/bin/env python3
+"""Tests for check_regression_bounds.py.
+
+The checker reads five files by path relative to the working directory, so each
+test assembles a scratch tree holding copies of the real inputs, mutates one
+thing, and runs the checker as a subprocess there. Testing the real entry point
+is deliberate: the contract under test is the exit code CI reads, and an
+in-process call would not exercise it.
+
+Two groups:
+
+* the input-handling contract -- a placeholder baseline must PASS because that
+  is the documented bootstrap state, while a missing, unreadable or malformed
+  input must FAIL. A checker that returns success without having checked
+  anything is the failure this whole gate exists to prevent;
+* one case per rule (A to E), so a rule that stops flagging is caught.
+
+stdlib unittest only; the repo installs no third-party runner for CI.
+"""
+
+import json
+import os
+import shutil
+import stat
+import subprocess
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+SCRIPT_DIR = Path(__file__).resolve().parent
+CHECKER = SCRIPT_DIR / "check_regression_bounds.py"
+REPO = SCRIPT_DIR.parents[2]
+
+WORKLOAD = "docker/telemetry/workload"
+BASELINE = f"{WORKLOAD}/baselines/baseline-timings.json"
+THRESHOLDS = f"{WORKLOAD}/regression-thresholds.json"
+METRICS = f"{WORKLOAD}/regression-metrics.json"
+COLLECTOR = "docker/telemetry/otel-collector-config.yaml"
+HEADER = "include/xrpl/telemetry/HistogramBuckets.h"
+INPUTS = (BASELINE, THRESHOLDS, METRICS, COLLECTOR, HEADER)
+
+
+class CheckerCase(unittest.TestCase):
+    """Base class giving each test an isolated copy of the checker's inputs."""
+
+    def setUp(self):
+        self.tree = Path(tempfile.mkdtemp())
+        self.addCleanup(self._cleanup)
+        for rel in INPUTS:
+            dest = self.tree / rel
+            dest.parent.mkdir(parents=True, exist_ok=True)
+            shutil.copy(REPO / rel, dest)
+        script = self.tree / ".github/scripts/telemetry/check_regression_bounds.py"
+        script.parent.mkdir(parents=True, exist_ok=True)
+        shutil.copy(CHECKER, script)
+
+    def _cleanup(self):
+        for path in self.tree.rglob("*"):
+            if path.is_file():
+                path.chmod(stat.S_IRUSR | stat.S_IWUSR)
+        shutil.rmtree(self.tree, ignore_errors=True)
+
+    def run_checker(self):
+        """Run the checker in the scratch tree, returning (code, stdout+stderr)."""
+        proc = subprocess.run(
+            [sys.executable, ".github/scripts/telemetry/check_regression_bounds.py"],
+            cwd=self.tree,
+            capture_output=True,
+            text=True,
+        )
+        return proc.returncode, proc.stdout + proc.stderr
+
+    def edit_json(self, rel, mutate):
+        """Load a scratch input, hand it to mutate(), write it back."""
+        path = self.tree / rel
+        data = json.loads(path.read_text())
+        mutate(data)
+        path.write_text(json.dumps(data, indent=2))
+
+
+class TestInputHandling(CheckerCase):
+    """A placeholder passes; a missing or broken input must not."""
+
+    def test_unmodified_tree_passes(self):
+        code, out = self.run_checker()
+        self.assertEqual(code, 0, out)
+        self.assertIn("gated key(s)", out)
+
+    def test_placeholder_flag_passes(self):
+        self.edit_json(BASELINE, lambda d: d.update(placeholder=True))
+        code, out = self.run_checker()
+        self.assertEqual(code, 0, out)
+        self.assertIn("placeholder", out)
+
+    def test_empty_metrics_baseline_passes(self):
+        self.edit_json(BASELINE, lambda d: d.update(metrics={}))
+        code, out = self.run_checker()
+        self.assertEqual(code, 0, out)
+        self.assertIn("placeholder", out)
+
+    def test_missing_baseline_fails_naming_the_input(self):
+        (self.tree / BASELINE).unlink()
+        code, out = self.run_checker()
+        self.assertEqual(code, 1, out)
+        self.assertIn("baseline-timings.json", out)
+
+    def test_missing_collector_config_fails_naming_the_input(self):
+        (self.tree / COLLECTOR).unlink()
+        code, out = self.run_checker()
+        self.assertEqual(code, 1, out)
+        self.assertIn("otel-collector-config.yaml", out)
+
+    @unittest.skipIf(os.geteuid() == 0, "root ignores the read permission bit")
+    def test_unreadable_baseline_fails(self):
+        (self.tree / BASELINE).chmod(0)
+        code, out = self.run_checker()
+        self.assertEqual(code, 1, out)
+        self.assertIn("baseline-timings.json", out)
+        self.assertIn("could not be read", out)
+        self.assertNotIn("Traceback", out)
+
+    def test_malformed_baseline_json_fails(self):
+        (self.tree / BASELINE).write_text("{ not json")
+        code, out = self.run_checker()
+        self.assertEqual(code, 1, out)
+        self.assertIn("valid JSON", out)
+
+    def test_malformed_thresholds_json_fails(self):
+        (self.tree / THRESHOLDS).write_text("]")
+        code, out = self.run_checker()
+        self.assertEqual(code, 1, out)
+        self.assertIn("valid JSON", out)
+
+
+class TestRules(CheckerCase):
+    """One case per rule, so a rule that stops flagging is caught."""
+
+    def test_rule_a_flags_baseline_key_not_declared(self):
+        self.edit_json(
+            BASELINE,
+            lambda d: d["metrics"].update(
+                {"span.rpc.process.p99": {"unit": "ms", "value": 9.0}}
+            ),
+        )
+        code, out = self.run_checker()
+        self.assertEqual(code, 1, out)
+        self.assertIn("(rule A)", out)
+
+    def test_rule_a_flags_declared_key_without_baseline(self):
+        self.edit_json(METRICS, lambda d: d["spans"]["names"].append("consensus.round"))
+        code, out = self.run_checker()
+        self.assertEqual(code, 1, out)
+        self.assertIn("(rule A)", out)
+
+    def test_rule_b_flags_missing_override(self):
+        self.edit_json(THRESHOLDS, lambda d: d["overrides"].pop("span.ledger.build"))
+        code, out = self.run_checker()
+        self.assertEqual(code, 1, out)
+        self.assertIn("(rule B)", out)
+
+    def test_rule_c_flags_rounded_bound(self):
+        self.edit_json(
+            THRESHOLDS,
+            lambda d: d["overrides"]["span.tx.process"]["p99"].update(
+                max_abs_increase_ms=4.0055
+            ),
+        )
+        code, out = self.run_checker()
+        self.assertEqual(code, 1, out)
+        self.assertIn("(rule C)", out)
+
+    def test_rule_c_accepts_bound_within_relative_tolerance(self):
+        """The tolerance is 1e-12 relative, not exact equality."""
+        exact = 4.005485184848892
+        self.edit_json(
+            THRESHOLDS,
+            lambda d: d["overrides"]["span.tx.process"]["p99"].update(
+                max_abs_increase_ms=exact * (1 + 5e-13)
+            ),
+        )
+        code, out = self.run_checker()
+        self.assertEqual(code, 0, out)
+
+    def test_rule_d_flags_percentage_bound_becoming_operative(self):
+        self.edit_json(
+            THRESHOLDS,
+            lambda d: d["overrides"]["span.tx.apply"]["p99"].update(
+                max_pct_increase=150.0
+            ),
+        )
+        code, out = self.run_checker()
+        self.assertEqual(code, 1, out)
+        self.assertIn("(rule D)", out)
+
+    def test_rule_e_flags_ladder_floor_signature(self):
+        """ledger.store's quantiles were the ladder floor times the quantile."""
+        store = {"p50": 0.005, "p95": 0.0095, "p99": 0.0099}
+        self.edit_json(METRICS, lambda d: d["spans"]["names"].append("ledger.store"))
+        self.edit_json(
+            BASELINE,
+            lambda d: d["metrics"].update(
+                {
+                    f"span.ledger.store.{q}": {"unit": "ms", "value": v}
+                    for q, v in store.items()
+                }
+            ),
+        )
+        self.edit_json(
+            THRESHOLDS,
+            lambda d: d["overrides"].update(
+                {
+                    "span.ledger.store": {
+                        q: {"max_pct_increase": 50.0, "max_abs_increase_ms": 0.05 - v}
+                        for q, v in store.items()
+                    }
+                }
+            ),
+        )
+        code, out = self.run_checker()
+        self.assertEqual(code, 1, out)
+        self.assertIn("(rule E)", out)
+
+
+if __name__ == "__main__":
+    unittest.main()
diff --git a/.github/workflows/reusable-check-otel-naming.yml b/.github/workflows/reusable-check-otel-naming.yml
index a37e7e1632..a92164572b 100644
--- a/.github/workflows/reusable-check-otel-naming.yml
+++ b/.github/workflows/reusable-check-otel-naming.yml
@@ -41,3 +41,20 @@ jobs:
         # spans reached 30s, so every quantile above 5s reported a flat 5000.
         # Nothing but a check keeps two lists in step.
         run: python .github/scripts/telemetry/check_bucket_parity.py
+      - name: Test the workload regression-bounds checker
+        # Its own tests, run before the check itself so a broken rule is
+        # reported as a broken rule rather than as a threshold violation. They
+        # also pin the input-handling contract: a placeholder baseline passes
+        # (that is the documented bootstrap state) while a missing, unreadable
+        # or malformed input fails, so deleting an input cannot silence the
+        # check. stdlib unittest only, as with the naming checker.
+        run: python -m unittest discover -s .github/scripts/telemetry -p 'test_*.py' --verbose
+      - name: Check workload regression bounds
+        # The workload gate's absolute bounds are derived from the committed
+        # baseline plus the two ladders, and nothing tied the three together.
+        # That let the gate break three times the same way -- the microsecond
+        # floor moved, the span floor moved, then the bounds stayed calibrated
+        # for a band the spans had left, so a 100x regression reported zero
+        # regressions and exit 0. Every failure looked like a green build.
+        # This asserts each bound is still the one its own baseline implies.
+        run: python .github/scripts/telemetry/check_regression_bounds.py
diff --git a/docker/telemetry/workload/README.md b/docker/telemetry/workload/README.md
index adba3a7864..4e5a1b9b4c 100644
--- a/docker/telemetry/workload/README.md
+++ b/docker/telemetry/workload/README.md
@@ -264,7 +264,11 @@ Per-run tuning:
 - `REGRESSION_WINDOW` env var overrides the default Prometheus `rate()`
   window (`3m`). Keep close to the workload duration.
 - Metric surface lives in `regression-metrics.json`; thresholds in
-  `regression-thresholds.json`; both are reviewed changes.
+  `regression-thresholds.json`; both are reviewed changes. Each gated key's
+  absolute bound is `hi_next - baseline` — the distance from its baseline to the
+  top of the next bucket up — so refreshing the baseline obliges you to
+  re-derive the bounds. See `_absolute_bound_derivation` in that file;
+  `.github/scripts/telemetry/check_regression_bounds.py` enforces it in CI.
 
 See [`baselines/README.md`](./baselines/README.md) for the baseline
 lifecycle and refresh process.
diff --git a/docker/telemetry/workload/baselines/README.md b/docker/telemetry/workload/baselines/README.md
index c80e1bf9dc..119bef5e5f 100644
--- a/docker/telemetry/workload/baselines/README.md
+++ b/docker/telemetry/workload/baselines/README.md
@@ -25,27 +25,109 @@ was invoked with. Capture and comparison are profile-agnostic — they only read
 Prometheus — so all existing profiles (`full-validation`, `quick-smoke`, `stress`)
 continue to work unchanged.
 
-## Current state: the baseline is a placeholder
+## Current state: 25 metrics gate, on a baseline captured 2026-08-24
 
-`baseline-timings.json` currently carries `"placeholder": true` and an empty `metrics` object,
-so **no metric gates right now**. Its entries were captured on 2026-06-05 against a spanmetrics
-ladder that was re-cut on 2026-08-04 in `3860c93db2`, which makes every sub-millisecond quantile
+`baseline-timings.json` holds real captured values for the 25 keys the harness gates. The
+previous entries, captured on 2026-06-05, were voided into a placeholder first: they predated the
+spanmetrics ladder re-cut of 2026-08-04 (`3860c93db2`), which made every sub-millisecond quantile
 in that capture bucket-edge arithmetic rather than a latency (a p95 of `0.95` ms is `0.95 × 1 ms`).
 Because the comparator only flags a metric when the current value _exceeds_ the baseline, a
-stale-high baseline passes everything silently — so the entries were voided instead of left in
-place. The file's `_note` records why, and which numbers were dropped.
+stale-high baseline passes everything silently, so the entries had to be dropped rather than left
+in place. They stay retrievable from this file's git history.
 
-To restore gating, follow [Bootstrapping the baseline](#bootstrapping-the-baseline) below — a
-placeholder is exactly the state that loop expects. Pasting the CI block **replaces the whole
-file**, `_note` included; that is intended, and the voided numbers stay retrievable from this
-file's git history.
+**A placeholder must not outlive one run.** CI stays green the whole time one stands, so an
+un-copied block is not a failure anyone will notice — it is a silent loss of regression coverage
+that looks identical to a passing gate. Voiding a baseline is the one hand edit this file allows;
+_setting_ one always comes from a printed CI block, per the "Refreshing the baseline" rule below.
 
-**Do not let the placeholder outlive one run.** CI stays green the whole time the placeholder
-stands, so an un-copied block is not a failure anyone will notice — it is a silent loss of
-regression coverage that looks identical to a passing gate.
+## Absolute bounds are derived per metric, from the ladder
 
-Voiding a baseline is the one hand edit this file allows; _setting_ one always comes from a
-printed CI block, per the "Refreshing the baseline" rule below.
+`../regression-thresholds.json` gives every gated key its own `max_abs_increase_*`, equal to
+**`hi_next − baseline`**: locate the baseline in the half-open bucket `(lo, hi]` of its ladder,
+take `hi_next` as the next edge above `hi`, and the bound is the distance from the baseline to
+`hi_next`. The trip point is therefore exactly `hi_next` — the gate fires only once the reading
+clears the bucket **above** the baseline's own.
+
+That is what buys the guarantee. `histogram_quantile` returns a value interpolated inside
+whichever bucket the true quantile falls in, so any reading taken while the quantile is still in
+the baseline's bucket, or anywhere in the one immediately above, is at most `hi_next` and cannot
+fire. Firing needs the quantile to have moved at least two buckets up. A multiple of the
+_enclosing_ bucket's width cannot deliver this, because once the quantile crosses `hi` the
+interpolation happens across the **next** bucket, which on this ladder is up to 8x wider —
+`(0.5, 1]` is 0.5 ms wide and `(1, 5]` is 4 ms wide. The full derivation, both ladders, and a
+per-key table of the arithmetic are in that file's `_absolute_bound_derivation` and
+`_derivation_table`.
+
+Two earlier generations of this bound were wrong, in opposite directions:
+
+| generation                     | bound                                        | 10x regression caught | single-crossing false positive reachable |
+| ------------------------------ | -------------------------------------------- | --------------------- | ---------------------------------------- |
+| flat                           | 10 ms `p50`/`p95`, 15 ms `p99`, 20000 us job | 5 / 28 keys           | 2 / 25 keys                              |
+| 2 × enclosing bucket width     | per metric                                   | 28 / 28 keys          | **21 / 25 keys**                         |
+| `hi_next − baseline` (current) | per metric                                   | 25 / 25 keys          | **0 / 25 keys**                          |
+
+The flat bound was calibrated for a 5-25 ms band the spans do not occupy: 18 of the 28
+quantiles gated at the time sat below 1 ms, so it sat 1.15x to 2000x above the metric it guarded,
+and because the rule is an `AND` the percentage bound could never carry a regression alone. A 100x
+regression injected into `span.ledger.store.p95` reported **0 regressions, exit 0**. The second
+generation fixed the magnitude but kept an assumption that does not hold — that the reading's
+excursion is bounded by the enclosing bucket's width — which put 21 of 25 trip points inside the
+adjacent bucket, so a single legitimate bucket crossing could turn CI red.
+
+**Refreshing the baseline means re-deriving the bounds**, because a refreshed value can land in a
+different bucket and so get a different `hi_next`. This is no longer a documentation-only rule:
+[`.github/scripts/telemetry/check_regression_bounds.py`](../../../../.github/scripts/telemetry/check_regression_bounds.py)
+fails CI when a bound is not the one its own baseline implies, when a gated key has no override,
+when a baseline key is not declared by `../regression-metrics.json` (or the reverse), when the
+percentage bound would become the operative one, and when a baseline carries the ladder-floor
+signature described below.
+
+### Which keys are only weakly guarded
+
+The guarantee costs sensitivity where the ladder is coarse: the detection floor is
+`hi_next / baseline`, so a baseline sitting just above an edge is guarded loosely. Measured over
+the current baseline the floor ranges 2.02x to 9.43x. Do **not** read these as guarded:
+
+| key                               | baseline   | fires at  | floor |
+| --------------------------------- | ---------- | --------- | ----- |
+| `span.ledger.validate.p99`        | 1.0600 ms  | 10 ms     | 9.43x |
+| `span.ledger.build.p50`           | 1.0612 ms  | 10 ms     | 9.42x |
+| `span.tx.process.p95`             | 0.7240 ms  | 5 ms      | 6.91x |
+| `span.tx.apply.p50`               | 0.7917 ms  | 5 ms      | 6.32x |
+| `span.rpc.ws_message.p95`         | 0.8443 ms  | 5 ms      | 5.92x |
+| `job.acceptLedger.running.p95`    | 17428.6 us | 100000 us | 5.74x |
+| `span.consensus.accept.p50`       | 1.7436 ms  | 10 ms     | 5.74x |
+| `span.consensus.ledger_close.p99` | 0.9314 ms  | 5 ms      | 5.37x |
+| `span.rpc.ws_message.p99`         | 0.9878 ms  | 5 ms      | 5.06x |
+| `span.tx.process.p99`             | 0.9945 ms  | 5 ms      | 5.03x |
+
+`span.ledger.build.p50` is the one that matters most: ledger construction is the hot path this
+gate exists to guard, and at a 9.42x floor it could get almost ten times slower and still pass.
+All ten are limited by two 5x-wide
+ladder steps, 1 ms → 5 ms and 5000 us → 25000 us. The fix is a 2 ms edge (ideally 3 ms as well) in
+the collector's spanmetrics `buckets` list plus the matching entries in `kMillisecondBuckets`, and
+a 10000 us edge in `kMicrosecondBuckets`. That work belongs to the branch that owns the ladders.
+
+## Known exclusion: `ledger.store` is below the ladder's resolution
+
+`span.ledger.store` is **not** gated. The 2026-08-24 capture returned p50/p95/p99 of exactly
+`0.005` / `0.0095` / `0.0099` ms, which is `0.5` / `0.95` / `0.99 × 0.01` ms — the ladder's first
+edge times the quantile, the signature of every sample landing in the first bucket. Those numbers
+are interpolation arithmetic on the bucket floor, not latencies. It is physically plausible:
+[`LedgerMaster.cpp:463`](../../../../src/xrpld/app/ledger/detail/LedgerMaster.cpp#L463) wraps an
+in-memory `ledgerHistory_.insert`, which completes in single-digit microseconds.
+
+While all the mass stays under 10 us the reported quantile cannot move materially, so **no
+absolute bound can gate this key** — every `ledger.store` slowing from 2 us to 9 us, a 4.5x
+regression, leaves the reported value unchanged. Three keys that read as covered but cannot fire
+are worse than no keys, the same argument that excluded `rpc.process`, so they were removed from
+`../regression-metrics.json` rather than left in with a bound that looks derived.
+
+Restoring the key needs sub-10 us edges on the collector's spanmetrics ladder (for example
+`0.001ms` and `0.005ms`) plus the matching entries in `HistogramBuckets.h`. `ledger.store`
+presence is still asserted by `../expected_spans.json` and `docker/telemetry/integration-test.sh`,
+and its rate is still on the ledger-operations dashboard; only the latency gate drops it.
+`check_regression_bounds.py` rule E fails the build if a key with this signature is gated again.
 
 ## Bootstrapping the baseline
 
@@ -70,6 +152,12 @@ new numbers should become the norm, open a PR pasting the fresh timings into
 Do **not** edit `baseline-timings.json` by hand outside of this process — every entry
 should trace back to a real CI run so variance characteristics are preserved.
 
+Refreshing the baseline also obliges you to re-derive the absolute bounds in
+`../regression-thresholds.json`, per
+[Absolute bounds are derived per metric](#absolute-bounds-are-derived-per-metric-from-the-ladder).
+A value that moves into a different bucket needs a different bound, and a bound left behind
+either stops catching regressions or starts firing on quantization noise.
+
 ## The baseline is only valid at the log level it was captured at
 
 Every timing here is coupled to the `log_level` that `run-full-validation.sh` writes into
diff --git a/docker/telemetry/workload/baselines/baseline-timings.json b/docker/telemetry/workload/baselines/baseline-timings.json
index 9a1b452478..c02ca2b558 100644
--- a/docker/telemetry/workload/baselines/baseline-timings.json
+++ b/docker/telemetry/workload/baselines/baseline-timings.json
@@ -54,18 +54,6 @@
       "unit": "ms",
       "value": 5.075000000000041
     },
-    "span.ledger.store.p50": {
-      "unit": "ms",
-      "value": 0.005
-    },
-    "span.ledger.store.p95": {
-      "unit": "ms",
-      "value": 0.0095
-    },
-    "span.ledger.store.p99": {
-      "unit": "ms",
-      "value": 0.0099
-    },
     "span.ledger.validate.p50": {
       "unit": "ms",
       "value": 0.07787769784172663
diff --git a/docker/telemetry/workload/regression-metrics.json b/docker/telemetry/workload/regression-metrics.json
index 050ce10a4f..911868b595 100644
--- a/docker/telemetry/workload/regression-metrics.json
+++ b/docker/telemetry/workload/regression-metrics.json
@@ -2,6 +2,7 @@
   "_description": "Metric surface for the OTel-driven regression gate. Each entry names a metric, the quantiles to capture, and how to query Prometheus. The comparator compares current run against baseline-timings.json under these exact keys.",
   "_key_format": "{category}.{name}.p{quantile}  (e.g. span.tx.process.p99, job.transaction.queued.p95). Only the categories defined below are captured; there is no rpc_methods group, so no rpc.* key is produced or gated (FU-4).",
   "_excluded_spans": "rpc.process is deliberately absent from spans.names. It is created only in ServerHandler::processRequest() on the HTTP/JSON-RPC path, which the workload load generators, being WebSocket-only, never reach, so its quantiles were captured as null every run and could never gate. (The harness shell scripts do issue a few HTTP JSON-RPC health polls, far too few to produce a meaningful quantile.) See baselines/README.md.",
+  "_excluded_ledger_store": "ledger.store is deliberately absent from spans.names too, for a different reason: it is below the ladder's resolution. The 2026-08-24 capture returned p50/p95/p99 of exactly 0.005/0.0095/0.0099 ms, which is 0.5/0.95/0.99 x the ladder's first edge of 0.01 ms — the signature of every sample landing in the first bucket, so the numbers are interpolation arithmetic on the bucket floor rather than latencies. That is physically plausible: LedgerMaster.cpp:463 wraps an in-memory ledgerHistory_.insert, which completes in single-digit microseconds. While all mass stays under 10 us the reported quantile cannot move materially, so NO absolute bound can gate it — every ledger.store slowing from 2 us to 9 us, 4.5x, leaves the reported value unchanged. Three keys that read as covered but cannot fire are worse than no keys (the same argument that excluded rpc.process), so they were removed rather than left in with a bound that looks derived. Restoring the key needs sub-10us edges on the collector's spanmetrics ladder (for example 0.001ms and 0.005ms) plus the matching entries in HistogramBuckets.h — that is the ladder's branch, not this file. ledger.store presence is still asserted by expected_spans.json and docker/telemetry/integration-test.sh, and its rate is still on the ledger-operations dashboard; only the latency gate drops it.",
   "spans": {
     "_query_template": "histogram_quantile({quantile}, sum by (le) (rate(span_duration_milliseconds_bucket{span_name=\"{name}\"}[{window}])))",
     "_unit": "ms",
@@ -12,7 +13,6 @@
       "tx.apply",
       "ledger.build",
       "ledger.validate",
-      "ledger.store",
       "consensus.ledger_close",
       "consensus.accept"
     ]
diff --git a/docker/telemetry/workload/regression-thresholds.json b/docker/telemetry/workload/regression-thresholds.json
index 7e474569a9..ada4245ff5 100644
--- a/docker/telemetry/workload/regression-thresholds.json
+++ b/docker/telemetry/workload/regression-thresholds.json
@@ -1,26 +1,167 @@
 {
-  "_description": "Per-metric regression thresholds. A metric regresses when current - baseline exceeds BOTH the percentage and absolute bounds (AND, not OR — this tolerates small-value noise). Defaults apply unless a per-metric override exists.",
-  "_bucket_note": "SpanMetrics latency histograms use explicit buckets [0.01,0.05,0.1,0.25,0.5,1,5,10,25,50,100,250,500]ms then [1,2,3,4,5,10,30]s (20 edges; docker/telemetry/otel-collector-config.yaml is the authoritative list). An earlier version of this note claimed 15 edges starting at 1ms and justified the 10ms absolute span bound as \"~2 low-end bucket widths\" — that derivation is void, because the sub-millisecond edges make the low-end bucket width 0.01ms, not 5ms. The 10ms bound is retained on its own merit: it is roughly two bucket widths in the 5-25ms band where most span quantiles actually sit, so it still absorbs single-bucket quantization jitter while catching multi-bucket regressions. Second-scale consensus spans have 2s/3s/4s boundaries, so their quantiles quantize to ~1s widths there. The job_queue running bound is widened similarly — per-ledger apply work scales with TxQ burst load. NOTE: BOTH ladders were re-cut, and a baseline captured before its own ladder changed is an interpolation artefact, not a latency. The native job_queue histograms are microsecond-valued and their floor moved 100us → 1us. The span ladder was re-cut too, on 2026-08-04 in 3860c93db2, moving the floor 1ms → 0.01ms; so any sub-millisecond span quantile captured before that date is equally void — a p95 reading 0.95ms is 0.95 × the old 1ms first edge, not a measurement. An earlier note asserted that the surviving span baselines were unaffected by the ladder work; that is wrong for every span quantile below 1ms. Only the band from 1ms to 1s is safe: those edges are byte-identical across the two ladders. The re-cut also ADDED edges above 1s (2s/3s/4s/10s/30s), so a span whose quantiles land in the second-scale range — consensus.round ~3.9s, consensus.establish ~1.9s, the ledger.acquire tail — is distorted just as much, and any pre-2026-08-04 baseline for it is equally void. Do not read this note as licensing a stale second-scale baseline.",
+  "_description": "Per-metric regression thresholds. A metric regresses when current - baseline exceeds BOTH the percentage and absolute bounds (AND, not OR \u2014 this tolerates small-value noise). Defaults apply unless a per-metric override exists.",
+  "_bucket_note": "SpanMetrics latency histograms use explicit buckets [0.01,0.05,0.1,0.25,0.5,1,5,10,25,50,100,250,500]ms then [1,2,3,4,5,10,30]s (20 edges; docker/telemetry/otel-collector-config.yaml is the authoritative list). Second-scale consensus spans have 2s/3s/4s boundaries, so their quantiles quantize to ~1s widths there \u2014 the ladder is NOT uniformly 2x-or-coarser, which matters for _percentage_bound_note. The native job_queue histograms are microsecond-valued on the ladder [1,2,5,10,25,50,100,250,500,1000,5000,25000,100000,500000]us then [1,5,10,30,60]s (19 edges; include/xrpl/telemetry/HistogramBuckets.h is authoritative). NOTE: BOTH ladders were re-cut, and a baseline captured before its own ladder changed is an interpolation artefact, not a latency. The job_queue floor moved 100us \u2192 1us. The span ladder was re-cut on 2026-08-04 in 3860c93db2, moving the floor 1ms \u2192 0.01ms; so any sub-millisecond span quantile captured before that date is equally void \u2014 a p95 reading 0.95ms is 0.95 \u00d7 the old 1ms first edge, not a measurement. An earlier note asserted that the surviving span baselines were unaffected by the ladder work; that is wrong for every span quantile below 1ms. Only the band from 1ms to 1s is safe: those edges are byte-identical across the two ladders. The re-cut also ADDED edges above 1s (2s/3s/4s/10s/30s), so a span whose quantiles land in the second-scale range \u2014 consensus.round ~3.9s, consensus.establish ~1.9s, the ledger.acquire tail \u2014 is distorted just as much, and any pre-2026-08-04 baseline for it is equally void. Do not read this note as licensing a stale second-scale baseline.",
+  "_absolute_bound_derivation": "HOW EVERY max_abs_increase_* NUMBER BELOW WAS OBTAINED. Rule: locate the baseline value in the half-open bucket (lo, hi] of its own ladder, take hi_next = the next edge above hi, and set the bound to (hi_next - baseline). The trip point is therefore exactly hi_next: the gate fires only when the reported value EXCEEDS the top of the bucket above the baseline's own bucket. WHY THAT AND NOT A MULTIPLE OF THE BUCKET WIDTH: histogram_quantile returns a value interpolated inside whichever bucket the true quantile falls in, so a reading taken while the true quantile sits anywhere in the baseline's bucket OR anywhere in the one immediately above is at most hi_next and cannot fire. Firing requires the true quantile to have moved at least two buckets up. A multiple of the ENCLOSING width cannot deliver that, because once the quantile crosses hi the interpolation happens across the NEXT bucket, which on this ladder is up to 8x wider \u2014 (0.5,1] has width 0.5 and (1,5] has width 4 \u2014 so the reading's excursion is not bounded by any multiple of the enclosing width. Worked example: span.tx.process.p99 has baseline 0.9945ms in bucket (0.5, 1], hi_next = 5, so its bound is 4.0055ms and the gate fires only above 5ms. Bounds are stored as exact doubles rather than rounded figures so that rounding cannot break the guarantee and so check_regression_bounds.py can assert each one against the ladder to within a 1e-12 relative tolerance -- tight enough that a bound rounded for readability, such as 4.0055 for 4.005485184848892, is rejected; _derivation_table below shows the arithmetic for each one. Measured over the committed baseline this rule yields a detection floor of 2.02x to 9.43x of baseline, per key. WHAT THIS REPLACED, IN TWO GENERATIONS: (1) a single flat pair of bounds (10ms for span p50/p95, 15ms for span p99, 20000us for job_queue p95) justified as 'roughly two bucket widths in the 5-25ms band where most span quantiles actually sit'. The 2026-08-24 capture falsifies that premise \u2014 18 of the 28 quantiles gated at that time sat below 1ms \u2014 so the absolute bound sat 1.15x to 2000x above the metric it guarded and, because the rule is an AND, the percentage bound could never carry a regression on its own; a 10x regression injected into each key in turn was caught on only 5 of 28, and a 100x regression injected into span.ledger.store.p95 produced 0 regressions and exit 0. (2) a first correction to 2 \u00d7 the ENCLOSING bucket width, which caught 10x on 28 of 28 but placed the trip point INSIDE the adjacent bucket -- and so left a single-crossing false positive reachable -- on 21 of the 25 keys gated today, 4 of them tripping on a tail-mass shift under 1.5% of samples. That is the assumption this rule removes. RE-DERIVE THESE NUMBERS whenever baseline-timings.json is refreshed or either ladder changes: a refreshed baseline can land in a different bucket, which changes hi_next. .github/scripts/telemetry/check_regression_bounds.py enforces the rule in CI so a stale bound cannot survive a baseline refresh. LIMITATION \u2014 WHICH KEYS ARE ONLY WEAKLY GUARDED: the guarantee costs sensitivity wherever the ladder is coarse, and the detection floor is hi_next/baseline, so a baseline sitting just above an edge is guarded loosely. span.ledger.build.p50 (baseline 1.0612ms, fires at 10ms, 9.42x) and span.ledger.validate.p99 (1.0600ms, 9.43x) are NOT meaningfully guarded: ledger.build p50 fires only at 10ms, so it could get up to 9.4x slower -- 1.06ms to just under 10ms -- and still pass. span.tx.process.p95 (6.91x), span.tx.apply.p50 (6.32x), span.rpc.ws_message.p95 (5.92x), job.acceptLedger.running.p95 (5.74x), span.consensus.accept.p50 (5.74x), span.consensus.ledger_close.p99 (5.37x), span.rpc.ws_message.p99 (5.06x) and span.tx.process.p99 (5.03x) are weak. All ten are limited by two 5x-wide ladder steps, 1ms\u21925ms and 5000us\u219225000us. The fix is a 2ms edge (and ideally 3ms) in the collector's spanmetrics ladder plus the matching edges in kMillisecondBuckets, and a 10000us edge in kMicrosecondBuckets \u2014 that work belongs to the branch that owns the ladders, not here. Until then do not read these keys as guarded. span.ledger.store is absent from the overrides below because it was removed from the gated surface entirely: its quantiles were the ladder floor times the quantile, so no bound could gate it. See _excluded_ledger_store in regression-metrics.json.",
+  "_percentage_bound_note": "For every key gated today the absolute bound is the binding half of the AND and the percentage bound never decides the outcome: measured, (bound / baseline) ranges from 102% (span.tx.apply.p99) to 843% (span.ledger.validate.p99), all above the 50% and 5% percentage bounds configured here, and the minimum trip multiple of all 25 keys is set by the absolute bound. THIS IS NOT A GENERAL GUARANTEE, and an earlier version of this note wrongly claimed it was, on the false premise that 'every step of both ladders is at least a factor of 2'. The span ladder breaks that three times at the top: 2s->3s is 1.5x, 3s->4s is 1.33x, 4s->5s is 1.25x, so second-scale consensus quantiles quantize to ~1s widths there. Because the bound is (hi_next - baseline), a baseline between about 2667ms and 3000ms, or between about 3334ms and 4000ms, gets an absolute bound worth less than 50% of itself and the PERCENTAGE bound becomes the operative one -- at which point the metric fires on a 50% move that is smaller than one bucket width, and the single-crossing guarantee in _absolute_bound_derivation is lost. That band is not hypothetical: the collector config names consensus.round (~3.9s) as a reason those edges exist, and 3900ms sits in the second sub-band with an absolute bound of 5000 - 3900 = 1100, only 28.2% of baseline. Whoever gates a key whose baseline lands in either sub-band MUST lower its max_pct_increase below (bound / baseline) for that key, or state explicitly that the metric is percentage-gated and the bucket guarantee does not hold for it. check_regression_bounds.py enforces this as rule D so the trap cannot be walked into silently. The percentage entries are required and still meaningful regardless: compare_to_baseline.py treats a missing max_pct_increase as 'no threshold configured' and would stop gating the metric entirely; they record the intended relative tolerance (consensus spans 5%, everything else 50%); and they are the operative bound on the defaults path (see _defaults_note).",
+  "_defaults_note": "A MISSING OVERRIDE IS DETECTED BY CI, NOT BY THESE DEFAULTS. .github/scripts/telemetry/check_regression_bounds.py fails the build at lint time, naming the key and the exact value its bound should have, before the workload ever runs. That is the mechanism; the defaults below are only a runtime backstop for the case where that check is bypassed. The defaults carry the FLOOR of each ladder as their absolute bound \u2014 0.01ms for spans, 1us for job_queue \u2014 deliberately too small to bind for any real metric, which leaves max_pct_increase (50%) as the operative bound on this path. Measured: a metric with no override and a baseline of 3900ms passes at +49% and fires at +51%; a job metric with a baseline of 5000us behaves the same. The backstop is honestly imperfect and the earlier version of this note oversold it. At 50% relative it CAN false-fire: a metric whose baseline is 1.06ms inside the 4ms-wide (1,5] bucket fires on a single-bucket-width move (measured: 1.06 \u2192 5.06ms, +377%, regressed). An earlier note called that 'the intended signal that the override is missing', which was wrong \u2014 CI prints REGRESSION and a reader cannot tell it from a real one, and rejecting a tighter alternative for exactly that cries-wolf risk while shipping it here would be inconsistent. The check is what makes the signal legible. The backstop is kept only because a metric silently not gated at all is the worse of the two failures.",
+  "_derivation_table": {
+    "_format": "override key: <quantile> <baseline> in <bucket> -> hi_next - baseline = <bound>",
+    "job.acceptLedger.queued": "p95 91.10576923076925 in (50,100] -> hi_next 250 - baseline = 158.89423076923075",
+    "job.acceptLedger.running": "p95 17428.571428571428 in (5000,25000] -> hi_next 100000 - baseline = 82571.42857142858",
+    "job.transaction.queued": "p95 476.6129032258061 in (250,500] -> hi_next 1000 - baseline = 523.3870967741939",
+    "job.transaction.running": "p95 427.1084337349388 in (250,500] -> hi_next 1000 - baseline = 572.8915662650612",
+    "span.consensus.accept": "p50 1.7435897435897438 in (1,5] -> hi_next 10 - baseline = 8.256410256410255 | p95 8.9296875 in (5,10] -> hi_next 25 - baseline = 16.0703125 | p99 15.150000000000082 in (10,25] -> hi_next 50 - baseline = 34.849999999999916",
+    "span.consensus.ledger_close": "p50 0.15142857142857144 in (0.1,0.25] -> hi_next 0.5 - baseline = 0.34857142857142853 | p95 0.49328358208955236 in (0.25,0.5] -> hi_next 1 - baseline = 0.5067164179104476 | p99 0.9314285714285726 in (0.5,1] -> hi_next 5 - baseline = 4.068571428571428",
+    "span.ledger.build": "p50 1.0612244897959187 in (1,5] -> hi_next 10 - baseline = 8.938775510204081 | p95 4.679591836734694 in (1,5] -> hi_next 10 - baseline = 5.320408163265306 | p99 5.075000000000041 in (5,10] -> hi_next 25 - baseline = 19.924999999999958",
+    "span.ledger.validate": "p50 0.07787769784172663 in (0.05,0.1] -> hi_next 0.25 - baseline = 0.17212230215827337 | p95 0.2404310344827586 in (0.1,0.25] -> hi_next 0.5 - baseline = 0.2595689655172414 | p99 1.060000000000033 in (1,5] -> hi_next 10 - baseline = 8.939999999999968",
+    "span.rpc.ws_message": "p50 0.15990942879094971 in (0.1,0.25] -> hi_next 0.5 - baseline = 0.3400905712090503 | p95 0.8443074003795064 in (0.5,1] -> hi_next 5 - baseline = 4.155692599620494 | p99 0.9878368121442125 in (0.5,1] -> hi_next 5 - baseline = 4.012163187855787",
+    "span.tx.apply": "p50 0.791666666666667 in (0.5,1] -> hi_next 5 - baseline = 4.208333333333333 | p95 4.627368421052632 in (1,5] -> hi_next 10 - baseline = 5.372631578947368 | p99 4.959157894736842 in (1,5] -> hi_next 10 - baseline = 5.040842105263158",
+    "span.tx.process": "p50 0.34281067382135394 in (0.25,0.5] -> hi_next 1 - baseline = 0.6571893261786461 | p95 0.7239865112994349 in (0.5,1] -> hi_next 5 - baseline = 4.276013488700565 | p99 0.9945148151511076 in (0.5,1] -> hi_next 5 - baseline = 4.005485184848892"
+  },
   "defaults": {
     "span": {
-      "p50": { "max_pct_increase": 50.0, "max_abs_increase_ms": 10.0 },
-      "p95": { "max_pct_increase": 50.0, "max_abs_increase_ms": 10.0 },
-      "p99": { "max_pct_increase": 50.0, "max_abs_increase_ms": 15.0 }
+      "p50": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_ms": 0.01
+      },
+      "p95": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_ms": 0.01
+      },
+      "p99": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_ms": 0.01
+      }
     },
     "job_queue": {
-      "p95": { "max_pct_increase": 50.0, "max_abs_increase_us": 20000.0 }
+      "p95": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_us": 1.0
+      }
     }
   },
   "overrides": {
-    "span.consensus.ledger_close": {
-      "p50": { "max_pct_increase": 5.0, "max_abs_increase_ms": 200.0 },
-      "p95": { "max_pct_increase": 5.0, "max_abs_increase_ms": 500.0 },
-      "p99": { "max_pct_increase": 5.0, "max_abs_increase_ms": 1000.0 }
+    "job.acceptLedger.queued": {
+      "p95": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_us": 158.89423076923075
+      }
+    },
+    "job.acceptLedger.running": {
+      "p95": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_us": 82571.42857142858
+      }
+    },
+    "job.transaction.queued": {
+      "p95": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_us": 523.3870967741939
+      }
+    },
+    "job.transaction.running": {
+      "p95": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_us": 572.8915662650612
+      }
     },
     "span.consensus.accept": {
-      "p50": { "max_pct_increase": 5.0, "max_abs_increase_ms": 200.0 },
-      "p95": { "max_pct_increase": 5.0, "max_abs_increase_ms": 500.0 },
-      "p99": { "max_pct_increase": 5.0, "max_abs_increase_ms": 1000.0 }
+      "p50": {
+        "max_pct_increase": 5.0,
+        "max_abs_increase_ms": 8.256410256410255
+      },
+      "p95": {
+        "max_pct_increase": 5.0,
+        "max_abs_increase_ms": 16.0703125
+      },
+      "p99": {
+        "max_pct_increase": 5.0,
+        "max_abs_increase_ms": 34.849999999999916
+      }
+    },
+    "span.consensus.ledger_close": {
+      "p50": {
+        "max_pct_increase": 5.0,
+        "max_abs_increase_ms": 0.34857142857142853
+      },
+      "p95": {
+        "max_pct_increase": 5.0,
+        "max_abs_increase_ms": 0.5067164179104476
+      },
+      "p99": {
+        "max_pct_increase": 5.0,
+        "max_abs_increase_ms": 4.068571428571428
+      }
+    },
+    "span.ledger.build": {
+      "p50": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_ms": 8.938775510204081
+      },
+      "p95": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_ms": 5.320408163265306
+      },
+      "p99": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_ms": 19.924999999999958
+      }
+    },
+    "span.ledger.validate": {
+      "p50": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_ms": 0.17212230215827337
+      },
+      "p95": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_ms": 0.2595689655172414
+      },
+      "p99": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_ms": 8.939999999999968
+      }
+    },
+    "span.rpc.ws_message": {
+      "p50": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_ms": 0.3400905712090503
+      },
+      "p95": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_ms": 4.155692599620494
+      },
+      "p99": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_ms": 4.012163187855787
+      }
+    },
+    "span.tx.apply": {
+      "p50": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_ms": 4.208333333333333
+      },
+      "p95": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_ms": 5.372631578947368
+      },
+      "p99": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_ms": 5.040842105263158
+      }
+    },
+    "span.tx.process": {
+      "p50": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_ms": 0.6571893261786461
+      },
+      "p95": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_ms": 4.276013488700565
+      },
+      "p99": {
+        "max_pct_increase": 50.0,
+        "max_abs_increase_ms": 4.005485184848892
+      }
     }
   }
 }
diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md
index 95a1369593..e51afb528a 100644
--- a/docs/telemetry-runbook.md
+++ b/docs/telemetry-runbook.md
@@ -3703,10 +3703,48 @@ Key properties:
 
 - **A metric regresses only when it exceeds BOTH the percentage and the absolute
   bound.** The `AND` is deliberate: SpanMetrics latency histograms use explicit
-  buckets, so a quantile sitting near a low bucket boundary can jump a whole
-  bucket (1 ms to 5 ms) with no real change. Bounds live in
-  `regression-thresholds.json` — `defaults` per category and quantile, with
-  per-metric `overrides` (e.g. `span.consensus.ledger_close` is held to 5%).
+  buckets, so a quantile sitting near a bucket boundary can jump a whole bucket
+  with no real change. Bounds live in `regression-thresholds.json` — `defaults`
+  per category and quantile, plus a per-metric `override` for every gated key.
+- **The absolute bound is derived per metric, as `hi_next − baseline`.** Locate
+  the baseline in the half-open bucket `(lo, hi]` of its ladder and take
+  `hi_next` as the next edge above `hi`; the trip point is then exactly
+  `hi_next`, so the gate fires only once the reading clears the bucket _above_
+  the baseline's own. That is what makes a single bucket crossing unable to turn
+  CI red: `histogram_quantile` interpolates inside whichever bucket the quantile
+  falls in, so any reading produced while the quantile is at most one bucket
+  above the baseline's is at most `hi_next`. A multiple of the _enclosing_
+  bucket width cannot deliver that, because after crossing `hi` the
+  interpolation happens across the next bucket, which here is up to 8x wider
+  (`(0.5, 1]` is 0.5 ms, `(1, 5]` is 4 ms). Derivation, both ladders and a
+  per-key table live in `regression-thresholds.json` under
+  `_absolute_bound_derivation` and `_derivation_table`. **Refreshing
+  `baseline-timings.json` obliges you to re-derive these bounds** — a value that
+  moves into a different bucket gets a different `hi_next` — and
+  `.github/scripts/telemetry/check_regression_bounds.py` fails CI if you do not.
+- **A single flat bound cannot work here.** The gated quantiles span 0.078 ms to
+  17 ms, so one figure is inert at the bottom of that range and trigger-happy at
+  the top. The flat 10/15 ms span bound it replaced sat 1.15x to 2000x above the
+  metric it guarded, and a 10x regression injected into each key in turn was
+  caught on only 5 of 28.
+- **The detection floor is `hi_next / baseline`, so some keys are only weakly
+  guarded.** It ranges 2.02x to 9.43x over the current baseline;
+  `span.ledger.build.p50` and `span.ledger.validate.p99` are effectively
+  not guarded at 9.4x. `baselines/README.md` lists all ten weak keys and the
+  ladder edges that would fix them.
+- **For every currently gated metric the absolute bound decides; the percentage
+  bound does not.** Measured, the bound is 102%-843% of its own baseline, above
+  both configured percentage bounds. This is _not_ a general property: the span
+  ladder's top steps are only 1.25x-1.5x apart, so a baseline between about
+  2667-3000 ms or 3334-4000 ms gets an absolute bound worth under 50% of itself
+  and the percentage bound takes over — `consensus.round` at ~3.9 s lands
+  exactly there. `check_regression_bounds.py` rule D fails the build rather than
+  letting that happen silently. The percentage entries are still required (a
+  missing one turns the metric into "no threshold configured" and stops it
+  gating) and they are the operative bound on the `defaults` path.
+- **`span.ledger.store` is not gated**, because its quantiles are the ladder's
+  first edge times the quantile — every sample lands under 10 us, so no bound
+  can move. See `baselines/README.md`.
 - **A metric with no configured threshold is captured but never gates.** It is
   reported with a note instead. Today only `span.*` and `job.*` keys have
   thresholds; `rpc.*` is not produced and would not gate if it were (see

From 9b1b4dfdf095b70eda9ee6471556bd40c758c22c Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Tue, 25 Aug 2026 15:04:34 +0100
Subject: [PATCH 30/38] fix(telemetry): drop the unused <string_view> include
 from consensus span names

Splitting the consensus span labels into their own header moved both
constexpr std::string_view helpers out of ConsensusSpanNames.h, but left
the include they needed behind. The header names string_view nowhere now,
and SpanNames.h already provides the type for the conversion operator.

clang-tidy misc-include-cleaner reports this as an error under
-warnings-as-errors, which fails the clang-tidy job for the whole chain.
---
 include/xrpl/consensus/ConsensusSpanNames.h | 2 --
 1 file changed, 2 deletions(-)

diff --git a/include/xrpl/consensus/ConsensusSpanNames.h b/include/xrpl/consensus/ConsensusSpanNames.h
index 4945db30d0..fc56adcd44 100644
--- a/include/xrpl/consensus/ConsensusSpanNames.h
+++ b/include/xrpl/consensus/ConsensusSpanNames.h
@@ -88,8 +88,6 @@
 
 #include <xrpl/telemetry/SpanNames.h>
 
-#include <string_view>
-
 namespace xrpl::telemetry::consensus::span {
 
 // ===== Span name segments ====================================================

From 8547adb5c135d5dcdc5ac908cb97d3c1ef03cd81 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Tue, 25 Aug 2026 15:05:36 +0100
Subject: [PATCH 31/38] test(telemetry): close the 20-metric harness coverage
 gap

Dashboards and alert rules reference 186 metrics; the harness asserted 57.
Excluding the 107 per-category overlay-traffic expansions, the meaningful
gap was 20 names. This closes it under the contract file's own doctrine:
assert only what the workload guarantees, and record the rest with a
precise reason.

Asserted 18, taking the metric checks from 61 to 79 and the whole metric
phase from 66 to 84. No pre-existing check name or position changes.

statsd_gauges gains the nine state_accounting_* siblings of the one member
already asserted, plus the two NodeFamily full-below-cache gauges and
overlay_peer_disconnects. All twelve rest on one mechanism the group
description now spells out: on the OTel path a beast gauge is an
Int64ObservableGauge, every instance self-registers in its constructor,
onCollectionReady arms all of them unconditionally, and the armed callback
Observes on every export cycle whether or not set was ever called, so the
series exist at 0. The state_accounting family is set in one unconditional
block in NetworkOPsImp::collectMetrics, and full_transitions is the input
to the NodeStateFlapping alert rule, so the alert's own signal had been
going unverified.

A new job_queue_per_type_gauges group asserts the six per-job-type gauges
that a panel or a rule names literally, jobq_manifest_waiting among them
as the ManifestJobQueueConvoy rule's input. The description records why
those six and not all 105: the guarantee is identical for every
non-special job type, so the discriminator is consumer coverage, and the
remaining names are only reached through topk queries over the family that
do not depend on any single type being present.

Recorded four more in not_asserted rather than asserting them.
pathfind_fast_milliseconds is unreachable for this workload, not merely
rare: reportFast fires only from the doCreate fast pass, which is guarded
by !hasCompletion(), and both ripple_path_find entry points construct the
request with a completion function. Only the path_find subscription
reaches it, and the generator does not use it.
pathfind_full_milliseconds is reachable but only one ledger close after
the request, through PathRequestManager::updateAll, and nothing in the
harness arranges or checks that, so the guarantee is probabilistic.
warn_total and drop_total are resource-manager meters gated on a consumer
crossing the warn or drop threshold; their rpc-pathfinding panels are
correct and render empty only because the condition has not occurred,
which is worth stating because both were briefly mis-read as phantoms.

Runbook check counts updated to match.
---
 .../telemetry/workload/expected_metrics.json  | 31 +++++++++++++++++--
 docs/telemetry-runbook.md                     |  2 +-
 2 files changed, 30 insertions(+), 3 deletions(-)

diff --git a/docker/telemetry/workload/expected_metrics.json b/docker/telemetry/workload/expected_metrics.json
index d07da27b4d..81ca83d848 100644
--- a/docker/telemetry/workload/expected_metrics.json
+++ b/docker/telemetry/workload/expected_metrics.json
@@ -32,14 +32,26 @@
     "_dimension_labels_note": "Bare label names as configured in otel-collector-config.yaml spanmetrics dimensions. Informational only (not asserted by the validator)."
   },
   "statsd_gauges": {
-    "description": "beast::insight gauges exported via OTLP/HTTP to the collector (server=otel).",
+    "description": "beast::insight gauges exported via OTLP/HTTP to the collector (server=otel). What guarantees these is the export mechanism, not the workload. On the OTel path a beast gauge is an Int64ObservableGauge (OTelCollector.cpp:703); every OTelGaugeImpl registers itself with the collector in its own constructor (:690); onCollectionReady() arms every registered gauge with no per-metric condition (:941-979); and the armed callback Observes currentValue() on each export cycle regardless of the value and regardless of whether set() was ever called (:719-727). Series therefore exist from the first export, at 0 if nothing has happened. The 'gauges only mark dirty on value changes' caveat in _poll_series_count is a property of the StatsD backend and does not apply here. The consequence is that a beast gauge whose object is constructed on an unconditional startup path is as safe to assert as an observable gauge from MetricsRegistry — which is why the state_accounting family below is asserted whole rather than one member deep. The ten state_accounting_* names are one family with one guarantee. NetworkOPsImp::Stats creates all ten in the NetworkOPsImp constructor (NetworkOPs.cpp:1069-1080; makeGauge(\"State_Accounting\", \"Full_duration\") joins on '.' at Collector.h:151-155 and formatName lowercases it, giving state_accounting_full_duration), and NetworkOPsImp::collectMetrics() set()s all ten unconditionally in a single block (:5204-5231). No state has to be entered for its pair to exist: a node that never disconnects still publishes state_accounting_disconnected_duration and _transitions at 0. All ten appear on the node-health dashboard and its grafanacloud copy, and state_accounting_full_transitions is also the input to the NodeStateFlapping alert rule (provisioning/alerting/rules.yaml, uid xrpld-node-state-flapping). Asserting only full_duration left the other nine unchecked, the alert's own input among them, even though a regression could not plausibly drop one sibling without dropping full_duration too. node_family_full_below_cache_hit_rate and _size are the TaggedCache Stats pair (TaggedCache.h:282-283), created in the cache constructor (TaggedCache.ipp:63-66) and published by collectMetrics(), which reports a hit_rate of 0 rather than skipping the gauge when hits+misses is 0 (TaggedCache.ipp:751-765). The instance is NodeFamily's full-below cache, constructed with the real collector in NodeFamily's constructor (NodeFamily.cpp:28-35), which every node builds. Both are on node-health. overlay_peer_disconnects is the first member of OverlayImpl::Stats (OverlayImpl.h:603) and OverlayImpl::collectMetrics() assigns it getPeerDisconnect() unconditionally (:646), in the same hook that publishes the overlay_traffic group's gauges. It is on network-traffic and log-derived-insights. It is listed here rather than in overlay_traffic because that group's contract is an explicit subset of the per-category traffic family, and a disconnect count is not one of those categories.",
     "metrics": [
       "ledgermaster_validated_ledger_age",
       "ledgermaster_published_ledger_age",
       "state_accounting_full_duration",
       "peer_finder_active_inbound_peers",
       "peer_finder_active_outbound_peers",
-      "jobq_job_count"
+      "jobq_job_count",
+      "state_accounting_connected_duration",
+      "state_accounting_connected_transitions",
+      "state_accounting_disconnected_duration",
+      "state_accounting_disconnected_transitions",
+      "state_accounting_full_transitions",
+      "state_accounting_syncing_duration",
+      "state_accounting_syncing_transitions",
+      "state_accounting_tracking_duration",
+      "state_accounting_tracking_transitions",
+      "node_family_full_below_cache_hit_rate",
+      "node_family_full_below_cache_size",
+      "overlay_peer_disconnects"
     ]
   },
   "statsd_counters": {
@@ -100,6 +112,17 @@
       "job_running_us_sum"
     ]
   },
+  "job_queue_per_type_gauges": {
+    "description": "Per-job-type queue-depth gauges from the beast::insight \"jobq\" group, for the six job types the ledger-sync diagnostics single out. JobTypeData's constructor creates waiting/running/deferred for every non-special job type (JobTypeData.h:96-103), the JobQueue constructor eagerly constructs a JobTypeData for every type in JobTypes (JobQueue.cpp:45-53), and JobQueue::collect() assigns all three from one snapshot for every type on every hook cycle, with no per-type condition (JobQueue.cpp:104-106). The collector is the \"jobq\" group, so GroupImp::makeName() prefixes \"jobq.\" and formatName lowercases the job type: JtTxnData's \"fetchTxnData\" + \"_running\" exports as jobq_fetchtxndata_running (JobTypeData.h:29-37). Existence rests on the same ObservableGauge mechanism documented on statsd_gauges, so these series exist at 0 on an idle node. All six were present in the CI validation run's own emitted-metric list, which is the strongest evidence behind any entry in this file. All six job types are non-special: makeFetchPack (limit 1), manifest (maxLimit), ledgerRequest (3), ledgerData (3), updatePaths (1) and fetchTxnData (5) all declare a non-zero limit in JobTypes.h, so none is skipped by the special() guard in JobTypeData's constructor. The 11 special types (limit 0) get no gauges at all, which is why no jobq_pathfind_* or jobq_peercommand_* name can be asserted. Why these six and not all 105. The guarantee is identical for every one of the 35 non-special types x 3 states, so the discriminator here is consumer coverage rather than emission: these six are the names a dashboard or an alert queries literally, so losing one blanks a specific panel or silently disarms a rule. jobq_manifest_waiting is the input to the ManifestJobQueueConvoy alert rule (provisioning/alerting/rules.yaml, uid xrpld-manifest-job-convoy); the other five are per-type saturation panels on node-health and its grafanacloud copy. The remaining names are only ever reached through the __name__=~\"jobq_.*_waiting\" / \"jobq_.*_deferred\" topk queries on ledger-data-sync, which keep working as long as the family exists and do not depend on any single job type being present. Asserting all 105 would add 99 checks that share one failure mode with these six while costing poll bandwidth in the shared 45 s window, so it would buy no signal.",
+    "metrics": [
+      "jobq_fetchtxndata_running",
+      "jobq_ledgerdata_running",
+      "jobq_ledgerrequest_running",
+      "jobq_makefetchpack_running",
+      "jobq_manifest_waiting",
+      "jobq_updatepaths_running"
+    ]
+  },
   "rpc_in_flight": {
     "description": "In-flight RPC gauge via the XRPL_METRIC_UPDOWN_ADD call-site macro (PerfLogImp.cpp, +1 rpcStart / -1 rpcEnd). UpDownCounter: no _total suffix.",
     "metrics": ["rpc_in_flight_requests"]
@@ -187,6 +210,10 @@
       "getobject_charge": "GetObjectMetricNames.h:105, PeerImp.cpp:2931. Same gate.",
       "rpc_size_bytes": "ServerHandler.cpp:191, group('rpc')->makeEvent('size', Unit::Bytes). The OTLP Prometheus exporter derives the metric-name suffix from the declared unit, so a byte unit yields rpc_size_bytes. The Unit::Bytes declaration itself landed earlier, in 76c9051203; what 24094e427b changed was the exporter finally consuming it, replacing a hardcoded CreateDoubleHistogram(name, 'Duration in ms', 'ms') with otelUnitDescription(unit)/otelUnitCode(unit), and that is what renamed the series off rpc_size_milliseconds and the millisecond bucket ladder. Neither name was ever recorded here, so the harness could confirm neither the rename nor a regression back onto that ladder. Notified from ServerHandler::processRequest:1133, the HTTP JSON-RPC path — it computes an HTTP status and appends a trailing newline — and the load generators are WebSocket-only, the same gate regression-metrics.json:4 records for rpc.process, so only the harness's handful of HTTP health polls reach it. Real coverage needs an HTTP JSON-RPC phase in rpc_load_generator.py; that is a workload change rather than a harness correction, and is deliberately out of scope here.",
       "rpc_time_milliseconds": "ServerHandler.cpp:192, group('rpc')->makeEvent('time') with the default millisecond unit. Notified from ServerHandler::processRequest:1129, the same HTTP JSON-RPC call site as rpc_size_bytes and behind the same WebSocket-only gate.",
+      "pathfind_fast_milliseconds": "PathRequestManager.h:35, makeEvent('pathfind_fast') with the default millisecond unit, so the exported form is the pathfind_fast_milliseconds_bucket/_count/_sum triple and there is no bare series — the same convention io_latency and rpc_method_us follow, and rpc-pathfinding queries the _bucket. Notified from PathRequestManager::reportFast (:81-84), whose only caller is PathRequest.cpp:852, inside the 'if (fast && quickReply_ == {})' branch of PathRequest::doUpdate. The only doUpdate call that passes fast=true is in PathRequest::doCreate (PathRequest.cpp:259), and it is guarded by '!hasCompletion()'. That guard is what makes this unreachable for the harness: ripple_path_find has two entry points and both construct the PathRequest with a completion function, so hasCompletion() (:161-164) is true and the fast pass is skipped. doRipplePathFind with no ledger specified goes to makeLegacyPathRequest, which passes the coroutine-post lambda (RipplePathFind.cpp:140-160); with a ledger specified it goes to doLegacyPathRequest, which passes an empty-body but non-null lambda (PathRequestManager.cpp:317). Only the path_find streaming subscription reaches reportFast: makePathRequest builds the request from a subscriber with no completion (PathRequestManager.cpp:261), so hasCompletion() is false. The load generator does not use path_find and says so at rpc_load_generator.py:72-73, so the 3% ripple_path_find weight cannot produce this metric at all — not rarely, never. That matches the Grafana Cloud reading of zero series in 180 days on devnet nodes serving live traffic. Covering it needs a path_find subscription phase in the generator, which is a workload change and out of scope here.",
+      "pathfind_full_milliseconds": "PathRequestManager.h:36, makeEvent('pathfind_full'); same histogram naming as pathfind_fast_milliseconds above. Notified from reportFull (:87-90) via PathRequest.cpp:857, the 'else if (!fast && fullReply_ == {})' branch. Unlike the fast event this one is reachable from the harness workload, but not on the request's own turn, which is why it is recorded here rather than asserted. The generator sends no ledger/ledger_index/ledger_hash (rpc_load_generator.py:343-350) and CI is not standalone, so doRipplePathFind takes the makeLegacyPathRequest branch, which only enqueues the request; the fast=false update happens later, when the next ledger close drives PathRequestManager::updateAll into its one-shot branch (PathRequestManager.cpp:160-166). Emission is therefore one ledger close behind the request and conditional on the request still being alive and still passing isValid() at that point (PathRequest.cpp:772-773) — none of which the harness arranges or checks, and a phase that ends before the next close records nothing. The synchronous route that would make this deterministic, doLegacyPathRequest calling doUpdate(cache, false) directly (PathRequestManager.cpp:321), needs an explicit ledger parameter the generator does not send. A probabilistic path is exactly what the doctrine at the top of this file excludes. Grafana Cloud shows zero series in 180 days on the devnet nodes, consistent with those nodes receiving no pathfinding RPC rather than with a broken exporter.",
+      "warn_total": "include/xrpl/resource/detail/Logic.h:41, makeMeter('warn'). makeMeter maps to CreateUInt64Counter (OTelCollector.cpp:878-881 -> :773-777), so the Prometheus exporter appends _total; the meter is created on the bare collector with no group, hence the unprefixed name. Incremented only at Logic.h:481, inside the 'if (notify)' branch reached when a consumer's balance crosses kWarningThreshold. A cooperating 5-node cluster plus a rate-limited load generator never charges a consumer that far, and Grafana Cloud confirms zero series in 180 days. Recorded explicitly because this was briefly mis-diagnosed as a phantom metric: the rpc-pathfinding panel that queries it is correct, and renders empty only because the condition has not occurred.",
+      "drop_total": "include/xrpl/resource/detail/Logic.h:42, makeMeter('drop'); same CreateUInt64Counter mapping and same _total suffix as warn_total. Incremented only at Logic.h:505, when a consumer's balance is at or above kDropThreshold and the connection is dropped. Grafana Cloud shows 2 live series, so unlike warn_total this one does fire in the wild — but only on a genuinely abusive consumer, which the harness deliberately does not create, so it is condition-gated all the same. Its rpc-pathfinding panel is likewise correct rather than phantom.",
       "jobq_*_milliseconds, jobq_*_q_milliseconds": "This key is a pattern rather than a literal metric name — unlike every other entry in this map it stands for a whole family, one pair per job type. Created per job type in JobTypeData.h:97-98 from info.name() and info.name() + kSuffixQueued ('_q'), so the exported names are jobq_<jobtype>_milliseconds and jobq_<jobtype>_q_milliseconds with the job type lowercased by formatName. Which job types appear depends on which jobs a run happens to schedule, so no individual name is guaranteed. They are also rounded up to a whole millisecond at source (Event.h:47-51 applies ceil to a millisecond value type), which is why 6e2b2da772 moved the ledger-data-sync q-wait panels off jobq_<jobtype>_q_milliseconds_bucket onto job_queued_us_bucket — they are poor assertion targets regardless."
     }
   },
diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md
index e51afb528a..947588e5e7 100644
--- a/docs/telemetry-runbook.md
+++ b/docs/telemetry-runbook.md
@@ -3624,7 +3624,7 @@ stands today.
 | Category   | Checks                                                                                                                                                                               | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
 | ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
 | Spans      | Every **required** entry in `expected_spans.json` — 41 span types at the time of writing: 26 required, 15 marked `"optional": true`                                                  | Span name found in Tempo carrying its `required_attributes`, plus the declared parent-child relationships. An `"optional": true` entry that does not fire is recorded as a skip, not a failure — it needs traffic the harness may not generate (HTTP/JSON-RPC client, gRPC client, missing-ledger fetch, mode transitions).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
-| Metrics    | Every entry in every asserted category of `expected_metrics.json` — 66 checks across 24 asserting categories at the time of writing: 61 metric names plus 5 `required_labels` checks | SpanMetrics, `beast::insight` gauges/counters exported over OTLP, and the `MetricsRegistry` OTLP metrics. Each must have > 0 Prometheus series; none are optional. A category may also declare `required_labels`, and each label there becomes one additional check that at least one of that category's series carries it with a non-empty value (matched as `<label>!=""`, because Prometheus cannot distinguish an absent label from an empty one). Those labels were declared but never actually read until the check was generalised, so they were documented as required while going unverified; `spanmetrics` contributes 4 and `job_queue` 1. The separate `not_asserted` group lists metrics deliberately left out of the gate because they are workload-gated or defect-gated; it has neither a `metrics` nor a `required_labels` key, so the validator skips it entirely. |
+| Metrics    | Every entry in every asserted category of `expected_metrics.json` — 84 checks across 25 asserting categories at the time of writing: 79 metric names plus 5 `required_labels` checks | SpanMetrics, `beast::insight` gauges/counters exported over OTLP, and the `MetricsRegistry` OTLP metrics. Each must have > 0 Prometheus series; none are optional. A category may also declare `required_labels`, and each label there becomes one additional check that at least one of that category's series carries it with a non-empty value (matched as `<label>!=""`, because Prometheus cannot distinguish an absent label from an empty one). Those labels were declared but never actually read until the check was generalised, so they were documented as required while going unverified; `spanmetrics` contributes 4 and `job_queue` 1. The separate `not_asserted` group lists metrics deliberately left out of the gate because they are workload-gated or defect-gated; it has neither a `metrics` nor a `required_labels` key, so the validator skips it entirely. |
 | Logs       | 2 checks                                                                                                                                                                             | `trace_id`/`span_id` present in Loki, and a Tempo trace id resolves in Loki. Skipped in CI, which runs `--skip-loki`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
 | Parity     | 10 checks                                                                                                                                                                            | 6 span attributes the external-parity dashboard panels read, plus 4 metric value-sanity bounds.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
 | Dashboards | Every uid in `expected_metrics.json` under `grafana_dashboards.uids` — currently all 15 provisioned dashboards                                                                       | Each listed dashboard loads and reports a panel count. This is a provisioning check only: it does **not** execute the panels' queries, so a dashboard can pass while individual panels render empty. `log-derived-insights` is Loki-backed, so under `--skip-loki` only its provisioning is meaningfully covered.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |

From 638ab2f4f5698888b775bfd4f2829fceff8136a3 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Tue, 25 Aug 2026 15:05:53 +0100
Subject: [PATCH 32/38] test(telemetry): log every emitted metric family, not
 19 prefixes

_log_prometheus_metric_names exists to make name mismatches between
expected_metrics.json and actual emissions visible in CI logs, but it kept
only names matching 19 hard-coded prefixes. On the last CI run that showed
147 of 422 families, and none of the prefixes covered state_accounting_*,
node_family_*, overlay_peer_disconnects or the pathfind_* histograms, so
the coverage gap the preceding commit closes could not be seen through it
at all.

An allow-list can only ever surface names someone already thought to look
for, which is the opposite of what a discovery aid has to do, so the
filter is removed rather than extended. The whole list is a few kilobytes
of CI log. Sorted, so two runs' output can be diffed directly; the
Prometheus API promises no order.
---
 .../telemetry/workload/validate_telemetry.py  | 48 +++++++------------
 1 file changed, 17 insertions(+), 31 deletions(-)

diff --git a/docker/telemetry/workload/validate_telemetry.py b/docker/telemetry/workload/validate_telemetry.py
index 51b4f7bd74..5a583ee3ab 100644
--- a/docker/telemetry/workload/validate_telemetry.py
+++ b/docker/telemetry/workload/validate_telemetry.py
@@ -649,12 +649,25 @@ async def _validate_parent_child(
 async def _log_prometheus_metric_names(
     session: aiohttp.ClientSession, prometheus_url: str
 ) -> None:
-    """Log the harness-relevant metric names Prometheus currently knows.
+    """Log every metric family name Prometheus currently knows.
 
     Diagnostic only — this output appears in CI logs and helps debug name
     mismatches between expected_metrics.json and actual emissions. Failures
     are warnings, never check failures.
 
+    Deliberately unfiltered. This used to keep only names matching 19
+    hard-coded prefixes, which made it useless for the job it exists to do: on
+    the last CI run it printed 147 of 422 families, and none of the prefixes
+    covered state_accounting_*, node_family_*, overlay_peer_disconnects or the
+    pathfind_* histograms, so a coverage gap in exactly those families could
+    not be seen here. An allow-list can only ever show names someone already
+    thought to look for, which is the opposite of what a discovery aid needs
+    to do. The whole list is a few kilobytes of CI log, so there is nothing to
+    save by truncating it.
+
+    Sorted so two runs' output can be diffed directly; the Prometheus API does
+    not promise an order.
+
     Args:
         session:        aiohttp client session.
         prometheus_url: Prometheus base URL.
@@ -664,38 +677,11 @@ async def _log_prometheus_metric_names(
             f"{prometheus_url}/api/v1/label/__name__/values"
         ) as resp:
             label_data = await resp.json()
-            all_metrics = label_data.get("data", [])
-            relevant = [
-                m
-                for m in all_metrics
-                if m.startswith(
-                    (
-                        "span_",
-                        "rpc_method",
-                        "cache_",
-                        "txq_",
-                        "object_count",
-                        "load_factor",
-                        "nodestore",
-                        "ledgermaster",
-                        "peer_finder",
-                        "jobq_",
-                        "total_bytes",
-                        "total_messages",
-                        "validation_agreement",
-                        "validator_health",
-                        "peer_quality",
-                        "ledger_economy",
-                        "state_tracking",
-                        "storage_detail",
-                    )
-                )
-            ]
+            all_metrics = sorted(label_data.get("data", []))
             logger.info(
-                "Prometheus metrics (relevant, %d of %d total): %s",
-                len(relevant),
+                "Prometheus metric families (%d total): %s",
                 len(all_metrics),
-                relevant,
+                all_metrics,
             )
     except Exception as exc:
         logger.warning("Failed to fetch Prometheus metric names: %s", exc)

From 7d35eb872a4f04d7ebf1a47744312f3f4f7369d8 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Tue, 25 Aug 2026 15:41:30 +0100
Subject: [PATCH 33/38] docs(telemetry): correct the harness contract's
 pathfinding and gauge reasons

The assert / do-not-assert decisions in expected_metrics.json and
expected_spans.json were all correct, but several recorded reasons were not.
Pathfinding is disabled outright on every harness node: Config.cpp:725-726
zeroes pathSearchMax whenever a [validation_seed] or [validator_token] section
is present, run-full-validation.sh writes [validation_seed] for every node and
has no [path_search] override, and both handlers return rpcNOT_SUPPORTED
before constructing a PathRequest.

- pathfind_full_milliseconds no longer claims a probabilistic path, nor
  prescribes an explicit ledger index, which cannot help: the config gate
  fires before the ledger parameter is read.
- pathfind_fast_milliseconds keeps its hasCompletion argument but now leads
  with the config gate, which is the operative blocker.
- The pathfind.compute and pathfind.discover notes and the
  pathfind.request to pathfind.compute skip reason no longer blame missing
  liquidity. pathfind.update_all now records why its request list stays empty.
- statsd_gauges states the arming precondition: a beast gauge is only as safe
  as an observable gauge when its object exists before Application.cpp:1570,
  where onCollectionReady arms the registered gauges exactly once.
- Alert wiring claims softened: every rule in rules.yaml is paused.
- The per job type gauge group loses its bogus poll bandwidth reason, and its
  regex claim is corrected: there is no running state regex, so 30 of those
  gauges have no consumer at all.
- overlay_peer_disconnects has one query consumer, not two.
- Cloud dashboard copies dropped from consumer counts: that tree is ignored by
  git and has no tracked files.
- rpc_method_errored_total explains that a refused RPC is a normal return, not
  a throw, so the refusals above do not make it fire.
- 09-data-collection-reference.md no longer claims a Prometheus name query in
  expected_metrics.json.

No behaviour change: the flattened check name list is byte identical.
---
 OpenTelemetryPlan/09-data-collection-reference.md | 14 ++++++++++++--
 docker/telemetry/workload/expected_metrics.json   | 14 +++++++-------
 docker/telemetry/workload/expected_spans.json     | 10 +++++-----
 3 files changed, 24 insertions(+), 14 deletions(-)

diff --git a/OpenTelemetryPlan/09-data-collection-reference.md b/OpenTelemetryPlan/09-data-collection-reference.md
index 896b679b70..68ce5b4723 100644
--- a/OpenTelemetryPlan/09-data-collection-reference.md
+++ b/OpenTelemetryPlan/09-data-collection-reference.md
@@ -1204,10 +1204,10 @@ docker/telemetry/workload/benchmark.sh --xrpld .build/xrpld --duration 300
 | ------------------------------ | ------------------- | -------------------------------- | ----------------------- |
 | Trace spans                    | 40 of 41 emitted    | Tempo API query                  | `expected_spans.json`   |
 | Span attributes                | 67 required         | Per-span attribute assertion     | `expected_spans.json`   |
-| Legacy beast::insight families | ~270 (≈224 traffic) | Prometheus `__name__` query      | `expected_metrics.json` |
+| Legacy beast::insight families | ~270 (≈224 traffic) | Named subset asserted            | `expected_metrics.json` |
 | Native MetricsRegistry         | 35 instruments      | Prometheus query                 | `expected_metrics.json` |
 | Call-site `XRPL_METRIC_*`      | 7 instruments       | Prometheus query                 | `expected_metrics.json` |
-| Per-job-type gauges            | 105 (35 types × 3)  | Prometheus `__name__` query      | `expected_metrics.json` |
+| Per-job-type gauges            | 105 (35 types × 3)  | 6 asserted by literal name       | `expected_metrics.json` |
 | SpanMetrics RED                | 4 per span          | Prometheus query                 | `expected_metrics.json` |
 | Grafana dashboards             | all 15 on disk      | Dashboard API load + panel count | `expected_metrics.json` |
 | Log-trace links                | Present             | Loki query + Tempo reverse check | —                       |
@@ -1217,6 +1217,16 @@ docker/telemetry/workload/benchmark.sh --xrpld .build/xrpld --duration 300
 > the **41** families the code emits ([§1.1](#11-complete-span-inventory-41-spans)) —
 > `rpc.ws_upgrade` has no entry — and 67 distinct required attributes (the
 > manifest's own `total_unique_attributes: 58` field is stale).
+> The **Validation Method** column for the two bulk beast rows used to read
+> "Prometheus `__name__` query", which never described anything real:
+> `expected_metrics.json` contains no `__name__` query and
+> `validate_telemetry.py` issues none. Both rows are covered by literal names —
+> `overlay_traffic` asserts the four `total_*` families out of ~228, and
+> `job_queue_per_type_gauges` asserts 6 of the 105 per-job-type gauges — with
+> the remainder not listed at all. The single `__name__` string inside
+> `expected_metrics.json` is prose quoting a `ledger-data-sync` dashboard query,
+> not a check.
+>
 > `expected_metrics.json` lists all **15** dashboard uids in
 > `docker/telemetry/grafana/dashboards/`, so dashboard coverage does not differ;
 > `log-derived-insights` is listed for the provisioning check only, and its panel
diff --git a/docker/telemetry/workload/expected_metrics.json b/docker/telemetry/workload/expected_metrics.json
index 81ca83d848..f94b4d78e3 100644
--- a/docker/telemetry/workload/expected_metrics.json
+++ b/docker/telemetry/workload/expected_metrics.json
@@ -1,5 +1,5 @@
 {
-  "description": "Expected metric inventory for xrpld telemetry validation. Metric names have no prefix (the xrpld_ prefix was removed). beast::insight metrics are lowercased by formatName. Every name here was verified against its declaration in MetricsRegistry.cpp or include/xrpl/telemetry/GetObjectMetricNames.h and against a panel query under docker/telemetry/grafana/dashboards/. IMPORTANT: validate_telemetry.py has no notion of an optional metric — validate_metrics() iterates every group that has a \"metrics\" key and hard-fails any name with 0 Prometheus series after a 45 s poll. A metric is therefore listed only when the harness workload guarantees it will appear: observable gauges/counters whose callbacks Observe unconditionally (series exist at value 0), or push counters/histograms on a path every run exercises. Workload-gated and defect-gated names are recorded in the \"not_asserted\" group, which intentionally has no \"metrics\" key so the validator skips it. Only series existence is checked, never a value, except for the four bounds checks hardcoded in PARITY_VALUE_SANITY. A group may additionally declare \"required_labels\": each label there becomes one check that at least one of the group's series carries it with a non-empty value, matched as <label>!=\"\" because Prometheus cannot tell an absent label from an empty one. The same guarantee rule applies — list a label only where the workload guarantees it.",
+  "description": "Expected metric inventory for xrpld telemetry validation. Metric names have no prefix (the xrpld_ prefix was removed). beast::insight metrics are lowercased by formatName. Every name here was verified against its declaration in MetricsRegistry.cpp or include/xrpl/telemetry/GetObjectMetricNames.h and against a panel query under docker/telemetry/grafana/dashboards/. IMPORTANT: validate_telemetry.py has no notion of an optional metric — validate_metrics() iterates every group that has a \"metrics\" key and hard-fails any name with 0 Prometheus series after a 45 s poll. A metric is therefore listed only when the harness workload guarantees it will appear: observable gauges/counters whose callbacks Observe unconditionally (series exist at value 0), or push counters/histograms on a path every run exercises. Workload-gated and defect-gated names are recorded in the \"not_asserted\" group, which intentionally has no \"metrics\" key so the validator skips it. Only series existence is checked, never a value, except for the four bounds checks hardcoded in PARITY_VALUE_SANITY. A group may additionally declare \"required_labels\": each label there becomes one check that at least one of the group's series carries it with a non-empty value, matched as <label>!=\"\" because Prometheus cannot tell an absent label from an empty one. The same guarantee rule applies — list a label only where the workload guarantees it. Two conventions apply to the consumer evidence cited in the group descriptions below. First, only tracked files count: dashboards under docker/telemetry/grafana/dashboards/ and rules under docker/telemetry/grafana/provisioning/alerting/. The docker/telemetry/grafanacloud/ tree is gitignored (sourceCode/.gitignore:27) and has zero tracked files, so a cloud copy cannot be cited as evidence on a clean checkout and is not counted here. Second, every rule in provisioning/alerting/rules.yaml is provisioned with isPaused: true (14 of 14), so a metric feeding a rule is a wiring dependency, not a live pager: losing it leaves the rule wired to a signal that no longer resolves, and it starts silent rather than becoming silent.",
   "spanmetrics": {
     "description": "SpanMetrics-derived RED metrics from the OTel Collector spanmetrics connector.",
     "metrics": [
@@ -32,7 +32,7 @@
     "_dimension_labels_note": "Bare label names as configured in otel-collector-config.yaml spanmetrics dimensions. Informational only (not asserted by the validator)."
   },
   "statsd_gauges": {
-    "description": "beast::insight gauges exported via OTLP/HTTP to the collector (server=otel). What guarantees these is the export mechanism, not the workload. On the OTel path a beast gauge is an Int64ObservableGauge (OTelCollector.cpp:703); every OTelGaugeImpl registers itself with the collector in its own constructor (:690); onCollectionReady() arms every registered gauge with no per-metric condition (:941-979); and the armed callback Observes currentValue() on each export cycle regardless of the value and regardless of whether set() was ever called (:719-727). Series therefore exist from the first export, at 0 if nothing has happened. The 'gauges only mark dirty on value changes' caveat in _poll_series_count is a property of the StatsD backend and does not apply here. The consequence is that a beast gauge whose object is constructed on an unconditional startup path is as safe to assert as an observable gauge from MetricsRegistry — which is why the state_accounting family below is asserted whole rather than one member deep. The ten state_accounting_* names are one family with one guarantee. NetworkOPsImp::Stats creates all ten in the NetworkOPsImp constructor (NetworkOPs.cpp:1069-1080; makeGauge(\"State_Accounting\", \"Full_duration\") joins on '.' at Collector.h:151-155 and formatName lowercases it, giving state_accounting_full_duration), and NetworkOPsImp::collectMetrics() set()s all ten unconditionally in a single block (:5204-5231). No state has to be entered for its pair to exist: a node that never disconnects still publishes state_accounting_disconnected_duration and _transitions at 0. All ten appear on the node-health dashboard and its grafanacloud copy, and state_accounting_full_transitions is also the input to the NodeStateFlapping alert rule (provisioning/alerting/rules.yaml, uid xrpld-node-state-flapping). Asserting only full_duration left the other nine unchecked, the alert's own input among them, even though a regression could not plausibly drop one sibling without dropping full_duration too. node_family_full_below_cache_hit_rate and _size are the TaggedCache Stats pair (TaggedCache.h:282-283), created in the cache constructor (TaggedCache.ipp:63-66) and published by collectMetrics(), which reports a hit_rate of 0 rather than skipping the gauge when hits+misses is 0 (TaggedCache.ipp:751-765). The instance is NodeFamily's full-below cache, constructed with the real collector in NodeFamily's constructor (NodeFamily.cpp:28-35), which every node builds. Both are on node-health. overlay_peer_disconnects is the first member of OverlayImpl::Stats (OverlayImpl.h:603) and OverlayImpl::collectMetrics() assigns it getPeerDisconnect() unconditionally (:646), in the same hook that publishes the overlay_traffic group's gauges. It is on network-traffic and log-derived-insights. It is listed here rather than in overlay_traffic because that group's contract is an explicit subset of the per-category traffic family, and a disconnect count is not one of those categories.",
+    "description": "beast::insight gauges exported via OTLP/HTTP to the collector (server=otel). What guarantees these is the export mechanism, not the workload. On the OTel path a beast gauge is an Int64ObservableGauge (OTelCollector.cpp:703); every OTelGaugeImpl registers itself with the collector in its own constructor (:690); onCollectionReady() arms every registered gauge with no per-metric condition (:941-979); and the armed callback Observes currentValue() on each export cycle regardless of the value and regardless of whether set() was ever called (:719-727). Series therefore exist from the first export, at 0 if nothing has happened. The 'gauges only mark dirty on value changes' caveat in _poll_series_count is a property of the StatsD backend and does not apply here. The consequence is that a beast gauge whose object is constructed on an unconditional startup path is as safe to assert as an observable gauge from MetricsRegistry — provided the object is constructed before Application.cpp:1570, which is where onCollectionReady() runs. That precondition is load-bearing and is the one way a gauge on this path can still go missing. Registration and arming are separate steps: addGauge() (:927-931) only appends to gauges_ under the lock, and onCollectionReady() snapshots that vector and calls arm() on each entry, exactly once per process (Application.cpp:1570, on the setup path, with no second call site). A gauge whose owner is built after that line is registered, never armed, and never exported — no callback, no series, and no error. Every gauge asserted in this group clears the precondition: jobQueue_, nodeFamily_, ledgerMaster_ and networkOPs_ are all in ApplicationImp's member-init list (Application.cpp:377, :430, :439, :465), and overlay_ — which owns peerFinder_ in its own member-init list (OverlayImpl.cpp:185) and Stats::peerDisconnects (OverlayImpl.h:603) — is built at Application.cpp:1551, nineteen lines before the arming call. The comment at Application.cpp:1258-1272 states that ordering deliberately: overlay_ is the last service the callbacks read, which is why arming waits for it. This is also why the state_accounting family below is asserted whole rather than one member deep. The ten state_accounting_* names are one family with one guarantee. NetworkOPsImp::Stats creates all ten in the NetworkOPsImp constructor (NetworkOPs.cpp:1069-1080; makeGauge(\"State_Accounting\", \"Full_duration\") joins on '.' at Collector.h:151-155 and formatName lowercases it, giving state_accounting_full_duration), and NetworkOPsImp::collectMetrics() set()s all ten unconditionally in a single block (:5204-5231). No state has to be entered for its pair to exist: a node that never disconnects still publishes state_accounting_disconnected_duration and _transitions at 0. All ten appear on the node-health dashboard, and state_accounting_full_transitions is also the input to the NodeStateFlapping alert rule (provisioning/alerting/rules.yaml:645-649, uid xrpld-node-state-flapping — provisioned isPaused: true, so this is a wiring dependency and not a live pager). Asserting only full_duration left the other nine unchecked, the alert's own input among them, even though a regression could not plausibly drop one sibling without dropping full_duration too. node_family_full_below_cache_hit_rate and _size are the TaggedCache Stats pair (TaggedCache.h:282-283), created in the cache constructor (TaggedCache.ipp:63-66) and published by collectMetrics(), which reports a hit_rate of 0 rather than skipping the gauge when hits+misses is 0 (TaggedCache.ipp:751-765). The instance is NodeFamily's full-below cache, constructed with the real collector in NodeFamily's constructor (NodeFamily.cpp:28-35), which every node builds. Both are on node-health. overlay_peer_disconnects is the first member of OverlayImpl::Stats (OverlayImpl.h:603) and OverlayImpl::collectMetrics() assigns it getPeerDisconnect() unconditionally (:646), in the same hook that publishes the overlay_traffic group's gauges. Its only query consumer is network-traffic.json:148. It also appears once in log-derived-insights.json:1908, but that occurrence is inside a panel description which explicitly disclaims it ('overlay_peer_disconnects exists as a metric but carries no reason breakdown, which is what this panel adds') — that panel is Loki-derived and queries nothing from Prometheus, so it is not a consumer. The third occurrence, validate_dashboards.py:17, is a lint rule listing the name as a cumulative gauge that must be rate()-wrapped; also not a query consumer. It is listed here rather than in overlay_traffic because that group's contract is an explicit subset of the per-category traffic family, and a disconnect count is not one of those categories.",
     "metrics": [
       "ledgermaster_validated_ledger_age",
       "ledgermaster_published_ledger_age",
@@ -59,7 +59,7 @@
     "metrics": ["rpc_requests_total", "ledger_fetches_total"]
   },
   "io_latency": {
-    "description": "io_context scheduling-latency histogram — a beast::insight Event exported over OTLP (Application.cpp:515, makeEvent('ios_latency') with the default millisecond unit). It is the one beast Event the node itself guarantees, which is why it is asserted while rpc_time_milliseconds and the jobq_* pairs sit in not_asserted: the sampler starts on the unconditional startup path (Application.cpp:1697, outside the 'if (withTimers)' guard), and its handler always emits the first sample whatever its value (Application.cpp:185-189, 'firstSample_.exchange(false) || lastSample >= 10ms', with a comment stating the point is to register the metric downstream). The OTel histogram is cumulative, so that one notify creates series that persist for the rest of the run. Named with the _bucket/_count/_sum suffixes the Prometheus exporter emits for a histogram — no bare ios_latency_milliseconds series exists, the same convention rpc_method_us and span_duration_milliseconds follow, and every consumer queries ios_latency_milliseconds_bucket: 2 distinct panels (ledger-data-sync 'I/O Scheduler Latency p95' and node-health 'I/O Latency', each mirrored in a grafanacloud copy) plus one alert rule (grafana/provisioning/alerting/rules.yaml:586). Only existence is asserted: past the first sample the handler reports only latencies >= 10 ms, so neither the sample count nor the value is predictable.",
+    "description": "io_context scheduling-latency histogram — a beast::insight Event exported over OTLP (Application.cpp:515, makeEvent('ios_latency') with the default millisecond unit). It is the one beast Event the node itself guarantees, which is why it is asserted while rpc_time_milliseconds and the jobq_* pairs sit in not_asserted: the sampler starts on the unconditional startup path (Application.cpp:1697, outside the 'if (withTimers)' guard), and its handler always emits the first sample whatever its value (Application.cpp:185-189, 'firstSample_.exchange(false) || lastSample >= 10ms', with a comment stating the point is to register the metric downstream). The OTel histogram is cumulative, so that one notify creates series that persist for the rest of the run. Named with the _bucket/_count/_sum suffixes the Prometheus exporter emits for a histogram — no bare ios_latency_milliseconds series exists, the same convention rpc_method_us and span_duration_milliseconds follow, and every consumer queries ios_latency_milliseconds_bucket: 2 distinct panels (ledger-data-sync 'I/O Scheduler Latency p95' and node-health 'I/O Latency') plus one alert rule (grafana/provisioning/alerting/rules.yaml:586). Only existence is asserted: past the first sample the handler reports only latencies >= 10 ms, so neither the sample count nor the value is predictable.",
     "metrics": [
       "ios_latency_milliseconds_bucket",
       "ios_latency_milliseconds_count",
@@ -113,7 +113,7 @@
     ]
   },
   "job_queue_per_type_gauges": {
-    "description": "Per-job-type queue-depth gauges from the beast::insight \"jobq\" group, for the six job types the ledger-sync diagnostics single out. JobTypeData's constructor creates waiting/running/deferred for every non-special job type (JobTypeData.h:96-103), the JobQueue constructor eagerly constructs a JobTypeData for every type in JobTypes (JobQueue.cpp:45-53), and JobQueue::collect() assigns all three from one snapshot for every type on every hook cycle, with no per-type condition (JobQueue.cpp:104-106). The collector is the \"jobq\" group, so GroupImp::makeName() prefixes \"jobq.\" and formatName lowercases the job type: JtTxnData's \"fetchTxnData\" + \"_running\" exports as jobq_fetchtxndata_running (JobTypeData.h:29-37). Existence rests on the same ObservableGauge mechanism documented on statsd_gauges, so these series exist at 0 on an idle node. All six were present in the CI validation run's own emitted-metric list, which is the strongest evidence behind any entry in this file. All six job types are non-special: makeFetchPack (limit 1), manifest (maxLimit), ledgerRequest (3), ledgerData (3), updatePaths (1) and fetchTxnData (5) all declare a non-zero limit in JobTypes.h, so none is skipped by the special() guard in JobTypeData's constructor. The 11 special types (limit 0) get no gauges at all, which is why no jobq_pathfind_* or jobq_peercommand_* name can be asserted. Why these six and not all 105. The guarantee is identical for every one of the 35 non-special types x 3 states, so the discriminator here is consumer coverage rather than emission: these six are the names a dashboard or an alert queries literally, so losing one blanks a specific panel or silently disarms a rule. jobq_manifest_waiting is the input to the ManifestJobQueueConvoy alert rule (provisioning/alerting/rules.yaml, uid xrpld-manifest-job-convoy); the other five are per-type saturation panels on node-health and its grafanacloud copy. The remaining names are only ever reached through the __name__=~\"jobq_.*_waiting\" / \"jobq_.*_deferred\" topk queries on ledger-data-sync, which keep working as long as the family exists and do not depend on any single job type being present. Asserting all 105 would add 99 checks that share one failure mode with these six while costing poll bandwidth in the shared 45 s window, so it would buy no signal.",
+    "description": "Per-job-type queue-depth gauges from the beast::insight \"jobq\" group, for the six job types the ledger-sync diagnostics single out. JobTypeData's constructor creates waiting/running/deferred for every non-special job type (JobTypeData.h:96-103), the JobQueue constructor eagerly constructs a JobTypeData for every type in JobTypes (JobQueue.cpp:45-53), and JobQueue::collect() assigns all three from one snapshot for every type on every hook cycle, with no per-type condition (JobQueue.cpp:104-106). The collector is the \"jobq\" group, so GroupImp::makeName() prefixes \"jobq.\" and formatName lowercases the job type: JtTxnData's \"fetchTxnData\" + \"_running\" exports as jobq_fetchtxndata_running (JobTypeData.h:29-37). Existence rests on the same ObservableGauge mechanism documented on statsd_gauges, so these series exist at 0 on an idle node. All six were present in the CI validation run's own emitted-metric list, which is the strongest evidence behind any entry in this file. All six job types are non-special: makeFetchPack (limit 1), manifest (maxLimit), ledgerRequest (3), ledgerData (3), updatePaths (1) and fetchTxnData (5) all declare a non-zero limit in JobTypes.h, so none is skipped by the special() guard in JobTypeData's constructor. The 11 special types (limit 0) get no gauges at all, which is why no jobq_pathfind_* or jobq_peercommand_* name can be asserted. Why these six and not all 105. The guarantee is identical for every one of the 35 non-special types x 3 states, so the discriminator here is consumer coverage rather than emission: these six are the only per-type names a dashboard or an alert queries literally, so losing one blanks a specific panel or leaves a rule wired to a signal that no longer resolves. jobq_manifest_waiting is the input to the ManifestJobQueueConvoy alert rule (provisioning/alerting/rules.yaml:796-800, uid xrpld-manifest-job-convoy — provisioned isPaused: true like every rule in that file, so the consequence is a broken wire rather than a silenced pager); the other five are per-type saturation panels on node-health.json:5267-5291. The rest of the family is reached only through two regex queries, both on ledger-data-sync: __name__=~\"jobq_.*_deferred\" at :1701 and __name__=~\"jobq_.*_waiting\" at :1708. Those are the only two jobq regexes anywhere in the tree — there is no jobq_.*_running regex — so the 30 unasserted _running gauges have zero consumers of any kind, and the unasserted _waiting and _deferred gauges are reached only by a topk that keeps working as long as the family exists and depends on no single job type. Consumer coverage is the whole argument and it stands alone. An earlier revision also cited poll bandwidth; that reason was wrong and has been removed, because METRIC_POLL_CONCURRENCY is 8 and the semaphore is released across each sleep, so 105 targets cost on the order of a second against the shared 45 s deadline. A bogus cost argument would be a bad reason to refuse a legitimate expansion later: if a panel or rule starts naming a seventh job type, add it.",
     "metrics": [
       "jobq_fetchtxndata_running",
       "jobq_ledgerdata_running",
@@ -199,7 +199,7 @@
   "not_asserted": {
     "description": "Emitted-and-dashboarded metrics deliberately left unasserted because they are workload-gated or defect-gated: the harness workload cannot guarantee they appear, and a check that fails on a healthy run is worse than no check. This group has no \"metrics\" key, so validate_telemetry.py skips it (validate_metrics iterates category_data.get(\"metrics\", [])). Promote an entry into an asserted group only after the workload is changed to guarantee it.",
     "metrics_excluded": {
-      "rpc_method_errored_total": "MetricsRegistry.cpp:354, push counter — needs an RPC that returns an error. rpc_load_generator.py issues only well-formed server_info / fee / ledger / ripple_path_find calls, so no series may ever be created.",
+      "rpc_method_errored_total": "MetricsRegistry.cpp:332-333, push counter. 'Errored' here means a thrown C++ exception, not an error status in the JSON reply: the only caller is PerfLogImp.cpp:409 under 'if (!finish)', reached only through PerfLogImp::rpcError (PerfLogImp.h:150-153), whose only call site is the catch (std::exception&) handler in RPCHandler.cpp:213. An RPC that returns an error status normally still takes the rpcFinish path at RPCHandler.cpp:190 and increments rpc_method_finished_total. That distinction matters here, because the generator's ripple_path_find calls are in fact refused — pathfinding is off on every harness node, so doRipplePathFind returns rpcNOT_SUPPORTED (see pathfind_fast_milliseconds below) — and it would be easy to conclude from that alone that this counter must fire. It does not: a refusal is a normal return, not a throw. Nothing in rpc_load_generator.py's server_info / fee / ledger / ripple_path_find mix is expected to throw, so no series may ever be created.",
       "ledger_history_mismatch_total": "MetricsRegistry.cpp:377, incremented only from LedgerHistory.cpp:332 on a built-vs-validated ledger mismatch. On a healthy run it never fires — asserting it would mean asserting a defect.",
       "txq_expired_total": "MetricsRegistry.cpp:379, incremented only at TxQ.cpp:1428 when a queued tx expires past its LastLedgerSequence. CI does run a txq-burst phase (workload-profiles.json:41, 30 s of single-type Payment at 60 TPS), but that does not guarantee sustained fee escalation followed by expiry: a run in which every other check passed still exposed only txq_metrics and no txq_expired_total.",
       "txq_dropped_total": "MetricsRegistry.cpp:381, incremented only at TxQ.cpp:1302 / :1347 on queue-full admission refusal. Same reason as txq_expired_total.",
@@ -210,8 +210,8 @@
       "getobject_charge": "GetObjectMetricNames.h:105, PeerImp.cpp:2931. Same gate.",
       "rpc_size_bytes": "ServerHandler.cpp:191, group('rpc')->makeEvent('size', Unit::Bytes). The OTLP Prometheus exporter derives the metric-name suffix from the declared unit, so a byte unit yields rpc_size_bytes. The Unit::Bytes declaration itself landed earlier, in 76c9051203; what 24094e427b changed was the exporter finally consuming it, replacing a hardcoded CreateDoubleHistogram(name, 'Duration in ms', 'ms') with otelUnitDescription(unit)/otelUnitCode(unit), and that is what renamed the series off rpc_size_milliseconds and the millisecond bucket ladder. Neither name was ever recorded here, so the harness could confirm neither the rename nor a regression back onto that ladder. Notified from ServerHandler::processRequest:1133, the HTTP JSON-RPC path — it computes an HTTP status and appends a trailing newline — and the load generators are WebSocket-only, the same gate regression-metrics.json:4 records for rpc.process, so only the harness's handful of HTTP health polls reach it. Real coverage needs an HTTP JSON-RPC phase in rpc_load_generator.py; that is a workload change rather than a harness correction, and is deliberately out of scope here.",
       "rpc_time_milliseconds": "ServerHandler.cpp:192, group('rpc')->makeEvent('time') with the default millisecond unit. Notified from ServerHandler::processRequest:1129, the same HTTP JSON-RPC call site as rpc_size_bytes and behind the same WebSocket-only gate.",
-      "pathfind_fast_milliseconds": "PathRequestManager.h:35, makeEvent('pathfind_fast') with the default millisecond unit, so the exported form is the pathfind_fast_milliseconds_bucket/_count/_sum triple and there is no bare series — the same convention io_latency and rpc_method_us follow, and rpc-pathfinding queries the _bucket. Notified from PathRequestManager::reportFast (:81-84), whose only caller is PathRequest.cpp:852, inside the 'if (fast && quickReply_ == {})' branch of PathRequest::doUpdate. The only doUpdate call that passes fast=true is in PathRequest::doCreate (PathRequest.cpp:259), and it is guarded by '!hasCompletion()'. That guard is what makes this unreachable for the harness: ripple_path_find has two entry points and both construct the PathRequest with a completion function, so hasCompletion() (:161-164) is true and the fast pass is skipped. doRipplePathFind with no ledger specified goes to makeLegacyPathRequest, which passes the coroutine-post lambda (RipplePathFind.cpp:140-160); with a ledger specified it goes to doLegacyPathRequest, which passes an empty-body but non-null lambda (PathRequestManager.cpp:317). Only the path_find streaming subscription reaches reportFast: makePathRequest builds the request from a subscriber with no completion (PathRequestManager.cpp:261), so hasCompletion() is false. The load generator does not use path_find and says so at rpc_load_generator.py:72-73, so the 3% ripple_path_find weight cannot produce this metric at all — not rarely, never. That matches the Grafana Cloud reading of zero series in 180 days on devnet nodes serving live traffic. Covering it needs a path_find subscription phase in the generator, which is a workload change and out of scope here.",
-      "pathfind_full_milliseconds": "PathRequestManager.h:36, makeEvent('pathfind_full'); same histogram naming as pathfind_fast_milliseconds above. Notified from reportFull (:87-90) via PathRequest.cpp:857, the 'else if (!fast && fullReply_ == {})' branch. Unlike the fast event this one is reachable from the harness workload, but not on the request's own turn, which is why it is recorded here rather than asserted. The generator sends no ledger/ledger_index/ledger_hash (rpc_load_generator.py:343-350) and CI is not standalone, so doRipplePathFind takes the makeLegacyPathRequest branch, which only enqueues the request; the fast=false update happens later, when the next ledger close drives PathRequestManager::updateAll into its one-shot branch (PathRequestManager.cpp:160-166). Emission is therefore one ledger close behind the request and conditional on the request still being alive and still passing isValid() at that point (PathRequest.cpp:772-773) — none of which the harness arranges or checks, and a phase that ends before the next close records nothing. The synchronous route that would make this deterministic, doLegacyPathRequest calling doUpdate(cache, false) directly (PathRequestManager.cpp:321), needs an explicit ledger parameter the generator does not send. A probabilistic path is exactly what the doctrine at the top of this file excludes. Grafana Cloud shows zero series in 180 days on the devnet nodes, consistent with those nodes receiving no pathfinding RPC rather than with a broken exporter.",
+      "pathfind_fast_milliseconds": "PathRequestManager.h:35, makeEvent('pathfind_fast') with the default millisecond unit, so the exported form is the pathfind_fast_milliseconds_bucket/_count/_sum triple and there is no bare series — the same convention io_latency and rpc_method_us follow, and rpc-pathfinding queries the _bucket. THE OPERATIVE BLOCKER IS THE CONFIG, NOT THE CALL GRAPH: pathfinding is disabled outright on every harness node, so no PathRequest is ever constructed and no pathfind_* histogram can exist. Config.cpp:725-726 sets pathSearchMax to 0 whenever a [validation_seed] or [validator_token] section is present ('By default, validators don't have pathfinding enabled'); run-full-validation.sh writes [validation_seed] into every generated node cfg (:308) and contains no [path_search], [path_search_fast] or [path_search_max] section to put it back (grep count 0 for path_search in that file — the only [path_search*] sections in docker/telemetry/ are in xrpld-telemetry.cfg and xrpld-telemetry-mainnet.cfg, neither of which the harness uses); and doRipplePathFind returns rpcNOT_SUPPORTED at RipplePathFind.cpp:48-49, before context.loadType is set and before any branch on the ledger parameter. So every ripple_path_find call the generator makes (3% weight, rpc_load_generator.py:74) is refused at the front door. Read that first: the structural argument below is correct and matters if pathfinding is ever enabled, but it is not why the metric is missing today. STRUCTURAL ARGUMENT (verified, applies once pathSearchMax is non-zero): reportFast's only caller is PathRequest.cpp:852, inside the 'if (fast && quickReply_ == {})' branch of PathRequest::doUpdate. The only doUpdate call that passes fast=true is in PathRequest::doCreate (PathRequest.cpp:259), guarded by '!hasCompletion()'. Both ripple_path_find entry points construct the PathRequest with a completion function, so hasCompletion() (:161-164) is true and the fast pass is skipped: with no ledger specified doRipplePathFind goes to makeLegacyPathRequest, which passes the coroutine-post lambda (RipplePathFind.cpp:140-160); with a ledger specified it goes to doLegacyPathRequest, which passes an empty-body but non-null lambda (PathRequestManager.cpp:317). Only the path_find streaming subscription reaches reportFast, because makePathRequest builds the request from a subscriber with no completion (PathRequestManager.cpp:261). The load generator does not use path_find and says so at rpc_load_generator.py:72-73. Covering this metric therefore needs both a [path_search_max] override (or a non-validator node) in run-full-validation.sh and a path_find subscription phase in the generator. Both are harness/workload changes and out of scope here. Grafana Cloud shows zero series in 180 days on the devnet nodes, consistent with those nodes receiving no pathfinding RPC.",
+      "pathfind_full_milliseconds": "PathRequestManager.h:36, makeEvent('pathfind_full'); same histogram naming as pathfind_fast_milliseconds above. Notified from reportFull (:87-90) via PathRequest.cpp:857, the 'else if (!fast && fullReply_ == {})' branch. Blocked by exactly the same config gate as pathfind_fast_milliseconds, and the probability of emission under the harness workload is zero, not low: pathSearchMax is 0 on every harness node (Config.cpp:725-726 plus the [validation_seed] section at run-full-validation.sh:308, with no [path_search*] override anywhere in that file), so doRipplePathFind returns rpcNOT_SUPPORTED at RipplePathFind.cpp:48-49 and no PathRequest object is ever constructed for reportFull to fire from. An earlier revision of this entry described the path as reachable-but-probabilistic — emitting one ledger close behind the request via PathRequestManager::updateAll's one-shot branch (PathRequestManager.cpp:160-166) — and prescribed sending an explicit ledger_index from the generator so doLegacyPathRequest would call doUpdate(cache, false) synchronously (PathRequestManager.cpp:321). Both halves were wrong. The probability is zero rather than merely unreliable, and the prescribed remedy cannot work at all, because the pathSearchMax guard fires before the ledger parameter is read: adding ledger_index changes nothing while pathfinding is off. Covering this metric needs a [path_search_max] override (or a non-validator node) in run-full-validation.sh — a harness-topology change, out of scope here. Grafana Cloud shows zero series in 180 days on the devnet nodes, consistent with those nodes receiving no pathfinding RPC rather than with a broken exporter.",
       "warn_total": "include/xrpl/resource/detail/Logic.h:41, makeMeter('warn'). makeMeter maps to CreateUInt64Counter (OTelCollector.cpp:878-881 -> :773-777), so the Prometheus exporter appends _total; the meter is created on the bare collector with no group, hence the unprefixed name. Incremented only at Logic.h:481, inside the 'if (notify)' branch reached when a consumer's balance crosses kWarningThreshold. A cooperating 5-node cluster plus a rate-limited load generator never charges a consumer that far, and Grafana Cloud confirms zero series in 180 days. Recorded explicitly because this was briefly mis-diagnosed as a phantom metric: the rpc-pathfinding panel that queries it is correct, and renders empty only because the condition has not occurred.",
       "drop_total": "include/xrpl/resource/detail/Logic.h:42, makeMeter('drop'); same CreateUInt64Counter mapping and same _total suffix as warn_total. Incremented only at Logic.h:505, when a consumer's balance is at or above kDropThreshold and the connection is dropped. Grafana Cloud shows 2 live series, so unlike warn_total this one does fire in the wild — but only on a genuinely abusive consumer, which the harness deliberately does not create, so it is condition-gated all the same. Its rpc-pathfinding panel is likewise correct rather than phantom.",
       "jobq_*_milliseconds, jobq_*_q_milliseconds": "This key is a pattern rather than a literal metric name — unlike every other entry in this map it stands for a whole family, one pair per job type. Created per job type in JobTypeData.h:97-98 from info.name() and info.name() + kSuffixQueued ('_q'), so the exported names are jobq_<jobtype>_milliseconds and jobq_<jobtype>_q_milliseconds with the job type lowercased by formatName. Which job types appear depends on which jobs a run happens to schedule, so no individual name is guaranteed. They are also rounded up to a whole millisecond at source (Event.h:47-51 applies ceil to a millisecond value type), which is why 6e2b2da772 moved the ledger-data-sync q-wait panels off jobq_<jobtype>_q_milliseconds_bucket onto job_queued_us_bucket — they are poor assertion targets regardless."
diff --git a/docker/telemetry/workload/expected_spans.json b/docker/telemetry/workload/expected_spans.json
index fad53d1a24..bb1e68d734 100644
--- a/docker/telemetry/workload/expected_spans.json
+++ b/docker/telemetry/workload/expected_spans.json
@@ -343,7 +343,7 @@
         "pathfind_dest_account"
       ],
       "config_flag": "trace_rpc",
-      "note": "Fires on ripple_path_find / path_find RPC. Driven by the ripple_path_find load in rpc_load_generator.py. Created as an ambient (scoped) child inside the RPC command handler (RipplePathFind.cpp:35-36, PathFind.cpp:26-27), so its parent is the enclosing rpc.command.* span — RipplePathFind.cpp:30 states this explicitly."
+      "note": "Fires on ripple_path_find / path_find RPC. Driven by the ripple_path_find load in rpc_load_generator.py. Created as an ambient (scoped) child inside the RPC command handler (RipplePathFind.cpp:35-36, PathFind.cpp:26-27), so its parent is the enclosing rpc.command.* span — RipplePathFind.cpp:30 states this explicitly. This span still emits on the harness even though pathfinding itself is disabled there, and the reason is purely positional: the ScopedSpanGuard is constructed at RipplePathFind.cpp:35, above the 'if (pathSearchMax == 0) return rpcError(RpcNotSupported)' guard at :48-49, so the span opens and closes around a request that is refused before any PathRequest exists. That is why this entry is asserted while pathfind.compute and pathfind.discover below are not."
     },
     {
       "name": "pathfind.compute",
@@ -352,7 +352,7 @@
       "required_attributes": ["pathfind_fast"],
       "config_flag": "trace_rpc",
       "optional": true,
-      "note": "Only fires when PathRequest::doUpdate runs a computation; the self-to-self XRP probe from the load generator returns early without computing paths in a fresh cluster with no liquidity."
+      "note": "Created by PathRequest::doUpdate (PathRequest.cpp:749-750), which the harness never reaches: pathfinding is disabled on every harness node, so no PathRequest is ever constructed. Config.cpp:725-726 sets pathSearchMax to 0 when a [validation_seed] or [validator_token] section is present, run-full-validation.sh writes [validation_seed] for every node (:308) and has no [path_search*] override, and doRipplePathFind then returns rpcNOT_SUPPORTED at RipplePathFind.cpp:48-49 — above the request-construction branches and below the pathfind.request span guard at :35. Liquidity is not the reason and never was: the call is refused before any path search is attempted, so the outcome does not depend on what the ledger holds. Enabling this span needs a [path_search_max] override (or a non-validator node) in run-full-validation.sh."
     },
     {
       "name": "pathfind.discover",
@@ -361,7 +361,7 @@
       "required_attributes": ["pathfind_search_level", "pathfind_num_paths"],
       "config_flag": "trace_rpc",
       "optional": true,
-      "note": "Graph exploration; only fires under pathfind.compute, which needs real path liquidity not present in the fresh test cluster."
+      "note": "Graph exploration; only fires under pathfind.compute, which the harness never reaches because pathfinding is disabled on every node (see the pathfind.compute entry above for the config chain). Liquidity is not the reason: the RPC is refused before any search runs."
     },
     {
       "name": "pathfind.update_all",
@@ -370,7 +370,7 @@
       "required_attributes": ["pathfind_ledger_index", "pathfind_num_requests"],
       "config_flag": "trace_rpc",
       "optional": true,
-      "note": "Async recomputation at ledger close; only fires when there are active path_find subscriptions (the one-shot ripple_path_find load does not register one)."
+      "note": "Async recomputation at ledger close. PathRequestManager::updateAll emits the span only when requests_ is non-empty (PathRequestManager.cpp:88-95), so it needs a live path_find subscription. On the harness requests_ can never become non-empty at all: the only two insertPathRequest call sites are makePathRequest (:268) and makeLegacyPathRequest (:296), and both handlers return rpcNOT_SUPPORTED first because pathfinding is disabled on every node (PathFind.cpp:39, RipplePathFind.cpp:48; see the pathfind.compute entry above for the config chain)."
     },
     {
       "name": "grpc.*",
@@ -424,7 +424,7 @@
       "child": "pathfind.compute",
       "description": "Pathfind request contains the compute sub-span",
       "skip": true,
-      "skip_reason": "pathfind.compute only fires when a path computation actually runs; the self-to-self XRP probe in a fresh cluster with no liquidity returns before computing, so the child is not emitted under the harness workload."
+      "skip_reason": "Real relationship (pathfind.compute is created inside PathRequest::doUpdate at PathRequest.cpp:749-750, under the pathfind.request scope), but the child never exists on the harness because pathfinding is disabled on every node: Config.cpp:725-726 zeroes pathSearchMax whenever a [validation_seed] or [validator_token] section is present, run-full-validation.sh writes [validation_seed] for all five nodes (:308) with no [path_search*] override, and doRipplePathFind returns rpcNOT_SUPPORTED at RipplePathFind.cpp:48-49 before constructing a PathRequest. The parent still appears only because its ScopedSpanGuard is created at RipplePathFind.cpp:35, above that guard. Liquidity has nothing to do with it — the earlier 'no liquidity, returns before computing' reason was wrong, because no path search is attempted at all. Asserting this relationship needs a [path_search_max] override (or a non-validator node) in run-full-validation.sh."
     }
   ],
   "total_span_types": 41,

From c863b83a1c16699b2ba8c18a65f1c842eaea405c Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Tue, 25 Aug 2026 15:51:06 +0100
Subject: [PATCH 34/38] feat(telemetry): warn on telemetry the harness contract
 does not account for

validate_metrics and validate_spans only ever run one direction: read the
contract, ask the backend whether each listed name exists. Nothing looked the
other way, so a metric family or span name the contract omitted was invisible
by construction. Both emitted inventories were already being fetched for the
CI log and neither was compared back, which is how a 345 family metric gap and
7 unknown span names went unnoticed.

Add two reverse checks, metric.reverse_coverage and span.reverse_coverage.
Each names every emitted family the contract never mentions, sorted, one per
line, with counts in the report details.

Warn only, by design. passed is hardcoded True in a single shared builder, so
an unaccounted name cannot turn CI red: downstream branches legitimately add
telemetry an upstream contract has not seen yet, and a hard failure would
redden all of them for doing the right thing.

Bulk families are accounted for declaratively. A new top level
accounted_patterns list in expected_metrics.json holds anchored regexes with a
written reason each, covering the 105 per job type queue gauges, the 70 per job
type histogram families, the 228 overlay per category traffic families, and the
Prometheus scrape plumbing that is not xrpld telemetry. Job type shapes are
reduced structurally because every job type name lowercases to letters only;
traffic categories are enumerated instead, because they contain underscores and
a structural pattern there would swallow unrelated names. Anything outside
these shapes still surfaces.

Exporter shapes are folded before matching, so a histogram triple is accounted
for by an entry written for its base family and is never reported as three
separate gaps. Spans need no pattern list: the reverse check reuses the same
matcher the forward check uses, so a glob such as rpc.command.* covers every
command it expands to, and an optional entry still counts as known.

Also fix the diagnostic these checks feed on: both emitted lists were logged as
a single Python list repr, about 15 kB on one line for 422 families, unreadable
and impossible to compare between runs. Both now print one name per line.

_metric_check_targets now selects groups by testing that the value is an
object, rather than by excluding two key names, so a non group top level key
cannot break it. Output is byte identical: 79 metric plus 5 label checks, same
names in the same order.
---
 .../09-data-collection-reference.md           |  37 +-
 docker/telemetry/workload/README.md           |  66 +++-
 .../telemetry/workload/expected_metrics.json  |  24 +-
 .../telemetry/workload/validate_telemetry.py  | 340 ++++++++++++++++--
 docs/telemetry-runbook.md                     |  15 +-
 5 files changed, 428 insertions(+), 54 deletions(-)

diff --git a/OpenTelemetryPlan/09-data-collection-reference.md b/OpenTelemetryPlan/09-data-collection-reference.md
index 68ce5b4723..a98bec8496 100644
--- a/OpenTelemetryPlan/09-data-collection-reference.md
+++ b/OpenTelemetryPlan/09-data-collection-reference.md
@@ -1200,17 +1200,17 @@ docker/telemetry/workload/benchmark.sh --xrpld .build/xrpld --duration 300
 > below as **families currently emitting** (idle nodes under-report — workload-gated metrics such as
 > per-RPC/error counters appear only once exercised, which is Phase 10's purpose).
 
-| Category                       | Expected Count      | Validation Method                | Config File             |
-| ------------------------------ | ------------------- | -------------------------------- | ----------------------- |
-| Trace spans                    | 40 of 41 emitted    | Tempo API query                  | `expected_spans.json`   |
-| Span attributes                | 67 required         | Per-span attribute assertion     | `expected_spans.json`   |
-| Legacy beast::insight families | ~270 (≈224 traffic) | Named subset asserted            | `expected_metrics.json` |
-| Native MetricsRegistry         | 35 instruments      | Prometheus query                 | `expected_metrics.json` |
-| Call-site `XRPL_METRIC_*`      | 7 instruments       | Prometheus query                 | `expected_metrics.json` |
-| Per-job-type gauges            | 105 (35 types × 3)  | 6 asserted by literal name       | `expected_metrics.json` |
-| SpanMetrics RED                | 4 per span          | Prometheus query                 | `expected_metrics.json` |
-| Grafana dashboards             | all 15 on disk      | Dashboard API load + panel count | `expected_metrics.json` |
-| Log-trace links                | Present             | Loki query + Tempo reverse check | —                       |
+| Category                       | Expected Count      | Validation Method                           | Config File             |
+| ------------------------------ | ------------------- | ------------------------------------------- | ----------------------- |
+| Trace spans                    | 40 of 41 emitted    | Tempo API query                             | `expected_spans.json`   |
+| Span attributes                | 67 required         | Per-span attribute assertion                | `expected_spans.json`   |
+| Legacy beast::insight families | ~270 (≈224 traffic) | Named subset asserted; rest regex-accounted | `expected_metrics.json` |
+| Native MetricsRegistry         | 35 instruments      | Prometheus query                            | `expected_metrics.json` |
+| Call-site `XRPL_METRIC_*`      | 7 instruments       | Prometheus query                            | `expected_metrics.json` |
+| Per-job-type gauges            | 105 (35 types × 3)  | 6 by literal name; rest regex-accounted     | `expected_metrics.json` |
+| SpanMetrics RED                | 4 per span          | Prometheus query                            | `expected_metrics.json` |
+| Grafana dashboards             | all 15 on disk      | Dashboard API load + panel count            | `expected_metrics.json` |
+| Log-trace links                | Present             | Loki query + Tempo reverse check            | —                       |
 
 > **These are the harness's numbers, not the code's, and two of them differ.**
 > `docker/telemetry/workload/expected_spans.json` carries 40 span entries against
@@ -1220,12 +1220,17 @@ docker/telemetry/workload/benchmark.sh --xrpld .build/xrpld --duration 300
 > The **Validation Method** column for the two bulk beast rows used to read
 > "Prometheus `__name__` query", which never described anything real:
 > `expected_metrics.json` contains no `__name__` query and
-> `validate_telemetry.py` issues none. Both rows are covered by literal names —
+> `validate_telemetry.py` issues none. The single `__name__` string in that file
+> is prose quoting a `ledger-data-sync` dashboard query, not a check. What is
+> actually asserted in both rows is a subset named literally —
 > `overlay_traffic` asserts the four `total_*` families out of ~228, and
-> `job_queue_per_type_gauges` asserts 6 of the 105 per-job-type gauges — with
-> the remainder not listed at all. The single `__name__` string inside
-> `expected_metrics.json` is prose quoting a `ledger-data-sync` dashboard query,
-> not a check.
+> `job_queue_per_type_gauges` asserts 6 of the 105 per-job-type gauges. The
+> remainder is now **accounted for** rather than asserted: anchored regexes
+> under the top-level `accounted_patterns` list declare both cross products, so
+> the `metric.reverse_coverage` check does not report them, while any name
+> outside those shapes surfaces as unaccounted. That check warns and never
+> fails; see the reverse-coverage row in
+> [`docs/telemetry-runbook.md`](../docs/telemetry-runbook.md).
 >
 > `expected_metrics.json` lists all **15** dashboard uids in
 > `docker/telemetry/grafana/dashboards/`, so dashboard coverage does not differ;
diff --git a/docker/telemetry/workload/README.md b/docker/telemetry/workload/README.md
index 4e5a1b9b4c..e866e49b2f 100644
--- a/docker/telemetry/workload/README.md
+++ b/docker/telemetry/workload/README.md
@@ -451,7 +451,14 @@ it is held back as its own change rather than folded into an unrelated push.
   "not_asserted": {
     "description": "Why these are excluded.",
     "metrics_excluded": { "metric_3": "reason" }
-  }
+  },
+  "accounted_patterns": [
+    {
+      "pattern": "^family_[a-z]+_(a|b)$",
+      "family": "Short label.",
+      "reason": "Why."
+    }
+  ]
 }
 ```
 
@@ -459,19 +466,64 @@ Every metric listed under a `metrics` array must produce > 0 Prometheus series d
 
 `required_labels` is optional and read for every category that declares one. Each label becomes one additional check, named `metric.<category>.label.<label>`, that at least one of that category's series carries the label with a non-empty value. It is matched as `<label>!=""` rather than `<label>=~".*"` because Prometheus cannot tell an absent label from an empty one, so a regex match would pass on a node that lost the label entirely. The guarantee rule is the same as for `metrics`: list a label only where the workload guarantees it. Declaring `required_labels` on a category with no `metrics` fails the check rather than skipping it, so the key can never sit unenforced.
 
-Three top-level keys are not metric categories:
+Four top-level keys are not metric categories:
 
-- `description` and `grafana_dashboards` are skipped explicitly by
-  `validate_metrics`. `grafana_dashboards.uids` drives the dashboard check, so
-  adding a dashboard to `docker/telemetry/grafana/dashboards/` does **not** put
-  it under the gate until its uid is added here too.
-- `not_asserted` is skipped structurally: the loop reads
+- `description` is a string, and `accounted_patterns` is a list, so
+  `_metric_check_targets` skips both structurally — it walks only top-level
+  values that are objects.
+- `grafana_dashboards` is an object but declares no `metrics`, so it contributes
+  no checks. `grafana_dashboards.uids` drives the dashboard check, so adding a
+  dashboard to `docker/telemetry/grafana/dashboards/` does **not** put it under
+  the gate until its uid is added here too.
+- `not_asserted` is skipped the same way: the loop reads
   `category_data.get("metrics", [])`, and this group deliberately has no
   `metrics` key — its entries live under `metrics_excluded` as a name-to-reason
   map. It documents metrics that are emitted and dashboarded but left unasserted
   because they are workload-gated or defect-gated (a check that fails on a
   healthy run is worse than no check). Promote an entry into an asserted group
   only after the workload is changed to guarantee it fires.
+- `accounted_patterns` asserts nothing. It is a list of `{pattern, family,
+reason}` entries feeding the reverse coverage check described below.
+
+### Reverse Coverage — emitted but not accounted for
+
+The checks above run in one direction only: they read the contract and ask the
+backend whether each listed name exists. That direction is blind to a name the
+contract omits, which is how a large metric gap and several unknown spans went
+unnoticed — both emitted inventories were already being fetched for the CI log,
+and neither was compared back.
+
+Two extra checks close the loop, `metric.reverse_coverage` and
+`span.reverse_coverage`. Each lists every name the backend reports that the
+contract never mentions, sorted, one per line in the log, with counts in the
+report's `details`.
+
+**They warn and never fail.** `passed` is hardcoded `True` in
+`_reverse_coverage_result`, so an unaccounted name cannot turn CI red. That is
+deliberate: downstream branches legitimately add telemetry an upstream contract
+has not seen yet, and a hard failure would redden every one of them for doing
+the right thing. The value is visibility, not enforcement.
+
+A metric family counts as accounted for when any of these is true:
+
+- a group's `metrics` array lists it (with any label matcher stripped),
+- a `not_asserted.metrics_excluded` key names it — deliberately unasserted is
+  not the same as unknown,
+- an `accounted_patterns` regex matches it (anchored, `fullmatch`).
+
+Exporter shapes are folded before matching, so a contract entry written for a
+family covers what the exporter derives from it: a histogram's
+`_bucket`/`_count`/`_sum` names fold back onto their base family, and counters
+are listed with the `_total` the exporter appends. Spans need no pattern list —
+`expected_spans.json` already carries globs such as `rpc.command.*`, and the
+reverse check reuses the same matcher the forward check uses.
+
+Use `accounted_patterns` only for a family whose membership is derived
+mechanically from a table in the code and so cannot be enumerated by hand — the
+per-job-type job-queue instruments and the overlay per-category traffic cross
+product are the two real cases, plus the Prometheus scrape plumbing that is not
+xrpld telemetry at all. Keep each pattern anchored and no wider than its family
+needs: a pattern that swallows unrelated names defeats the check.
 
 ### expected_spans.json Format
 
diff --git a/docker/telemetry/workload/expected_metrics.json b/docker/telemetry/workload/expected_metrics.json
index f94b4d78e3..f1cf8b1cd6 100644
--- a/docker/telemetry/workload/expected_metrics.json
+++ b/docker/telemetry/workload/expected_metrics.json
@@ -1,5 +1,5 @@
 {
-  "description": "Expected metric inventory for xrpld telemetry validation. Metric names have no prefix (the xrpld_ prefix was removed). beast::insight metrics are lowercased by formatName. Every name here was verified against its declaration in MetricsRegistry.cpp or include/xrpl/telemetry/GetObjectMetricNames.h and against a panel query under docker/telemetry/grafana/dashboards/. IMPORTANT: validate_telemetry.py has no notion of an optional metric — validate_metrics() iterates every group that has a \"metrics\" key and hard-fails any name with 0 Prometheus series after a 45 s poll. A metric is therefore listed only when the harness workload guarantees it will appear: observable gauges/counters whose callbacks Observe unconditionally (series exist at value 0), or push counters/histograms on a path every run exercises. Workload-gated and defect-gated names are recorded in the \"not_asserted\" group, which intentionally has no \"metrics\" key so the validator skips it. Only series existence is checked, never a value, except for the four bounds checks hardcoded in PARITY_VALUE_SANITY. A group may additionally declare \"required_labels\": each label there becomes one check that at least one of the group's series carries it with a non-empty value, matched as <label>!=\"\" because Prometheus cannot tell an absent label from an empty one. The same guarantee rule applies — list a label only where the workload guarantees it. Two conventions apply to the consumer evidence cited in the group descriptions below. First, only tracked files count: dashboards under docker/telemetry/grafana/dashboards/ and rules under docker/telemetry/grafana/provisioning/alerting/. The docker/telemetry/grafanacloud/ tree is gitignored (sourceCode/.gitignore:27) and has zero tracked files, so a cloud copy cannot be cited as evidence on a clean checkout and is not counted here. Second, every rule in provisioning/alerting/rules.yaml is provisioned with isPaused: true (14 of 14), so a metric feeding a rule is a wiring dependency, not a live pager: losing it leaves the rule wired to a signal that no longer resolves, and it starts silent rather than becoming silent.",
+  "description": "Expected metric inventory for xrpld telemetry validation. Metric names have no prefix (the xrpld_ prefix was removed). beast::insight metrics are lowercased by formatName. Every name here was verified against its declaration in MetricsRegistry.cpp or include/xrpl/telemetry/GetObjectMetricNames.h and against a panel query under docker/telemetry/grafana/dashboards/. IMPORTANT: validate_telemetry.py has no notion of an optional metric — validate_metrics() iterates every group that has a \"metrics\" key and hard-fails any name with 0 Prometheus series after a 45 s poll. A metric is therefore listed only when the harness workload guarantees it will appear: observable gauges/counters whose callbacks Observe unconditionally (series exist at value 0), or push counters/histograms on a path every run exercises. Workload-gated and defect-gated names are recorded in the \"not_asserted\" group, which intentionally has no \"metrics\" key so the validator skips it. Only series existence is checked, never a value, except for the four bounds checks hardcoded in PARITY_VALUE_SANITY. A group may additionally declare \"required_labels\": each label there becomes one check that at least one of the group's series carries it with a non-empty value, matched as <label>!=\"\" because Prometheus cannot tell an absent label from an empty one. The same guarantee rule applies — list a label only where the workload guarantees it. Two conventions apply to the consumer evidence cited in the group descriptions below. First, only tracked files count: dashboards under docker/telemetry/grafana/dashboards/ and rules under docker/telemetry/grafana/provisioning/alerting/. The docker/telemetry/grafanacloud/ tree is gitignored (sourceCode/.gitignore:27) and has zero tracked files, so a cloud copy cannot be cited as evidence on a clean checkout and is not counted here. Second, every rule in provisioning/alerting/rules.yaml is provisioned with isPaused: true (14 of 14), so a metric feeding a rule is a wiring dependency, not a live pager: losing it leaves the rule wired to a signal that no longer resolves, and it starts silent rather than becoming silent. Finally, the top-level \"accounted_patterns\" list is not a group and asserts nothing. It declares anchored regexes for the bulk families that cannot usefully be enumerated as literal names — the per-job-type gauges and histograms, the overlay per-category traffic cross product — plus the Prometheus scrape plumbing that is not xrpld telemetry at all. Its only consumer is the reverse coverage check in validate_telemetry.py, which walks the metric families Prometheus actually holds and names every one that neither a group's \"metrics\" entry, nor a \"metrics_excluded\" key, nor one of these patterns accounts for. That check warns and never fails: downstream branches legitimately add telemetry this contract has not seen yet, and a hard failure would redden all of them. Its value is visibility, not enforcement.",
   "spanmetrics": {
     "description": "SpanMetrics-derived RED metrics from the OTel Collector spanmetrics connector.",
     "metrics": [
@@ -217,6 +217,28 @@
       "jobq_*_milliseconds, jobq_*_q_milliseconds": "This key is a pattern rather than a literal metric name — unlike every other entry in this map it stands for a whole family, one pair per job type. Created per job type in JobTypeData.h:97-98 from info.name() and info.name() + kSuffixQueued ('_q'), so the exported names are jobq_<jobtype>_milliseconds and jobq_<jobtype>_q_milliseconds with the job type lowercased by formatName. Which job types appear depends on which jobs a run happens to schedule, so no individual name is guaranteed. They are also rounded up to a whole millisecond at source (Event.h:47-51 applies ceil to a millisecond value type), which is why 6e2b2da772 moved the ledger-data-sync q-wait panels off jobq_<jobtype>_q_milliseconds_bucket onto job_queued_us_bucket — they are poor assertion targets regardless."
     }
   },
+  "accounted_patterns": [
+    {
+      "pattern": "^jobq_[a-z]+_(waiting|running|deferred)$",
+      "family": "Per-job-type queue-depth gauges: 35 non-special job types x 3 states = 105 families.",
+      "reason": "Cannot be enumerated usefully. JobTypeData's constructor creates waiting/running/deferred for every non-special job type (JobTypeData.h:96-103) and JobQueue eagerly constructs a JobTypeData for every type in JobTypes (JobQueue.cpp:45-53), so the family's membership is derived mechanically from the JobTypes.h table rather than chosen. Six members are asserted by literal name in job_queue_per_type_gauges on consumer-coverage grounds; this pattern accounts for the rest so the reverse coverage check does not report 99 names that the contract has already reasoned about. Narrow by construction: every one of the 35 non-special job type names lowercases to ^[a-z]+$ with no digits and no underscores (verified mechanically against JobTypes.h:52-86), and the state suffix is a closed set of three, so nothing outside this family can match. A new job type therefore stays covered without an edit here, which is correct — adding a job type is not a telemetry design change. A renamed state, a job type containing a digit or an underscore, or any other new jobq gauge shape would surface as unaccounted."
+    },
+    {
+      "pattern": "^jobq_[a-z]+(_q)?_milliseconds$",
+      "family": "Per-job-type job-latency and queue-wait histogram families: 35 job types x 2 = 70 families, each exported as a _bucket/_count/_sum triple.",
+      "reason": "The same mechanically derived family as the gauges above, in histogram form: JobTypeData.h:97-98 creates one Event from info.name() and one from info.name() + '_q'. Which job types actually record a sample depends on what a run schedules, which is why not_asserted.metrics_excluded carries these under a single pattern key rather than as literal names. The reverse coverage check matches this pattern against the base family name, after stripping the exporter's _bucket/_count/_sum suffix, so all three series of every triple are accounted for by this one entry. Same narrowness argument as above: the job type part is ^[a-z]+$ and the optional _q plus the _milliseconds unit suffix close the shape."
+    },
+    {
+      "pattern": "^(getobject_account_state_node_get|getobject_account_state_node_share|getobject_cas_get|getobject_cas_share|getobject_fetch_pack_get|getobject_fetch_pack_share|getobject_get|getobject_ledger_get|getobject_ledger_share|getobject_share|getobject_transaction_get|getobject_transaction_node_get|getobject_transaction_node_share|getobject_transaction_share|getobject_transactions_get|have_transactions|ledger_account_state_node_get|ledger_account_state_node_share|ledger_data_account_state_node_get|ledger_data_account_state_node_share|ledger_data_get|ledger_data_share|ledger_data_transaction_node_get|ledger_data_transaction_node_share|ledger_data_transaction_set_candidate_get|ledger_data_transaction_set_candidate_share|ledger_get|ledger_share|ledger_transaction_node_get|ledger_transaction_node_share|ledger_transaction_set_candidate_get|ledger_transaction_set_candidate_share|overhead|overhead_cluster|overhead_manifest|overhead_overlay|proof_path_request|proof_path_response|proposals|proposals_duplicate|proposals_untrusted|replay_delta_request|replay_delta_response|requested_transactions|set_get|set_share|squelch|squelch_ignored|squelch_suppressed|total|transactions|transactions_duplicate|unknown|validations|validations_duplicate|validations_untrusted|validator_lists)_(bytes|messages)_(in|out)$",
+      "family": "Overlay per-category traffic gauges: 57 categories x {bytes,messages} x {in,out} = 228 families, of which the four total_* are asserted literally in overlay_traffic.",
+      "reason": "Also mechanically derived: OverlayImpl builds one TrafficGauges per entry of TrafficCount::counts_ (OverlayImpl.cpp:200-203), and TrafficGauges creates Bytes_In / Bytes_Out / Messages_In / Messages_Out for each (OverlayImpl.h:580-593), so the set is exactly the cross product of the category table with those four names. The 57 categories are the 56 entries of TrafficCount::toString's kCategoryMap (TrafficCount.h:239-294) plus 'unknown', which is what Category::Unknown falls through to because it has an entry in counts_ but none in the map. The categories are enumerated here rather than reduced to a shape, because unlike the job types they contain underscores: a structural pattern would have to be ^[a-z_]+_(bytes|messages)_(in|out)$, which would silently swallow any future non-overlay metric ending in _bytes_in. Enumerating means a new traffic category surfaces as unaccounted, which is the right outcome: overlay_traffic's description calls itself an explicit subset and should be revisited when the table grows. The category list was extracted mechanically from TrafficCount.h; regenerate it the same way rather than editing by hand."
+    },
+    {
+      "pattern": "^(target_info|up|scrape_[a-z_]+|promhttp_[a-z_]+)$",
+      "family": "Prometheus scrape metadata and exporter housekeeping series.",
+      "reason": "Not xrpld telemetry, and not emitted by anything in this repository, so there is nothing for the contract to assert. prometheus.yml defines exactly one scrape job against the collector's Prometheus exporter (otel-collector:8889), and Prometheus synthesises up and the scrape_* family per target on every scrape. target_info is written by the collector's Prometheus exporter to carry the OTLP resource attributes, and several dashboards read it as label_values(target_info, service_instance_id). promhttp_* is the metrics handler's own counter set, present only if the exporter's handler exposes it. Listed so the reverse coverage check reports genuine telemetry gaps instead of scrape plumbing. Deliberately does not cover otelcol_*: the collector's internal telemetry defaults to port 8888 and otel-collector-config.yaml declares no service::telemetry block, so those series are not scraped here — if they ever appear, that is a collector configuration change worth seeing."
+    }
+  ],
   "grafana_dashboards": {
     "description": "All 15 Grafana dashboards provisioned on disk under docker/telemetry/grafana/dashboards/ (UID == file stem for every one). validate_dashboards() checks that each UID resolves via GET /api/dashboards/uid/<uid> and reports its panel count — it verifies provisioning and loadability, not panel data. log-derived-insights is included on that basis even though its panels are Loki-backed and CI runs with --skip-loki: the dashboard itself must still provision cleanly. Its panel data is not asserted anywhere.",
     "uids": [
diff --git a/docker/telemetry/workload/validate_telemetry.py b/docker/telemetry/workload/validate_telemetry.py
index 5a583ee3ab..7f67118d8a 100644
--- a/docker/telemetry/workload/validate_telemetry.py
+++ b/docker/telemetry/workload/validate_telemetry.py
@@ -79,6 +79,12 @@ METRIC_POLL_INTERVAL_SEC = 5.0
 # not hammer the single-container Prometheus the harness runs.
 METRIC_POLL_CONCURRENCY = 8
 
+# The Prometheus exporter splits one histogram instrument into three series
+# names. Reverse coverage folds them back onto the base family so a contract
+# entry (or an accounted_patterns regex) written for the family accounts for
+# all three, and so a triple is never reported as three separate gaps.
+HISTOGRAM_SUFFIXES = ("_bucket", "_count", "_sum")
+
 
 # ---------------------------------------------------------------------------
 # Data classes
@@ -169,6 +175,95 @@ class ValidationReport:
         }
 
 
+# ---------------------------------------------------------------------------
+# Reverse coverage: what is emitted but not accounted for
+# ---------------------------------------------------------------------------
+#
+# The forward checks read the contract and ask Prometheus/Tempo whether each
+# listed name exists. That direction cannot see a name the contract omits, and
+# omissions are how a 345-family metric gap and 7 unknown spans went unnoticed:
+# both emitted inventories were already being fetched, and neither was compared
+# back against the contract.
+#
+# These helpers close the loop. They are deliberately WARN-ONLY. Downstream
+# branches legitimately add telemetry that an upstream contract has not seen
+# (the sync-diagnostics branch emits 7 spans this contract does not list), so a
+# hard failure here would redden every one of them for doing the right thing.
+# The value is visibility: name the gaps, in a form a human can read in a CI
+# log and diff between runs, and let a person decide.
+
+
+def _log_name_list(header: str, names: list[str]) -> None:
+    """Log a name list one entry per line, sorted.
+
+    A single-line Python list repr of 422 metric families is ~15 kB on one CI
+    log line: unreadable, and impossible to diff between runs. One name per line
+    makes two runs' logs comparable. Emitted as a single log record so the
+    lines cannot be interleaved by another task's output.
+
+    Args:
+        header: Line printed above the list; the count is appended to it.
+        names:  Names to print. Sorted here, so callers need not be.
+    """
+    body = "\n".join(f"  {name}" for name in sorted(names)) or "  (none)"
+    logger.info("%s (%d total):\n%s", header, len(names), body)
+
+
+def _reverse_coverage_result(
+    check_name: str,
+    category: str,
+    noun: str,
+    emitted_count: int,
+    unaccounted: list[str],
+) -> CheckResult:
+    """Build the always-passing CheckResult for one reverse coverage check.
+
+    ``passed`` is hardcoded True. This is the single place that decides the
+    check cannot fail CI, so the guarantee is auditable in one line rather than
+    spread over two call sites. The finding travels in ``message`` and in
+    ``details["unaccounted"]``, and the names are also logged one per line by
+    the caller.
+
+    Args:
+        check_name:    Report check name (e.g. "metric.reverse_coverage").
+        category:      Report category ("metric" or "span").
+        noun:          Plural noun for the message ("metric families", "span names").
+        emitted_count: How many names the backend reported.
+        unaccounted:   Names the contract does not account for.
+
+    Returns:
+        A CheckResult that always passes.
+    """
+    if emitted_count == 0:
+        message = (
+            f"reverse coverage not evaluated: no {noun} were reported "
+            f"(backend unreachable or empty) — warning only, never fails"
+        )
+    elif unaccounted:
+        message = (
+            f"WARNING: {len(unaccounted)} of {emitted_count} emitted {noun} are "
+            f"not accounted for by the contract: {', '.join(sorted(unaccounted)[:5])}"
+            f"{' …' if len(unaccounted) > 5 else ''} "
+            f"(full list logged above; warning only, never fails)"
+        )
+    else:
+        message = f"all {emitted_count} emitted {noun} are accounted for"
+
+    return CheckResult(
+        name=check_name,
+        category=category,
+        passed=True,
+        message=message,
+        details={
+            "emitted": emitted_count,
+            "accounted": emitted_count - len(unaccounted),
+            "unaccounted_count": len(unaccounted),
+            "unaccounted": sorted(unaccounted),
+            "enforced": False,
+        },
+    )
+
+
 # ---------------------------------------------------------------------------
 # Tempo API helpers
 # ---------------------------------------------------------------------------
@@ -268,6 +363,33 @@ def _span_name_matches(emitted_name: str, expected_name: str) -> bool:
     return emitted_name == expected_name
 
 
+def _unaccounted_span_names(emitted: list[str], expected: dict[str, Any]) -> list[str]:
+    """Names Tempo reports that no expected_spans.json entry accounts for.
+
+    "Accounted for" is the same relation the forward check uses, so a contract
+    glob such as "rpc.command.*" covers every command it expands to, and an
+    entry marked ``optional`` still accounts for its name — being unasserted is
+    not the same as being unknown. No separate pattern list is needed for spans
+    because the contract already carries globs.
+
+    Args:
+        emitted:  Span names as reported by Tempo's span.name tag values.
+        expected: The parsed expected_spans.json contract.
+
+    Returns:
+        Sorted list of emitted names with no matching contract entry.
+    """
+    contract = [span_def["name"] for span_def in expected.get("spans", [])]
+    return sorted(
+        name
+        for name in emitted
+        if name
+        and not any(
+            _span_name_matches(name, expected_name) for expected_name in contract
+        )
+    )
+
+
 # ---------------------------------------------------------------------------
 # Span Validation (Tempo API)
 # ---------------------------------------------------------------------------
@@ -327,21 +449,21 @@ async def validate_spans(
         )
         return
 
-    # Diagnostic: list all available operations (span names) for the xrpld
-    # service.  This output appears in CI logs and helps debug missing-span
-    # failures without needing to reproduce the full stack locally.
+    # List every span name Tempo holds. This is both a diagnostic (it makes a
+    # missing-span failure debuggable without reproducing the stack locally)
+    # and the input to the reverse coverage check added at the end of this
+    # function. Note the tag-values API is not service-scoped, so in a stack
+    # where something other than xrpld also sent traces this list would be a
+    # superset; on the harness only xrpld exports spans.
+    emitted_span_names: list[str] = []
     try:
         async with session.get(
             f"{tempo_url}/api/v2/search/tag/span.name/values"
         ) as resp:
             ops_data = await resp.json()
             tag_values = ops_data.get("tagValues", [])
-            operations = [tv.get("value", "") for tv in tag_values]
-            logger.info(
-                "Tempo operations (%d total): %s",
-                len(operations),
-                operations,
-            )
+            emitted_span_names = [tv.get("value", "") for tv in tag_values]
+            _log_name_list("Tempo span names", emitted_span_names)
     except Exception as exc:
         logger.warning("Failed to fetch Tempo operations: %s", exc)
 
@@ -447,6 +569,25 @@ async def validate_spans(
             continue
         await _validate_parent_child(session, tempo_url, rel, report)
 
+    # Reverse direction: span names Tempo holds that the contract never mentions.
+    # Added last so no existing check's position in the report moves.
+    unaccounted_spans = _unaccounted_span_names(emitted_span_names, expected)
+    if unaccounted_spans:
+        _log_name_list(
+            "Span names emitted but NOT accounted for by expected_spans.json "
+            "(warning only, does not fail CI)",
+            unaccounted_spans,
+        )
+    report.add(
+        _reverse_coverage_result(
+            check_name="span.reverse_coverage",
+            category="span",
+            noun="span names",
+            emitted_count=len(emitted_span_names),
+            unaccounted=unaccounted_spans,
+        )
+    )
+
 
 async def _check_attributes_on_first_trace(
     session: aiohttp.ClientSession,
@@ -648,12 +789,15 @@ async def _validate_parent_child(
 
 async def _log_prometheus_metric_names(
     session: aiohttp.ClientSession, prometheus_url: str
-) -> None:
-    """Log every metric family name Prometheus currently knows.
+) -> list[str]:
+    """Log every metric family name Prometheus knows, and return the list.
 
-    Diagnostic only — this output appears in CI logs and helps debug name
-    mismatches between expected_metrics.json and actual emissions. Failures
-    are warnings, never check failures.
+    Two jobs. It is the diagnostic that makes a name mismatch between
+    expected_metrics.json and actual emissions debuggable from a CI log, and it
+    is the emitted inventory the reverse coverage check compares the contract
+    against. Failures are warnings, never check failures; an empty return means
+    "could not determine", which the reverse check reports as not evaluated
+    rather than as a clean run.
 
     Deliberately unfiltered. This used to keep only names matching 19
     hard-coded prefixes, which made it useless for the job it exists to do: on
@@ -665,12 +809,16 @@ async def _log_prometheus_metric_names(
     to do. The whole list is a few kilobytes of CI log, so there is nothing to
     save by truncating it.
 
-    Sorted so two runs' output can be diffed directly; the Prometheus API does
-    not promise an order.
+    Sorted and printed one name per line so two runs' output can be diffed
+    directly; the Prometheus API does not promise an order, and the whole list
+    on one log line was neither readable nor diffable.
 
     Args:
         session:        aiohttp client session.
         prometheus_url: Prometheus base URL.
+
+    Returns:
+        Sorted metric family names, or an empty list if the fetch failed.
     """
     try:
         async with session.get(
@@ -678,13 +826,131 @@ async def _log_prometheus_metric_names(
         ) as resp:
             label_data = await resp.json()
             all_metrics = sorted(label_data.get("data", []))
-            logger.info(
-                "Prometheus metric families (%d total): %s",
-                len(all_metrics),
-                all_metrics,
-            )
+            _log_name_list("Prometheus metric families", all_metrics)
+            return all_metrics
     except Exception as exc:
         logger.warning("Failed to fetch Prometheus metric names: %s", exc)
+        return []
+
+
+def _selector_metric_name(selector: str) -> str:
+    """Strip any label matcher from a contract selector, leaving the name.
+
+    Contract entries are usually bare names but some carry a matcher, e.g.
+    ``ledger_economy{metric="base_fee_xrp"}``. Reverse coverage compares family
+    names, and Prometheus reports one ``__name__`` per family regardless of how
+    many label combinations it has, so the matcher must come off first.
+
+    Args:
+        selector: A metric name, optionally followed by a brace matcher group.
+
+    Returns:
+        The bare metric family name.
+    """
+    return selector.split("{", 1)[0].strip()
+
+
+def _metric_family_candidates(name: str) -> list[str]:
+    """The names a contract entry could use to account for an emitted series.
+
+    One histogram instrument reaches Prometheus as three names. Folding the
+    ``_bucket``/``_count``/``_sum`` suffix back off gives the base family, so a
+    contract entry or a regex written for the family covers all three.
+
+    The unfolded name is always kept as a candidate too, and a name is
+    accounted for if *any* candidate matches. That is what makes folding
+    ``_count`` safe even though a plain gauge can legitimately end in it
+    (``jobq_job_count`` does): the gauge matches on its own full name, and the
+    extra ``jobq_job`` candidate can only ever add coverage, never remove it.
+
+    Args:
+        name: Emitted Prometheus metric family name.
+
+    Returns:
+        ``[name]``, plus the base family if ``name`` carries a histogram suffix.
+    """
+    candidates = [name]
+    for suffix in HISTOGRAM_SUFFIXES:
+        if name.endswith(suffix) and len(name) > len(suffix):
+            candidates.append(name[: -len(suffix)])
+            break
+    return candidates
+
+
+def _accounted_metric_names(
+    expected: dict[str, Any],
+) -> tuple[set[str], list[re.Pattern[str]]]:
+    """Everything expected_metrics.json accounts for, in family-name form.
+
+    Three sources, all of which count as "the contract has considered this
+    name" — which is what reverse coverage asks, not "is this name asserted":
+
+    * every selector under a group's ``metrics`` (asserted),
+    * every key of ``not_asserted.metrics_excluded`` (deliberately unasserted
+      with a written reason),
+    * every regex under the top-level ``accounted_patterns`` (bulk families).
+
+    Literal histogram entries are expanded to the whole triple plus the base
+    family, so listing only ``foo_bucket`` still accounts for ``foo_count`` and
+    ``foo_sum``. The expansion is keyed off ``_bucket``/``_sum`` only, never off
+    ``_count`` alone, so a gauge named ``..._count`` does not silently claim a
+    shorter base family that nothing emits.
+
+    Args:
+        expected: The parsed expected_metrics.json contract.
+
+    Returns:
+        A (literal family names, compiled patterns) pair.
+    """
+    literals: set[str] = set()
+    for group in expected.values():
+        if not isinstance(group, dict):
+            continue
+        for selector in group.get("metrics", []):
+            literals.add(_selector_metric_name(selector))
+        # not_asserted records its entries under metrics_excluded, keyed by name.
+        literals.update(group.get("metrics_excluded", {}))
+
+    for name in list(literals):
+        for suffix in ("_bucket", "_sum"):
+            if name.endswith(suffix) and len(name) > len(suffix):
+                base = name[: -len(suffix)]
+                literals.add(base)
+                literals.update(base + s for s in HISTOGRAM_SUFFIXES)
+                break
+
+    patterns = [
+        re.compile(entry["pattern"])
+        for entry in expected.get("accounted_patterns", [])
+        if entry.get("pattern")
+    ]
+    return literals, patterns
+
+
+def _unaccounted_metric_names(
+    emitted: list[str], expected: dict[str, Any]
+) -> list[str]:
+    """Metric families Prometheus holds that the contract never mentions.
+
+    Args:
+        emitted:  Metric family names from the Prometheus __name__ label.
+        expected: The parsed expected_metrics.json contract.
+
+    Returns:
+        Sorted list of emitted family names nothing in the contract accounts for.
+    """
+    literals, patterns = _accounted_metric_names(expected)
+    unaccounted = []
+    for name in emitted:
+        candidates = _metric_family_candidates(name)
+        accounted = any(
+            candidate in literals
+            or any(pattern.fullmatch(candidate) for pattern in patterns)
+            for candidate in candidates
+        )
+        if not accounted:
+            unaccounted.append(name)
+    return sorted(unaccounted)
 
 
 def _metric_check_targets(
@@ -703,11 +969,20 @@ def _metric_check_targets(
         that declares it, not just spanmetrics. Nothing read the key at all
         until this was added, so the four labels the spanmetrics group
         documented as required had never actually been checked.
+
+    Note:
+        A group is any top-level value that is an object. The contract also
+        carries a string ("description"), a list ("accounted_patterns") and an
+        object that declares no metrics ("grafana_dashboards"); the isinstance
+        test skips the first two structurally rather than by name, so adding
+        another non-group key cannot break this function, and the third
+        contributes nothing because it has no "metrics" key. This selects
+        exactly the same groups the previous name-based exclusion list did.
     """
     groups = [
         (category_key, category_data)
         for category_key, category_data in expected.items()
-        if category_key not in ("description", "grafana_dashboards")
+        if isinstance(category_data, dict)
     ]
     targets = [
         (category_key, metric_name)
@@ -741,7 +1016,7 @@ async def validate_metrics(
     """
     logger.info("--- Metric Validation (Prometheus) ---")
 
-    await _log_prometheus_metric_names(session, prometheus_url)
+    emitted_metric_names = await _log_prometheus_metric_names(session, prometheus_url)
 
     with open(EXPECTED_METRICS_FILE) as f:
         expected = json.load(f)
@@ -789,6 +1064,25 @@ async def validate_metrics(
     for check in [*metric_checks, *label_checks]:
         report.add(check)
 
+    # Reverse direction: metric families Prometheus holds that the contract
+    # never mentions. Added last so no existing check's position moves.
+    unaccounted_metrics = _unaccounted_metric_names(emitted_metric_names, expected)
+    if unaccounted_metrics:
+        _log_name_list(
+            "Metric families emitted but NOT accounted for by expected_metrics.json "
+            "(warning only, does not fail CI)",
+            unaccounted_metrics,
+        )
+    report.add(
+        _reverse_coverage_result(
+            check_name="metric.reverse_coverage",
+            category="metric",
+            noun="metric families",
+            emitted_count=len(emitted_metric_names),
+            unaccounted=unaccounted_metrics,
+        )
+    )
+
 
 def _selector_with_label(metric_selector: str, label: str) -> str:
     """Add a "label is present and non-empty" matcher to a metric selector.
diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md
index 4933d36101..10d5796825 100644
--- a/docs/telemetry-runbook.md
+++ b/docs/telemetry-runbook.md
@@ -3621,13 +3621,14 @@ The counts are not hard-coded in the validator — it iterates the inventory fil
 so those files are authoritative. The figures below are the inventory as it
 stands today.
 
-| Category   | Checks                                                                                                                                                                               | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
-| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| Spans      | Every **required** entry in `expected_spans.json` — 41 span types at the time of writing: 26 required, 15 marked `"optional": true`                                                  | Span name found in Tempo carrying its `required_attributes`, plus the declared parent-child relationships. An `"optional": true` entry that does not fire is recorded as a skip, not a failure — it needs traffic the harness may not generate (HTTP/JSON-RPC client, gRPC client, missing-ledger fetch, mode transitions).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
-| Metrics    | Every entry in every asserted category of `expected_metrics.json` — 84 checks across 25 asserting categories at the time of writing: 79 metric names plus 5 `required_labels` checks | SpanMetrics, `beast::insight` gauges/counters exported over OTLP, and the `MetricsRegistry` OTLP metrics. Each must have > 0 Prometheus series; none are optional. A category may also declare `required_labels`, and each label there becomes one additional check that at least one of that category's series carries it with a non-empty value (matched as `<label>!=""`, because Prometheus cannot distinguish an absent label from an empty one). Those labels were declared but never actually read until the check was generalised, so they were documented as required while going unverified; `spanmetrics` contributes 4 and `job_queue` 1. The separate `not_asserted` group lists metrics deliberately left out of the gate because they are workload-gated or defect-gated; it has neither a `metrics` nor a `required_labels` key, so the validator skips it entirely. |
-| Logs       | 2 checks                                                                                                                                                                             | `trace_id`/`span_id` present in Loki, and a Tempo trace id resolves in Loki. Skipped in CI, which runs `--skip-loki`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
-| Parity     | 10 checks                                                                                                                                                                            | 6 span attributes the external-parity dashboard panels read, plus 4 metric value-sanity bounds.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
-| Dashboards | Every uid in `expected_metrics.json` under `grafana_dashboards.uids` — currently all 15 provisioned dashboards                                                                       | Each listed dashboard loads and reports a panel count. This is a provisioning check only: it does **not** execute the panels' queries, so a dashboard can pass while individual panels render empty. `log-derived-insights` is Loki-backed, so under `--skip-loki` only its provisioning is meaningfully covered.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
+| Category         | Checks                                                                                                                                                                               | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
+| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Spans            | Every **required** entry in `expected_spans.json` — 41 span types at the time of writing: 26 required, 15 marked `"optional": true`                                                  | Span name found in Tempo carrying its `required_attributes`, plus the declared parent-child relationships. An `"optional": true` entry that does not fire is recorded as a skip, not a failure — it needs traffic the harness may not generate (HTTP/JSON-RPC client, gRPC client, missing-ledger fetch, mode transitions).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
+| Metrics          | Every entry in every asserted category of `expected_metrics.json` — 84 checks across 25 asserting categories at the time of writing: 79 metric names plus 5 `required_labels` checks | SpanMetrics, `beast::insight` gauges/counters exported over OTLP, and the `MetricsRegistry` OTLP metrics. Each must have > 0 Prometheus series; none are optional. A category may also declare `required_labels`, and each label there becomes one additional check that at least one of that category's series carries it with a non-empty value (matched as `<label>!=""`, because Prometheus cannot distinguish an absent label from an empty one). Those labels were declared but never actually read until the check was generalised, so they were documented as required while going unverified; `spanmetrics` contributes 4 and `job_queue` 1. The separate `not_asserted` group lists metrics deliberately left out of the gate because they are workload-gated or defect-gated; it has neither a `metrics` nor a `required_labels` key, so the validator skips it entirely.                                                                                                                                                                                                                                                           |
+| Logs             | 2 checks                                                                                                                                                                             | `trace_id`/`span_id` present in Loki, and a Tempo trace id resolves in Loki. Skipped in CI, which runs `--skip-loki`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
+| Parity           | 10 checks                                                                                                                                                                            | 6 span attributes the external-parity dashboard panels read, plus 4 metric value-sanity bounds.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
+| Dashboards       | Every uid in `expected_metrics.json` under `grafana_dashboards.uids` — currently all 15 provisioned dashboards                                                                       | Each listed dashboard loads and reports a panel count. This is a provisioning check only: it does **not** execute the panels' queries, so a dashboard can pass while individual panels render empty. `log-derived-insights` is Loki-backed, so under `--skip-loki` only its provisioning is meaningfully covered.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
+| Reverse coverage | 2 checks — `metric.reverse_coverage` and `span.reverse_coverage`                                                                                                                     | The only checks that run in the opposite direction: they read the full emitted inventory (the Prometheus `__name__` label values, the Tempo `span.name` tag values) and name everything the contract never mentions, sorted and one per line in the log. **Warn only — `passed` is hardcoded `True` in `_reverse_coverage_result`, so these can never fail CI.** Downstream branches legitimately add telemetry an upstream contract has not seen, and a hard failure would redden all of them. A metric family is accounted for by a `metrics` entry, by a `not_asserted.metrics_excluded` key, or by an anchored regex under the top-level `accounted_patterns` list — which exists for families whose membership is derived mechanically from a table in the code (the per-job-type job-queue instruments, the overlay per-category traffic cross product) plus the Prometheus scrape plumbing that is not xrpld telemetry. Histogram `_bucket`/`_count`/`_sum` names fold onto their base family before matching. Spans need no pattern list: the check reuses the forward matcher, so `rpc.command.*` covers every command it expands to. |
 
 ### Running Individual Tools
 

From 2250bdfe526b213879c673fe2992202a2cd3f915 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Tue, 25 Aug 2026 16:19:15 +0100
Subject: [PATCH 35/38] ci(codecov): move the telemetry ignore block to the
 branch that adds telemetry
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The block was added on the StatsD branch, but telemetry sources start here.
Codecov config only flows child-ward, so every branch between this one and
that one kept reporting telemetry files as uncovered patch lines.

Also corrects the rationale. The old comment said telemetry is "not enabled
in coverage builds"; it is enabled — conanfile.py and CMakeLists.txt both
default it ON, and the coverage matrix leg does not turn it off. The reason
these files record no coverage is that the unit-test suite never starts an
exporter.

Adds *SpanLabels.h alongside *SpanNames.h: same compile-time-constant
category, and the existing glob did not match it.

This narrows the patch gap but does not close it — instrumentation added to
consensus and overlay files is still counted, so codecov/patch stays red on
the early phases.
---
 .codecov.yml | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/.codecov.yml b/.codecov.yml
index 4268758e44..f4f812ddf5 100644
--- a/.codecov.yml
+++ b/.codecov.yml
@@ -58,3 +58,15 @@ ignore:
   - "src/tests/"
   - "include/xrpl/beast/test/"
   - "include/xrpl/beast/unit_test/"
+  # Telemetry modules. Telemetry compiles in by default (conanfile.py,
+  # CMakeLists.txt), but the unit-test suite never starts an exporter, so these
+  # files record no coverage. This block belongs on the earliest branch that
+  # adds telemetry code — codecov config only flows child-ward, so an ignore
+  # added on a later branch can never cover the branches before it.
+  - "src/xrpld/telemetry/"
+  - "src/libxrpl/telemetry/"
+  - "include/xrpl/telemetry/"
+  # Per-module span-name and span-label constant headers: compile-time
+  # constants only, colocated with their subsystem rather than under telemetry/.
+  - "**/*SpanNames.h"
+  - "**/*SpanLabels.h"

From 59a0595a6ecd6d55a0b50b1bac9bd3f4158ef038 Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Tue, 25 Aug 2026 16:31:23 +0100
Subject: [PATCH 36/38] fix(telemetry): stop the workload harness issuing
 refused path-finding RPC

Every node the harness starts is a validator, and validators disable
pathfinding: Config.cpp:725-726 zeroes pathSearchMax whenever a
[validation_seed] or [validator_token] section is present, and
run-full-validation.sh writes [validation_seed] into every generated node
cfg (:308) with no [path_search] section to put the default back. So
doRipplePathFind refused every call at RipplePathFind.cpp:48-49 and the
3% ripple_path_find weight bought no coverage at all.

It was not free either. The pathfind.request guard is constructed at
RipplePathFind.cpp:35, above that refusal, so each refused call still
exported a span, and the enclosing rpc.command.ripple_path_find span
carried rpc_status=error. That put a steady 3% error floor into
span_calls_total for STATUS_CODE_ERROR: any error-rate threshold derived
from harness data before this change was measuring the harness rather
than xrpld, and needs re-deriving.

Removing the load makes pathfind.request unreachable, so it moves from
required to optional in expected_spans.json; without that the span check
would fail on every run. Three notes in that file and three in
expected_metrics.json made claims that are now false, two of them citing
line numbers this commit deletes; all six are corrected. The runbook
required/optional count moves 26/15 to 25/16.

Two facts a future reader needs.

First, the weights previously summed to 103, not 100, so every percentage
the docstring stated was wrong: health checks were really 38.8%, not 40%.
Dropping the 3 makes the sum exactly 100 and every stated percentage
correct for the first time. expected_spans.json also carried live
arithmetic off the old total, "25/103 ... roughly 43%", now 25/100 and
42%.

Second, baselines/baseline-timings.json was captured WITH this load. Only
span.rpc.ws_message p50/p95/p99 of the 25 gated keys sees the RPC mix,
and their trip points sit 3.1x to 5.9x above baseline, so the gate will
not fire. But a timing baseline is workload-specific and its profile
field still reads full-validation, so nothing will flag the drift:
refresh it from the next CI run's timings artifact.

Pathfinding now has no coverage in this harness at all. The workload
README section "Pathfinding is not exercised" records that cost, the
manual verification route, and a four-step restore recipe in which steps
1 and 2 alone only reinstate the error floor.
---
 docker/telemetry/workload/README.md           | 23 ++++++++++++++++++-
 .../telemetry/workload/expected_metrics.json  |  6 ++---
 docker/telemetry/workload/expected_spans.json | 11 +++++----
 .../telemetry/workload/rpc_load_generator.py  | 20 ++++++----------
 docs/telemetry-runbook.md                     |  2 +-
 5 files changed, 39 insertions(+), 23 deletions(-)

diff --git a/docker/telemetry/workload/README.md b/docker/telemetry/workload/README.md
index e866e49b2f..80af47300e 100644
--- a/docker/telemetry/workload/README.md
+++ b/docker/telemetry/workload/README.md
@@ -213,7 +213,7 @@ python3 tx_submitter.py --endpoint ws://localhost:6006 \
 
 Automated validation that all expected telemetry data exists. Every metric in `expected_metrics.json` is required — if it doesn't fire, the validation fails. Spans are required unless the entry carries `"optional": true`.
 
-- **Span validation**: All span types from `expected_spans.json` with required attributes and parent-child hierarchies. Entries marked `"optional": true` only fire under traffic the harness may not produce (HTTP/JSON-RPC client, gRPC client, missing-ledger fetch, mode transitions); their absence is recorded as a passing skip, not a failure.
+- **Span validation**: All span types from `expected_spans.json` with required attributes and parent-child hierarchies. Entries marked `"optional": true` only fire under traffic the harness may not produce (HTTP/JSON-RPC client, gRPC client, path-finding RPC — see [Pathfinding is not exercised](#pathfinding-is-not-exercised) — missing-ledger fetch, mode transitions); their absence is recorded as a passing skip, not a failure.
 - **Metric validation**: All metrics from `expected_metrics.json` — SpanMetrics, `beast::insight` gauges/counters/histograms, `MetricsRegistry` OTLP metrics. Every listed metric must have > 0 series. Uses the Prometheus `/api/v1/series` endpoint (not instant queries), polled until the metric appears or the poll window elapses, so a late-populating or quiet series is not a false negative.
 - **Log-trace correlation**: trace_id/span_id in Loki logs (requires Loki). The two checks are `log.trace_id_present` and `log.trace_id_cross_reference`, and they exist only when `--skip-loki` is **not** passed — `run_validation()` builds them inside an `if not skip_loki` branch, so with the flag they are absent from the report rather than reported as skipped. **CI always passes `--skip-loki`, so these two are never exercised there** — see [CI Integration](#ci-integration).
 - **Dashboard validation**: Every dashboard uid listed under `grafana_dashboards.uids` in `expected_metrics.json` loads with panels. That list currently covers **all 15** dashboards provisioned in `docker/telemetry/grafana/dashboards/`. Note the scope of this check: it asks the Grafana API whether the dashboard exists and returns a panel count — it does **not** run the panels' queries, so a dashboard can pass here while individual panels render empty.
@@ -422,6 +422,27 @@ Do that after any change to log formatting, span activation, the collector's
 would make CI exercise Loki ingestion and filelog mounting for the first time, so
 it is held back as its own change rather than folded into an unrelated push.
 
+### Pathfinding is not exercised
+
+`rpc_load_generator.py` stopped issuing `ripple_path_find` on 2026-08-25 — the weight, the request-builder branch and the docstring line went together.
+
+**Why.** Pathfinding is disabled on every node this harness starts, so those calls could only ever fail:
+
+- `src/xrpld/core/detail/Config.cpp:725-726` sets `pathSearchMax = 0` whenever a `[validation_seed]` or `[validator_token]` section is present — "by default, validators don't have pathfinding enabled".
+- `run-full-validation.sh:308` writes `[validation_seed]` into every generated node cfg, and that script carries no `[path_search]`, `[path_search_fast]` or `[path_search_max]` section to put the default back.
+- `src/xrpld/rpc/handlers/orderbook/RipplePathFind.cpp:48-49` therefore returns `rpcNOT_SUPPORTED`; `PathFind.cpp:39` does the same for `path_find`.
+
+**What removing it fixes.** The refusals were not silent. `pathfind.request` is opened at `RipplePathFind.cpp:35`, **above** that guard, so every refused call still exported a span, and the enclosing `rpc.command.ripple_path_find` span carried `rpc_status=error`. At a 3% weight that manufactured a steady ~3% error floor in `span_calls_total{status_code="STATUS_CODE_ERROR"}`. **Any error-rate threshold derived from harness data before this change was measuring the harness, not xrpld** — re-derive it.
+
+**What it costs.** Pathfinding now has no coverage here at all. Four spans (`pathfind.request`, `.compute`, `.discover`, `.update_all`) and two histograms (`pathfind_fast_milliseconds`, `pathfind_full_milliseconds`) go unexercised, and `pathfind.request` moved from required to `"optional": true` in `expected_spans.json` for that reason. Until the load returns, verify pathfinding by hand: the **PathFind** row of [`../TESTING.md`](../TESTING.md) carries a `curl` recipe, and `../xrpld-telemetry.cfg` is a non-validator config that already enables pathfinding.
+
+**Putting it back.** All four steps are required. The first two alone just restore the error floor:
+
+1. Add a `[path_search_max]` section to the node cfg `run-full-validation.sh` generates — or drop `[validation_seed]` and run a non-validator node. The `[path_search*]` block in `../xrpld-telemetry.cfg` is a working example.
+2. Restore the `ripple_path_find` weight in `DEFAULT_WEIGHTS` and its branch in `build_rpc_request()`. `path_find` is a streaming subscription and needs its own phase instead — the generator is strictly one request, one reply.
+3. Set `pathfind.request` back to required in `expected_spans.json`. Step 1 also makes `pathfind.compute` reachable, so the `pathfind.request -> pathfind.compute` relationship can lose its `"skip": true`.
+4. **Re-capture `baselines/baseline-timings.json`.** Restoring the load changes the RPC mix, and `span.rpc.ws_message.{p50,p95,p99}` is a gated key — a baseline captured under a different mix is stale. See [OTel Timings Regression Gate](#otel-timings-regression-gate).
+
 ## Configuration Files
 
 | File                              | Purpose                                                                                                                       |
diff --git a/docker/telemetry/workload/expected_metrics.json b/docker/telemetry/workload/expected_metrics.json
index f1cf8b1cd6..2ffebe7fb3 100644
--- a/docker/telemetry/workload/expected_metrics.json
+++ b/docker/telemetry/workload/expected_metrics.json
@@ -199,7 +199,7 @@
   "not_asserted": {
     "description": "Emitted-and-dashboarded metrics deliberately left unasserted because they are workload-gated or defect-gated: the harness workload cannot guarantee they appear, and a check that fails on a healthy run is worse than no check. This group has no \"metrics\" key, so validate_telemetry.py skips it (validate_metrics iterates category_data.get(\"metrics\", [])). Promote an entry into an asserted group only after the workload is changed to guarantee it.",
     "metrics_excluded": {
-      "rpc_method_errored_total": "MetricsRegistry.cpp:332-333, push counter. 'Errored' here means a thrown C++ exception, not an error status in the JSON reply: the only caller is PerfLogImp.cpp:409 under 'if (!finish)', reached only through PerfLogImp::rpcError (PerfLogImp.h:150-153), whose only call site is the catch (std::exception&) handler in RPCHandler.cpp:213. An RPC that returns an error status normally still takes the rpcFinish path at RPCHandler.cpp:190 and increments rpc_method_finished_total. That distinction matters here, because the generator's ripple_path_find calls are in fact refused — pathfinding is off on every harness node, so doRipplePathFind returns rpcNOT_SUPPORTED (see pathfind_fast_milliseconds below) — and it would be easy to conclude from that alone that this counter must fire. It does not: a refusal is a normal return, not a throw. Nothing in rpc_load_generator.py's server_info / fee / ledger / ripple_path_find mix is expected to throw, so no series may ever be created.",
+      "rpc_method_errored_total": "MetricsRegistry.cpp:332-333, push counter. 'Errored' here means a thrown C++ exception, not an error status in the JSON reply: the only caller is PerfLogImp.cpp:409 under 'if (!finish)', reached only through PerfLogImp::rpcError (PerfLogImp.h:150-153), whose only call site is the catch (std::exception&) handler in RPCHandler.cpp:213. An RPC that returns an error status normally still takes the rpcFinish path at RPCHandler.cpp:190 and increments rpc_method_finished_total. That distinction mattered here while the generator still issued ripple_path_find: those calls were in fact refused — pathfinding is off on every harness node, so doRipplePathFind returns rpcNOT_SUPPORTED (see pathfind_fast_milliseconds below) — and it would have been easy to conclude from that alone that this counter must fire. It did not, because a refusal is a normal return, not a throw. The load was removed on 2026-08-25, so the harness no longer issues that command at all and the question is moot, but the distinction is kept on record because it is the one that decides this entry. Nothing in rpc_load_generator.py's remaining server_info / fee / account / ledger / tx / DEX mix is expected to throw either, so no series may ever be created.",
       "ledger_history_mismatch_total": "MetricsRegistry.cpp:377, incremented only from LedgerHistory.cpp:332 on a built-vs-validated ledger mismatch. On a healthy run it never fires — asserting it would mean asserting a defect.",
       "txq_expired_total": "MetricsRegistry.cpp:379, incremented only at TxQ.cpp:1428 when a queued tx expires past its LastLedgerSequence. CI does run a txq-burst phase (workload-profiles.json:41, 30 s of single-type Payment at 60 TPS), but that does not guarantee sustained fee escalation followed by expiry: a run in which every other check passed still exposed only txq_metrics and no txq_expired_total.",
       "txq_dropped_total": "MetricsRegistry.cpp:381, incremented only at TxQ.cpp:1302 / :1347 on queue-full admission refusal. Same reason as txq_expired_total.",
@@ -210,8 +210,8 @@
       "getobject_charge": "GetObjectMetricNames.h:105, PeerImp.cpp:2931. Same gate.",
       "rpc_size_bytes": "ServerHandler.cpp:191, group('rpc')->makeEvent('size', Unit::Bytes). The OTLP Prometheus exporter derives the metric-name suffix from the declared unit, so a byte unit yields rpc_size_bytes. The Unit::Bytes declaration itself landed earlier, in 76c9051203; what 24094e427b changed was the exporter finally consuming it, replacing a hardcoded CreateDoubleHistogram(name, 'Duration in ms', 'ms') with otelUnitDescription(unit)/otelUnitCode(unit), and that is what renamed the series off rpc_size_milliseconds and the millisecond bucket ladder. Neither name was ever recorded here, so the harness could confirm neither the rename nor a regression back onto that ladder. Notified from ServerHandler::processRequest:1133, the HTTP JSON-RPC path — it computes an HTTP status and appends a trailing newline — and the load generators are WebSocket-only, the same gate regression-metrics.json:4 records for rpc.process, so only the harness's handful of HTTP health polls reach it. Real coverage needs an HTTP JSON-RPC phase in rpc_load_generator.py; that is a workload change rather than a harness correction, and is deliberately out of scope here.",
       "rpc_time_milliseconds": "ServerHandler.cpp:192, group('rpc')->makeEvent('time') with the default millisecond unit. Notified from ServerHandler::processRequest:1129, the same HTTP JSON-RPC call site as rpc_size_bytes and behind the same WebSocket-only gate.",
-      "pathfind_fast_milliseconds": "PathRequestManager.h:35, makeEvent('pathfind_fast') with the default millisecond unit, so the exported form is the pathfind_fast_milliseconds_bucket/_count/_sum triple and there is no bare series — the same convention io_latency and rpc_method_us follow, and rpc-pathfinding queries the _bucket. THE OPERATIVE BLOCKER IS THE CONFIG, NOT THE CALL GRAPH: pathfinding is disabled outright on every harness node, so no PathRequest is ever constructed and no pathfind_* histogram can exist. Config.cpp:725-726 sets pathSearchMax to 0 whenever a [validation_seed] or [validator_token] section is present ('By default, validators don't have pathfinding enabled'); run-full-validation.sh writes [validation_seed] into every generated node cfg (:308) and contains no [path_search], [path_search_fast] or [path_search_max] section to put it back (grep count 0 for path_search in that file — the only [path_search*] sections in docker/telemetry/ are in xrpld-telemetry.cfg and xrpld-telemetry-mainnet.cfg, neither of which the harness uses); and doRipplePathFind returns rpcNOT_SUPPORTED at RipplePathFind.cpp:48-49, before context.loadType is set and before any branch on the ledger parameter. So every ripple_path_find call the generator makes (3% weight, rpc_load_generator.py:74) is refused at the front door. Read that first: the structural argument below is correct and matters if pathfinding is ever enabled, but it is not why the metric is missing today. STRUCTURAL ARGUMENT (verified, applies once pathSearchMax is non-zero): reportFast's only caller is PathRequest.cpp:852, inside the 'if (fast && quickReply_ == {})' branch of PathRequest::doUpdate. The only doUpdate call that passes fast=true is in PathRequest::doCreate (PathRequest.cpp:259), guarded by '!hasCompletion()'. Both ripple_path_find entry points construct the PathRequest with a completion function, so hasCompletion() (:161-164) is true and the fast pass is skipped: with no ledger specified doRipplePathFind goes to makeLegacyPathRequest, which passes the coroutine-post lambda (RipplePathFind.cpp:140-160); with a ledger specified it goes to doLegacyPathRequest, which passes an empty-body but non-null lambda (PathRequestManager.cpp:317). Only the path_find streaming subscription reaches reportFast, because makePathRequest builds the request from a subscriber with no completion (PathRequestManager.cpp:261). The load generator does not use path_find and says so at rpc_load_generator.py:72-73. Covering this metric therefore needs both a [path_search_max] override (or a non-validator node) in run-full-validation.sh and a path_find subscription phase in the generator. Both are harness/workload changes and out of scope here. Grafana Cloud shows zero series in 180 days on the devnet nodes, consistent with those nodes receiving no pathfinding RPC.",
-      "pathfind_full_milliseconds": "PathRequestManager.h:36, makeEvent('pathfind_full'); same histogram naming as pathfind_fast_milliseconds above. Notified from reportFull (:87-90) via PathRequest.cpp:857, the 'else if (!fast && fullReply_ == {})' branch. Blocked by exactly the same config gate as pathfind_fast_milliseconds, and the probability of emission under the harness workload is zero, not low: pathSearchMax is 0 on every harness node (Config.cpp:725-726 plus the [validation_seed] section at run-full-validation.sh:308, with no [path_search*] override anywhere in that file), so doRipplePathFind returns rpcNOT_SUPPORTED at RipplePathFind.cpp:48-49 and no PathRequest object is ever constructed for reportFull to fire from. An earlier revision of this entry described the path as reachable-but-probabilistic — emitting one ledger close behind the request via PathRequestManager::updateAll's one-shot branch (PathRequestManager.cpp:160-166) — and prescribed sending an explicit ledger_index from the generator so doLegacyPathRequest would call doUpdate(cache, false) synchronously (PathRequestManager.cpp:321). Both halves were wrong. The probability is zero rather than merely unreliable, and the prescribed remedy cannot work at all, because the pathSearchMax guard fires before the ledger parameter is read: adding ledger_index changes nothing while pathfinding is off. Covering this metric needs a [path_search_max] override (or a non-validator node) in run-full-validation.sh — a harness-topology change, out of scope here. Grafana Cloud shows zero series in 180 days on the devnet nodes, consistent with those nodes receiving no pathfinding RPC rather than with a broken exporter.",
+      "pathfind_fast_milliseconds": "PathRequestManager.h:35, makeEvent('pathfind_fast') with the default millisecond unit, so the exported form is the pathfind_fast_milliseconds_bucket/_count/_sum triple and there is no bare series — the same convention io_latency and rpc_method_us follow, and rpc-pathfinding queries the _bucket. THE OPERATIVE BLOCKER IS THE CONFIG, NOT THE CALL GRAPH: pathfinding is disabled outright on every harness node, so no PathRequest is ever constructed and no pathfind_* histogram can exist. Config.cpp:725-726 sets pathSearchMax to 0 whenever a [validation_seed] or [validator_token] section is present ('By default, validators don't have pathfinding enabled'); run-full-validation.sh writes [validation_seed] into every generated node cfg (:308) and contains no [path_search], [path_search_fast] or [path_search_max] section to put it back (grep count 0 for path_search in that file — the only [path_search*] sections in docker/telemetry/ are in xrpld-telemetry.cfg and xrpld-telemetry-mainnet.cfg, neither of which the harness uses); and doRipplePathFind returns rpcNOT_SUPPORTED at RipplePathFind.cpp:48-49, before context.loadType is set and before any branch on the ledger parameter. That config gate is why the metric could never appear even while the generator was issuing the command (a 3% ripple_path_find weight, removed on 2026-08-25 precisely because every one of those calls was refused at the front door); the harness now issues no path-finding RPC at all, so there are two independent reasons. Read that first: the structural argument below is correct and matters if pathfinding is ever enabled, but it is not why the metric is missing today. STRUCTURAL ARGUMENT (verified, applies once pathSearchMax is non-zero): reportFast's only caller is PathRequest.cpp:852, inside the 'if (fast && quickReply_ == {})' branch of PathRequest::doUpdate. The only doUpdate call that passes fast=true is in PathRequest::doCreate (PathRequest.cpp:259), guarded by '!hasCompletion()'. Both ripple_path_find entry points construct the PathRequest with a completion function, so hasCompletion() (:161-164) is true and the fast pass is skipped: with no ledger specified doRipplePathFind goes to makeLegacyPathRequest, which passes the coroutine-post lambda (RipplePathFind.cpp:140-160); with a ledger specified it goes to doLegacyPathRequest, which passes an empty-body but non-null lambda (PathRequestManager.cpp:317). Only the path_find streaming subscription reaches reportFast, because makePathRequest builds the request from a subscriber with no completion (PathRequestManager.cpp:261). The load generator has never used path_find: it fires one request per send and awaits one reply, which a streaming subscription does not fit. Covering this metric therefore needs both a [path_search_max] override (or a non-validator node) in run-full-validation.sh and a path_find subscription phase in the generator. Both are harness/workload changes and out of scope here. Grafana Cloud shows zero series in 180 days on the devnet nodes, consistent with those nodes receiving no pathfinding RPC.",
+      "pathfind_full_milliseconds": "PathRequestManager.h:36, makeEvent('pathfind_full'); same histogram naming as pathfind_fast_milliseconds above. Notified from reportFull (:87-90) via PathRequest.cpp:857, the 'else if (!fast && fullReply_ == {})' branch. Blocked by exactly the same config gate as pathfind_fast_milliseconds, and the probability of emission under the harness workload is zero, not low: pathSearchMax is 0 on every harness node (Config.cpp:725-726 plus the [validation_seed] section at run-full-validation.sh:308, with no [path_search*] override anywhere in that file), so doRipplePathFind returns rpcNOT_SUPPORTED at RipplePathFind.cpp:48-49 and no PathRequest object is ever constructed for reportFull to fire from. An earlier revision of this entry described the path as reachable-but-probabilistic — emitting one ledger close behind the request via PathRequestManager::updateAll's one-shot branch (PathRequestManager.cpp:160-166) — and prescribed sending an explicit ledger_index from the generator so doLegacyPathRequest would call doUpdate(cache, false) synchronously (PathRequestManager.cpp:321). Both halves were wrong. The probability is zero rather than merely unreliable, and the prescribed remedy cannot work at all, because the pathSearchMax guard fires before the ledger parameter is read: adding ledger_index changes nothing while pathfinding is off. Since 2026-08-25 the generator issues no path-finding RPC either, so covering this metric needs a [path_search_max] override (or a non-validator node) in run-full-validation.sh AND the load restored — a harness-topology plus workload change, out of scope here. The workload README section 'Pathfinding is not exercised' holds the recipe. Grafana Cloud shows zero series in 180 days on the devnet nodes, consistent with those nodes receiving no pathfinding RPC rather than with a broken exporter.",
       "warn_total": "include/xrpl/resource/detail/Logic.h:41, makeMeter('warn'). makeMeter maps to CreateUInt64Counter (OTelCollector.cpp:878-881 -> :773-777), so the Prometheus exporter appends _total; the meter is created on the bare collector with no group, hence the unprefixed name. Incremented only at Logic.h:481, inside the 'if (notify)' branch reached when a consumer's balance crosses kWarningThreshold. A cooperating 5-node cluster plus a rate-limited load generator never charges a consumer that far, and Grafana Cloud confirms zero series in 180 days. Recorded explicitly because this was briefly mis-diagnosed as a phantom metric: the rpc-pathfinding panel that queries it is correct, and renders empty only because the condition has not occurred.",
       "drop_total": "include/xrpl/resource/detail/Logic.h:42, makeMeter('drop'); same CreateUInt64Counter mapping and same _total suffix as warn_total. Incremented only at Logic.h:505, when a consumer's balance is at or above kDropThreshold and the connection is dropped. Grafana Cloud shows 2 live series, so unlike warn_total this one does fire in the wild — but only on a genuinely abusive consumer, which the harness deliberately does not create, so it is condition-gated all the same. Its rpc-pathfinding panel is likewise correct rather than phantom.",
       "jobq_*_milliseconds, jobq_*_q_milliseconds": "This key is a pattern rather than a literal metric name — unlike every other entry in this map it stands for a whole family, one pair per job type. Created per job type in JobTypeData.h:97-98 from info.name() and info.name() + kSuffixQueued ('_q'), so the exported names are jobq_<jobtype>_milliseconds and jobq_<jobtype>_q_milliseconds with the job type lowercased by formatName. Which job types appear depends on which jobs a run happens to schedule, so no individual name is guaranteed. They are also rounded up to a whole millisecond at source (Event.h:47-51 applies ceil to a millisecond value type), which is why 6e2b2da772 moved the ledger-data-sync q-wait panels off jobq_<jobtype>_q_milliseconds_bucket onto job_queued_us_bucket — they are poor assertion targets regardless."
diff --git a/docker/telemetry/workload/expected_spans.json b/docker/telemetry/workload/expected_spans.json
index bb1e68d734..346746ed5d 100644
--- a/docker/telemetry/workload/expected_spans.json
+++ b/docker/telemetry/workload/expected_spans.json
@@ -1,5 +1,5 @@
 {
-  "description": "Expected span inventory for xrpld telemetry validation. Attribute keys follow the 2026-05-13 span-attr naming redesign (bare/underscore form; dotted xrpl.* reserved for resource attributes). Sourced from the *SpanNames.h headers and verified against the emitting call sites. Spans marked \"optional\": true are conditional — they only fire under traffic the harness may not produce (e.g. gRPC client, missing-ledger fetch, mode transitions) and are not failed when absent. \"parent\" is documentation only (validate_telemetry.py asserts hierarchy from parent_child_relationships, not from this field) and records the parent as the code actually produces it: null means the span is a root or an explicit freshRoot. required_attributes lists only attributes set on EVERY code path that creates the span — attributes set after an early return are described in the span's note instead, because _validate_span_attributes_otlp samples a single trace and would fail on a legitimate short-circuit path. total_unique_attributes is the size of the union of all required_attributes; total_span_types is len(spans). Span EVENTS (consensus.round phase.*/outcome.*, consensus.update_positions dispute.resolve, consensus.accept.apply tx.included) are NOT represented: validate_telemetry.py reads only span name, attributes and timestamps from Tempo, so an \"events\" key would be silently ignored. They are documented in the relevant span notes until the validator gains event support.",
+  "description": "Expected span inventory for xrpld telemetry validation. Attribute keys follow the 2026-05-13 span-attr naming redesign (bare/underscore form; dotted xrpl.* reserved for resource attributes). Sourced from the *SpanNames.h headers and verified against the emitting call sites. Spans marked \"optional\": true are conditional — they only fire under traffic the harness may not produce (e.g. gRPC client, path-finding RPC, missing-ledger fetch, mode transitions) and are not failed when absent. \"parent\" is documentation only (validate_telemetry.py asserts hierarchy from parent_child_relationships, not from this field) and records the parent as the code actually produces it: null means the span is a root or an explicit freshRoot. required_attributes lists only attributes set on EVERY code path that creates the span — attributes set after an early return are described in the span's note instead, because _validate_span_attributes_otlp samples a single trace and would fail on a legitimate short-circuit path. total_unique_attributes is the size of the union of all required_attributes; total_span_types is len(spans). Span EVENTS (consensus.round phase.*/outcome.*, consensus.update_positions dispute.resolve, consensus.accept.apply tx.included) are NOT represented: validate_telemetry.py reads only span name, attributes and timestamps from Tempo, so an \"events\" key would be silently ignored. They are documented in the relevant span notes until the validator gains event support.",
   "spans": [
     {
       "name": "rpc.ws_message",
@@ -343,7 +343,8 @@
         "pathfind_dest_account"
       ],
       "config_flag": "trace_rpc",
-      "note": "Fires on ripple_path_find / path_find RPC. Driven by the ripple_path_find load in rpc_load_generator.py. Created as an ambient (scoped) child inside the RPC command handler (RipplePathFind.cpp:35-36, PathFind.cpp:26-27), so its parent is the enclosing rpc.command.* span — RipplePathFind.cpp:30 states this explicitly. This span still emits on the harness even though pathfinding itself is disabled there, and the reason is purely positional: the ScopedSpanGuard is constructed at RipplePathFind.cpp:35, above the 'if (pathSearchMax == 0) return rpcError(RpcNotSupported)' guard at :48-49, so the span opens and closes around a request that is refused before any PathRequest exists. That is why this entry is asserted while pathfind.compute and pathfind.discover below are not."
+      "optional": true,
+      "note": "Fires on ripple_path_find / path_find RPC. Optional because the harness issues neither: the ripple_path_find weight was removed from rpc_load_generator.py's DEFAULT_WEIGHTS on 2026-08-25, and no workload-profiles.json phase names a path-finding command in a weights override, so no such RPC reaches a node at all. Created as an ambient (scoped) child inside the RPC command handler (RipplePathFind.cpp:35-36, PathFind.cpp:26-27), so its parent is the enclosing rpc.command.* span — RipplePathFind.cpp:30 states this explicitly. Note the span does NOT need pathfinding to be ENABLED, only the RPC to be issued: the ScopedSpanGuard is constructed at RipplePathFind.cpp:35, above the 'if (pathSearchMax == 0) return rpcError(RpcNotSupported)' guard at :48-49, so even a refused call opens and closes it. That positional accident is why the load alone used to satisfy this entry while pathfind.compute and pathfind.discover below stayed unasserted — and why those refusals were also manufacturing a steady ~3% STATUS_CODE_ERROR floor in span_calls_total, which removing the load has now cleared. Restoring the load alone would make this span required again and bring that error floor back; covering the whole family needs pathfinding actually enabled (a [path_search_max] override in run-full-validation.sh — see the pathfind.compute entry below). The workload README section 'Pathfinding is not exercised' carries the full restore recipe."
     },
     {
       "name": "pathfind.compute",
@@ -352,7 +353,7 @@
       "required_attributes": ["pathfind_fast"],
       "config_flag": "trace_rpc",
       "optional": true,
-      "note": "Created by PathRequest::doUpdate (PathRequest.cpp:749-750), which the harness never reaches: pathfinding is disabled on every harness node, so no PathRequest is ever constructed. Config.cpp:725-726 sets pathSearchMax to 0 when a [validation_seed] or [validator_token] section is present, run-full-validation.sh writes [validation_seed] for every node (:308) and has no [path_search*] override, and doRipplePathFind then returns rpcNOT_SUPPORTED at RipplePathFind.cpp:48-49 — above the request-construction branches and below the pathfind.request span guard at :35. Liquidity is not the reason and never was: the call is refused before any path search is attempted, so the outcome does not depend on what the ledger holds. Enabling this span needs a [path_search_max] override (or a non-validator node) in run-full-validation.sh."
+      "note": "Created by PathRequest::doUpdate (PathRequest.cpp:749-750), which the harness never reaches: pathfinding is disabled on every harness node, so no PathRequest is ever constructed. Config.cpp:725-726 sets pathSearchMax to 0 when a [validation_seed] or [validator_token] section is present, run-full-validation.sh writes [validation_seed] for every node (:308) and has no [path_search*] override, and doRipplePathFind then returns rpcNOT_SUPPORTED at RipplePathFind.cpp:48-49 — above the request-construction branches and below the pathfind.request span guard at :35. Liquidity is not the reason and never was: the call is refused before any path search is attempted, so the outcome does not depend on what the ledger holds. Since 2026-08-25 there is a second, independent reason: the harness sends no path-finding RPC at all, the ripple_path_find weight having been removed from rpc_load_generator.py. Enabling this span therefore needs BOTH a [path_search_max] override (or a non-validator node) in run-full-validation.sh AND the load restored — see the workload README section 'Pathfinding is not exercised'."
     },
     {
       "name": "pathfind.discover",
@@ -395,7 +396,7 @@
       "child": "rpc.command.*",
       "description": "WebSocket message contains the per-command span — the real relationship on the harness WS path (rpc::doCommand at RPCHandler.cpp:271 creates an ambient child of the rpc.ws_message scope inside the same coroutine)",
       "skip": true,
-      "skip_reason": "Code-verified real, but not assertable by the current validator. _validate_parent_child() collapses the wildcard to the single literal name via child_name.replace(\"*\", \"server_info\") and samples only the 3 most recent parent traces. Each rpc.ws_message trace carries exactly one command, and server_info is 25/103 of rpc_load_generator.py's DEFAULT_WEIGHTS, so roughly 43% of healthy runs would sample three non-server_info traces and fail. Asserting this needs the validator to accept a wildcard child as a prefix match (or to raise the trace sample size); until then the relationship is documented, not enforced."
+      "skip_reason": "Code-verified real, but not assertable by the current validator. _validate_parent_child() collapses the wildcard to the single literal name via child_name.replace(\"*\", \"server_info\") and samples only the 3 most recent parent traces. Each rpc.ws_message trace carries exactly one command, and server_info is 25/100 of rpc_load_generator.py's DEFAULT_WEIGHTS, so roughly 42% of healthy runs would sample three non-server_info traces and fail. Asserting this needs the validator to accept a wildcard child as a prefix match (or to raise the trace sample size); until then the relationship is documented, not enforced."
     },
     {
       "parent": "rpc.process",
@@ -424,7 +425,7 @@
       "child": "pathfind.compute",
       "description": "Pathfind request contains the compute sub-span",
       "skip": true,
-      "skip_reason": "Real relationship (pathfind.compute is created inside PathRequest::doUpdate at PathRequest.cpp:749-750, under the pathfind.request scope), but the child never exists on the harness because pathfinding is disabled on every node: Config.cpp:725-726 zeroes pathSearchMax whenever a [validation_seed] or [validator_token] section is present, run-full-validation.sh writes [validation_seed] for all five nodes (:308) with no [path_search*] override, and doRipplePathFind returns rpcNOT_SUPPORTED at RipplePathFind.cpp:48-49 before constructing a PathRequest. The parent still appears only because its ScopedSpanGuard is created at RipplePathFind.cpp:35, above that guard. Liquidity has nothing to do with it — the earlier 'no liquidity, returns before computing' reason was wrong, because no path search is attempted at all. Asserting this relationship needs a [path_search_max] override (or a non-validator node) in run-full-validation.sh."
+      "skip_reason": "Real relationship (pathfind.compute is created inside PathRequest::doUpdate at PathRequest.cpp:749-750, under the pathfind.request scope), but the child never exists on the harness because pathfinding is disabled on every node: Config.cpp:725-726 zeroes pathSearchMax whenever a [validation_seed] or [validator_token] section is present, run-full-validation.sh writes [validation_seed] for all five nodes (:308) with no [path_search*] override, and doRipplePathFind returns rpcNOT_SUPPORTED at RipplePathFind.cpp:48-49 before constructing a PathRequest. The parent used to appear anyway, because its ScopedSpanGuard is created at RipplePathFind.cpp:35, above that guard; since 2026-08-25 not even the parent appears, the ripple_path_find load having been removed from rpc_load_generator.py, so both ends of this relationship are now absent. Liquidity has nothing to do with it — the earlier 'no liquidity, returns before computing' reason was wrong, because no path search is attempted at all. Asserting this relationship needs the load restored AND a [path_search_max] override (or a non-validator node) in run-full-validation.sh."
     }
   ],
   "total_span_types": 41,
diff --git a/docker/telemetry/workload/rpc_load_generator.py b/docker/telemetry/workload/rpc_load_generator.py
index c6b07a6ac5..4d6ea894a3 100644
--- a/docker/telemetry/workload/rpc_load_generator.py
+++ b/docker/telemetry/workload/rpc_load_generator.py
@@ -5,13 +5,15 @@ Connects to one or more rippled WebSocket endpoints and fires all traced
 RPC commands at configurable rates with realistic production-like
 distribution.
 
-Command distribution (default weights):
+Command distribution (default weights, summing to 100):
   40%  Health checks:   server_info, fee
   30%  Wallet queries:  account_info, account_lines, account_objects
   15%  Explorer:        ledger, ledger_data
   10%  TX lookups:      tx, account_tx
    5%  DEX queries:     book_offers, amm_info
-   3%  Pathfinding:     ripple_path_find
+
+Path-finding RPC is deliberately absent — see "Pathfinding is not exercised"
+in README.md for why, and for how to put it back.
 
 Usage:
     python3 rpc_load_generator.py --endpoints ws://localhost:6006 --rate 50 --duration 120
@@ -68,10 +70,9 @@ DEFAULT_WEIGHTS: dict[str, int] = {
     # 5% DEX queries
     "book_offers": 3,
     "amm_info": 2,
-    # Pathfinding — exercises the pathfind.request/compute/discover spans.
-    # ripple_path_find is the synchronous (one-shot) variant that fits this
-    # fire-one-request WS client; path_find is a streaming subscription.
-    "ripple_path_find": 3,
+    # No path-finding command on purpose: every harness node disables
+    # pathfinding, so those calls only ever produced errors. README.md,
+    # "Pathfinding is not exercised", has the reason and the way back.
 }
 
 # Well-known genesis account for queries that require an account parameter.
@@ -340,13 +341,6 @@ def build_rpc_request(command: str) -> dict[str, Any]:
             "currency": "USD",
             "issuer": GENESIS_ACCOUNT,
         }
-    elif command == "ripple_path_find":
-        # Self-to-self XRP path search. It returns no usable paths, but the
-        # server still runs the full pathfinding pipeline (pathfind.request ->
-        # pathfind.compute -> pathfind.discover), which is what we trace.
-        req["source_account"] = GENESIS_ACCOUNT
-        req["destination_account"] = GENESIS_ACCOUNT
-        req["destination_amount"] = "1000000"  # 1 XRP in drops
 
     return req
 
diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md
index 10d5796825..071677c6ac 100644
--- a/docs/telemetry-runbook.md
+++ b/docs/telemetry-runbook.md
@@ -3623,7 +3623,7 @@ stands today.
 
 | Category         | Checks                                                                                                                                                                               | Description                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
 | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| Spans            | Every **required** entry in `expected_spans.json` — 41 span types at the time of writing: 26 required, 15 marked `"optional": true`                                                  | Span name found in Tempo carrying its `required_attributes`, plus the declared parent-child relationships. An `"optional": true` entry that does not fire is recorded as a skip, not a failure — it needs traffic the harness may not generate (HTTP/JSON-RPC client, gRPC client, missing-ledger fetch, mode transitions).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
+| Spans            | Every **required** entry in `expected_spans.json` — 41 span types at the time of writing: 25 required, 16 marked `"optional": true`                                                  | Span name found in Tempo carrying its `required_attributes`, plus the declared parent-child relationships. An `"optional": true` entry that does not fire is recorded as a skip, not a failure — it needs traffic the harness may not generate (HTTP/JSON-RPC client, gRPC client, missing-ledger fetch, mode transitions) or that it deliberately no longer generates (path-finding RPC — see "Pathfinding is not exercised" in [the workload README](../docker/telemetry/workload/README.md)).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
 | Metrics          | Every entry in every asserted category of `expected_metrics.json` — 84 checks across 25 asserting categories at the time of writing: 79 metric names plus 5 `required_labels` checks | SpanMetrics, `beast::insight` gauges/counters exported over OTLP, and the `MetricsRegistry` OTLP metrics. Each must have > 0 Prometheus series; none are optional. A category may also declare `required_labels`, and each label there becomes one additional check that at least one of that category's series carries it with a non-empty value (matched as `<label>!=""`, because Prometheus cannot distinguish an absent label from an empty one). Those labels were declared but never actually read until the check was generalised, so they were documented as required while going unverified; `spanmetrics` contributes 4 and `job_queue` 1. The separate `not_asserted` group lists metrics deliberately left out of the gate because they are workload-gated or defect-gated; it has neither a `metrics` nor a `required_labels` key, so the validator skips it entirely.                                                                                                                                                                                                                                                           |
 | Logs             | 2 checks                                                                                                                                                                             | `trace_id`/`span_id` present in Loki, and a Tempo trace id resolves in Loki. Skipped in CI, which runs `--skip-loki`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
 | Parity           | 10 checks                                                                                                                                                                            | 6 span attributes the external-parity dashboard panels read, plus 4 metric value-sanity bounds.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |

From c65cb0e2a89e5ba886bce1df61b79c652bdf10ae Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Tue, 25 Aug 2026 18:20:38 +0100
Subject: [PATCH 37/38] feat(telemetry): gate log-trace correlation in CI with
 per-leg diagnostics

The two log-correlation checks have never executed in CI: the workflow
hardcoded --skip-loki, so validate_telemetry.py never constructed
log.trace_id_present or log.trace_id_cross_reference. A green Telemetry
Validation therefore carried no evidence that a log line reaches Loki with
trace context. Drop the flag so both checks run and can fail the job.

Correlation spans four independent legs and a failed check names none of
them, so run-full-validation.sh now prints a per-leg diagnostic after the
suite whenever the checks are enabled:

  node      per-node debug.log line count, the count matching the injected
            trace_id/span_id shape, one sample line, and the severity mix,
            so "no log at all", "log level too high" and "no active sampled
            span" are distinguishable
  mount     the container-side listing of /var/log/xrpld, taken with the
            collector's own mounts and uid. That image is built from
            scratch and carries no shell, so the listing runs in a
            throwaway container with --volumes-from, not via docker exec
  collector the receiver's watched files, logs-pipeline warnings, and the
            internal log-record counters, read from inside the container's
            network namespace because that endpoint binds to the
            container's own localhost and its port is not published
  loki      the exact query used, the label inventory, and entry counts for
            the stream selector with and without the line filter, so "Loki
            has nothing" and "Loki has lines but none carry a trace id" are
            distinguishable

The diagnostics are non-fatal by construction: every leg runs in its own
subshell with errexit off, each docker and curl call is guarded, and the
coordinator always returns success. Verified with no containers and no Loki
reachable, with an emptied PATH, and with a leg forced to exit non-zero.

validate_telemetry.py gains a matching diagnostic beside the checks,
following _log_prometheus_metric_names: warnings only, never a check
result. Its stream selector and line filter move into module constants
that the shell diagnostic reads back, so the two cannot drift into
describing different queries.

No check was widened or auto-passed, and LOG_QUERY_WINDOW_SECONDS stays at
four hours; a wider window would let a check pass on a previous run's logs.
---
 .github/workflows/telemetry-validation.yml    |   2 +-
 OpenTelemetryPlan/06-implementation-phases.md |   8 +-
 docker/telemetry/workload/README.md           |  34 +-
 .../telemetry/workload/run-full-validation.sh | 327 ++++++++++++++++++
 .../telemetry/workload/validate_telemetry.py  | 133 ++++++-
 docs/telemetry-runbook.md                     |  39 ++-
 6 files changed, 502 insertions(+), 41 deletions(-)

diff --git a/.github/workflows/telemetry-validation.yml b/.github/workflows/telemetry-validation.yml
index 55c81d4465..8cb1305c32 100644
--- a/.github/workflows/telemetry-validation.yml
+++ b/.github/workflows/telemetry-validation.yml
@@ -234,7 +234,7 @@ jobs:
           # and never reads them. Load shape comes from the default
           # --profile full-validation. They are still passed so the flags stay
           # exercised if they are ever wired up.
-          ARGS="--xrpld ${{ env.BUILD_DIR }}/xrpld --skip-loki"
+          ARGS="--xrpld ${{ env.BUILD_DIR }}/xrpld"
           ARGS="$ARGS --rpc-rate $RPC_RATE"
           ARGS="$ARGS --rpc-duration $RPC_DURATION"
           ARGS="$ARGS --tx-tps $TX_TPS"
diff --git a/OpenTelemetryPlan/06-implementation-phases.md b/OpenTelemetryPlan/06-implementation-phases.md
index 4e925d16fb..9be3127c15 100644
--- a/OpenTelemetryPlan/06-implementation-phases.md
+++ b/OpenTelemetryPlan/06-implementation-phases.md
@@ -1044,7 +1044,8 @@ later metric families. The categories are:
   only asks Grafana for the dashboard and its panel count; it does not run the
   panel queries.
 - **Log-trace correlation** — `trace_id` present in Loki plus a Tempo reverse
-  lookup (skipped in CI via `--skip-loki`, not absent from the suite)
+  lookup (gated in CI; `run-full-validation.sh` prints a node/mount/collector/Loki
+  diagnostic alongside them so a failure names the leg that broke)
 
 See [Phase10_taskList.md](./Phase10_taskList.md) for the per-task breakdown.
 
@@ -1053,8 +1054,9 @@ See [Phase10_taskList.md](./Phase10_taskList.md) for the per-task breakdown.
 1. `rpc.process` -> `rpc.command.*` hierarchy — not assertable under the
    harness's WebSocket-only load, because `rpc.process` is created only on the
    HTTP path. This is a load-shape limitation, not a context-propagation bug.
-2. Log-trace correlation — implemented and passing locally; CI passes
-   `--skip-loki`.
+2. Log-trace correlation — the two `validate_telemetry.py` checks are now gated
+   in CI, but `integration-test.sh`'s own `check_log_correlation()` is still run
+   by no workflow.
 3. Legacy `beast::insight` coverage — `expected_metrics.json` asserts a
    representative subset, not all ~270 families.
 4. Sustained load / backpressure — the `stress` profile exists in
diff --git a/docker/telemetry/workload/README.md b/docker/telemetry/workload/README.md
index 80af47300e..5c10527978 100644
--- a/docker/telemetry/workload/README.md
+++ b/docker/telemetry/workload/README.md
@@ -400,27 +400,33 @@ Of the five `workflow_dispatch` inputs, only `run_benchmark` changes behaviour.
 them again — load shape comes entirely from `--profile` and
 `workload-profiles.json`. Their `description:` fields say so.
 
-### What CI does not cover
+### Log-trace correlation in CI
 
-The workflow invokes the harness with a hardcoded `--skip-loki`
-(`telemetry-validation.yml:237`), and `validate_telemetry.py` creates the
-`log.trace_id_present` and `log.trace_id_cross_reference` checks only inside an
-`if not skip_loki` branch (`validate_telemetry.py:1657`). Those two checks are
-therefore **never constructed on a CI run** — a green `Telemetry Validation`
-carries no evidence that log lines carry trace context or that a Tempo trace id
-resolves in Loki. `integration-test.sh` has its own `check_log_correlation()`, but
-no workflow runs that script either.
+The workflow no longer passes `--skip-loki`, so `log.trace_id_present` and
+`log.trace_id_cross_reference` are constructed and gated on every CI run. A green
+`Telemetry Validation` is now evidence that log lines carry trace context and
+that a logged trace id resolves to an exported trace. `integration-test.sh` has
+its own `check_log_correlation()`, but no workflow runs that script.
 
-Cover the gap locally by omitting the flag:
+Correlation depends on four independent legs, and a failed check on its own names
+none of them: the node must write a `debug.log` line carrying trace ids, the
+collector container must see that file, its `filelog` receiver must parse and
+export the line, and Loki must return it for the validator's LogQL.
+`run-full-validation.sh` prints a per-leg diagnostic after the suite whenever the
+Loki checks are enabled — per-node correlated-line counts and severity mix, the
+container-side view of `/var/log/xrpld`, the receiver's watched files and
+internal log-record counters, and Loki's own entry counts for the selector with
+and without the line filter. Read that block first; it identifies the broken leg
+without reproducing anything.
+
+The same block prints locally:
 
 ```bash
 docker/telemetry/workload/run-full-validation.sh --xrpld .build/xrpld
 ```
 
-Do that after any change to log formatting, span activation, the collector's
-`filelog` receiver, or the Loki exporter. Removing `--skip-loki` from the workflow
-would make CI exercise Loki ingestion and filelog mounting for the first time, so
-it is held back as its own change rather than folded into an unrelated push.
+Re-run it after any change to log formatting, span activation, the collector's
+`filelog` receiver, or the Loki exporter.
 
 ### Pathfinding is not exercised
 
diff --git a/docker/telemetry/workload/run-full-validation.sh b/docker/telemetry/workload/run-full-validation.sh
index d5295c9b3e..3524ef5648 100755
--- a/docker/telemetry/workload/run-full-validation.sh
+++ b/docker/telemetry/workload/run-full-validation.sh
@@ -87,6 +87,20 @@ BASELINE_FILE="${BASELINE_FILE:-$SCRIPT_DIR/baselines/baseline-timings.json}"
 THRESHOLDS_FILE="${THRESHOLDS_FILE:-$SCRIPT_DIR/regression-thresholds.json}"
 METRICS_FILE="${METRICS_FILE:-$SCRIPT_DIR/regression-metrics.json}"
 
+# Loki API base URL. Matches validate_telemetry.py's DEFAULT_LOKI, so the
+# diagnostics below query the same instance the log-correlation checks do.
+LOKI_URL="${LOKI_URL:-http://localhost:3100}"
+# Query window for the Loki diagnostics, in seconds. MUST stay equal to
+# LOG_QUERY_WINDOW_SECONDS in validate_telemetry.py (4h): a diagnostic that
+# looked further back than the check would report entries the check cannot see,
+# which is the one way these numbers could mislead rather than explain.
+DIAG_LOG_WINDOW_SECONDS=14400
+# The shape Logs::format() actually injects: a 32-hex trace_id followed by a
+# 16-hex span_id, both lowercase (src/libxrpl/basics/Log.cpp). Matching the
+# exact widths rather than a loose "trace_id=" substring keeps a truncated or
+# all-zero id from being counted as a correlated line.
+DIAG_TRACE_RE='trace_id=[0-9a-f]{32} span_id=[0-9a-f]{16}'
+
 GENESIS_ACCOUNT="rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh"
 GENESIS_SEED="snoPBrXtMeMyMHUVTgbuqAfg1SUTb"
 
@@ -528,6 +542,310 @@ else
     fold_exit 2
 fi
 
+# ---------------------------------------------------------------------------
+# Log-trace correlation diagnostics
+# ---------------------------------------------------------------------------
+# Log-trace correlation has four legs and a failed check names none of them:
+# the node must write a debug.log line carrying trace ids, the collector
+# container must see that file, its filelog receiver must parse and export the
+# line, and Loki must return it for the validator's own LogQL. Each leg below
+# reports what it observed, so a reader with only the CI log can tell which one
+# broke instead of guessing.
+#
+# Everything here is diagnostic. Every leg runs in its own subshell with
+# errexit off, and every external call is guarded, so a missing container, a
+# wedged docker daemon or an unreachable endpoint degrades to a printed note.
+# A diagnostic must never be the reason a run fails.
+
+# Fallback copies of the stream selector and line filter the log checks use.
+# Only reached when the constants cannot be read out of validate_telemetry.py
+# (see diag_validator_const); the module is the source of truth.
+DIAG_LOG_SELECTOR='{service_name="xrpld"}'
+DIAG_LOG_FILTER='|= "trace_id="'
+
+# Bound every external call: a hung docker daemon or endpoint must not stall
+# the run. Absent coreutils' timeout the calls still run, just unbounded.
+DIAG_HAVE_TIMEOUT=false
+if command -v timeout >/dev/null 2>&1; then
+    DIAG_HAVE_TIMEOUT=true
+fi
+
+diag_run() {
+    if [ "$DIAG_HAVE_TIMEOUT" = true ]; then
+        timeout 20 "$@"
+    else
+        "$@"
+    fi
+}
+
+# Container id of the running collector; empty when it is not up.
+diag_collector_cid() {
+    docker compose -f "$COMPOSE_FILE" ps -q otel-collector 2>/dev/null | head -1
+}
+
+# An image from this same stack that carries a shell, used to look inside the
+# collector's mounts and network namespace.
+#
+# The collector's own image cannot serve: it is built from scratch and ships
+# only the binary, so `docker exec <collector> ls` fails with "executable file
+# not found in $PATH". Read from the compose config rather than from a running
+# container so it resolves even when that service is down, and taken from the
+# stack so this never pulls an image the run did not already need.
+diag_shell_image() {
+    docker compose -f "$COMPOSE_FILE" config --format json 2>/dev/null |
+        jq -r '.services.prometheus.image // empty' 2>/dev/null
+}
+
+# Read a string constant out of validate_telemetry.py.
+#
+# Two copies of the same query would drift, and the copy with no check attached
+# is the one that ends up wrong — so the query is derived from the module that
+# runs the checks rather than restated here. $2 is the fallback, used only when
+# the import is impossible (e.g. aiohttp missing); it is the value the module
+# defines today, so the worst case is a stale literal rather than no output.
+diag_validator_const() {
+    python3 -c \
+        "import sys; sys.path.insert(0, '$SCRIPT_DIR'); import validate_telemetry as v; print(v.$1)" \
+        2>/dev/null || printf '%s\n' "$2"
+}
+
+# Sum an instant LogQL count over every stream it returns. Returns non-zero
+# when Loki could not be queried, which the caller reports as unavailable
+# rather than as zero — "Loki said none" and "Loki did not answer" are
+# different findings.
+diag_loki_count() {
+    local body
+    body=$(diag_run curl -sfG --max-time 10 "$LOKI_URL/loki/api/v1/query" \
+        --data-urlencode "query=$1" --data-urlencode "time=$2" 2>/dev/null) || return 1
+    printf '%s' "$body" |
+        jq -er '[.data.result[].value[1] | tonumber] | add // 0' 2>/dev/null || return 1
+}
+
+# Leg 1 — node: does xrpld emit correlated lines at all?
+#
+# Separates the three node-side failures that all surface as "no correlated
+# logs": no debug.log at all (the node died before opening its log sink), a log
+# level too high to reach the correlated call sites (severity mix shows no
+# NFO), and lines flowing but none emitted inside an active sampled span
+# (correlated=0 alongside a healthy severity mix).
+diag_node_logs() {
+    local i log bytes total correlated sample
+    echo "  [leg 1/4 node] debug.log lines matching '$DIAG_TRACE_RE'"
+    for i in $(seq 1 "$NUM_NODES"); do
+        log="$WORKDIR/node$i/debug.log"
+        if [ ! -f "$log" ]; then
+            echo "    node$i: no debug.log at $log — the node never opened its log sink"
+            continue
+        fi
+        bytes=$(wc -c <"$log" 2>/dev/null || echo 0)
+        total=$(wc -l <"$log" 2>/dev/null || echo 0)
+        # grep -c exits 1 on zero matches but still prints the count, so the
+        # guard keeps the 0 rather than replacing it with an empty string.
+        correlated=$(grep -cE "$DIAG_TRACE_RE" "$log" 2>/dev/null || true)
+        echo "    node$i: bytes=$bytes lines=$total correlated=${correlated:-0}"
+        # Severity mix from the '[partition:]SEV' token, which Logs::format()
+        # writes as the 4th whitespace-separated field. Fixed key order so two
+        # runs' output can be diffed directly. Lines whose 4th field is not a
+        # severity code are message continuations, counted as unparsed.
+        awk '{
+                 n = split($4, f, ":")
+                 s = f[n]
+                 if (s ~ /^(TRC|DBG|NFO|WRN|ERR|FTL)$/)
+                     c[s]++
+                 else
+                     other++
+             }
+             END {
+                 split("TRC DBG NFO WRN ERR FTL", order, " ")
+                 line = ""
+                 for (k = 1; k <= 6; k++)
+                     line = line sprintf("%s=%d ", order[k], c[order[k]] + 0)
+                 printf "      severity: %sunparsed=%d\n", line, other + 0
+             }' "$log" 2>/dev/null || echo "      severity: (could not be computed)"
+        sample=$(grep -m1 -E "$DIAG_TRACE_RE" "$log" 2>/dev/null || true)
+        if [ -n "$sample" ]; then
+            printf '      sample: %.200s\n' "$sample"
+        fi
+    done
+}
+
+# Leg 2 — mount: does the collector container see those files?
+#
+# A correct host-side log with an empty container-side view is the signature of
+# a mount or permission problem. The listing runs in a throwaway container
+# started with --volumes-from and the collector's own uid, so it reproduces the
+# collector's exact mount set and access rights instead of the host's.
+diag_collector_mount() {
+    local cid img usr binds
+    local -a user_flag=()
+    echo "  [leg 2/4 mount] container-side view of /var/log/xrpld"
+    if ! command -v docker >/dev/null 2>&1; then
+        echo "    docker is not on PATH — leg skipped"
+        return 0
+    fi
+    cid=$(diag_collector_cid)
+    if [ -z "$cid" ]; then
+        echo "    otel-collector container is not running — leg skipped"
+        return 0
+    fi
+    echo "    binds declared on the container:"
+    binds=$(diag_run docker inspect \
+        -f '{{range .Mounts}}{{.Type}} {{.Source}} -> {{.Destination}} rw={{.RW}}{{"\n"}}{{end}}' \
+        "$cid" 2>/dev/null || true)
+    if [ -n "$binds" ]; then
+        printf '%s\n' "$binds" | sed '/^[[:space:]]*$/d; s/^/      /'
+    else
+        echo "      (docker inspect reported no mounts)"
+    fi
+    img=$(diag_shell_image)
+    if [ -z "$img" ]; then
+        echo "    no stack image with a shell resolved — container-side listing skipped"
+        return 0
+    fi
+    usr=$(diag_run docker inspect -f '{{.Config.User}}' "$cid" 2>/dev/null || true)
+    if [ -n "$usr" ]; then
+        user_flag=(--user "$usr")
+    fi
+    echo "    listing as uid '${usr:-<image default>}' using $img:"
+    diag_run docker run --rm --volumes-from "$cid" \
+        ${user_flag[@]+"${user_flag[@]}"} --entrypoint /bin/sh "$img" -c \
+        'ls -la /var/log/xrpld 2>&1
+         find /var/log/xrpld -maxdepth 2 -name debug.log -exec ls -l {} \; 2>&1' |
+        sed 's/^/      /' || echo "      (container-side listing failed)"
+}
+
+# Leg 3 — collector: did the filelog receiver parse and export those lines?
+#
+# Two independent readings. The collector's own stderr names every file the
+# receiver opened and carries any filelog parse or Loki export error. Its
+# internal telemetry counts log records in and out: accepted>0 with sent=0 is
+# an export failure, accepted=0 while files are being watched is a parse
+# failure.
+#
+# That telemetry endpoint binds to the container's own localhost and its port
+# is not published, so it is unreachable from the host and is read from inside
+# the container's network namespace instead. The names below are a filter over
+# whatever the collector reports, not an assertion that any given series
+# exists; when it reports nothing matching, the leg says so.
+diag_collector_pipeline() {
+    local cid img watched problems metrics
+    echo "  [leg 3/4 collector] filelog receiver state"
+    if ! command -v docker >/dev/null 2>&1; then
+        echo "    docker is not on PATH — leg skipped"
+        return 0
+    fi
+    cid=$(diag_collector_cid)
+    if [ -z "$cid" ]; then
+        echo "    otel-collector container is not running — leg skipped"
+        return 0
+    fi
+    watched=$(diag_run docker logs "$cid" 2>&1 |
+        grep -F 'Started watching file' | grep -oE '"path": "[^"]*"' | sort -u || true)
+    if [ -n "$watched" ]; then
+        echo "    files the receiver opened:"
+        printf '%s\n' "$watched" | sed 's/^/      /'
+    else
+        echo "    the receiver reported opening no files"
+    fi
+    # Second filter keys on the collector's own logs-pipeline markers so this
+    # does not report warnings from the trace or metric pipelines. Nothing is
+    # excluded beyond that: the collector's benign config-alias deprecation
+    # notices ("filelog" -> "file_log") do surface here, and suppressing lines
+    # because they are usually harmless is how a diagnostic hides the one that
+    # was not.
+    problems=$(diag_run docker logs "$cid" 2>&1 |
+        grep -iE '(warn|error)' |
+        grep -iE 'filelog|fileconsumer|loki|signal": *"logs' |
+        tail -n 20 || true)
+    if [ -n "$problems" ]; then
+        echo "    logs-pipeline warnings and errors (last 20):"
+        printf '%s\n' "$problems" | cut -c1-300 | sed 's/^/      /'
+    else
+        echo "    no logs-pipeline warnings or errors in the collector's output"
+    fi
+    img=$(diag_shell_image)
+    if [ -z "$img" ]; then
+        echo "    no stack image with a shell resolved — internal telemetry skipped"
+        return 0
+    fi
+    metrics=$(diag_run docker run --rm --network "container:$cid" \
+        --entrypoint /bin/sh "$img" -c \
+        'wget -qO- --timeout=5 http://localhost:8888/metrics 2>/dev/null' 2>/dev/null |
+        grep -E '^otelcol_([a-z]+_)*log_records|^otelcol_fileconsumer' || true)
+    if [ -n "$metrics" ]; then
+        echo "    collector internal log-record counters:"
+        printf '%s\n' "$metrics" | sed 's/^/      /'
+    else
+        echo "    collector internal telemetry reported no log-record counters"
+    fi
+}
+
+# Leg 4 — Loki: is the entry queryable by the validator's own LogQL?
+#
+# Counts the selector on its own and the selector plus line filter separately,
+# so "Loki has nothing" is distinguishable from "Loki has lines but none carry
+# a trace id". The label inventory catches the third case: entries ingested
+# under a label set the validator's selector cannot match.
+diag_loki_stream() {
+    local now selector correlation query_all query_filtered
+    local total matching labels values sample
+    echo "  [leg 4/4 loki] entries in the last ${DIAG_LOG_WINDOW_SECONDS}s (the window the checks use)"
+    now=$(date +%s)
+    selector=$(diag_validator_const LOG_STREAM_SELECTOR "$DIAG_LOG_SELECTOR")
+    correlation=$(diag_validator_const LOG_CORRELATION_QUERY \
+        "$DIAG_LOG_SELECTOR $DIAG_LOG_FILTER")
+    [ -n "$selector" ] || selector="$DIAG_LOG_SELECTOR"
+    [ -n "$correlation" ] || correlation="$DIAG_LOG_SELECTOR $DIAG_LOG_FILTER"
+    query_all="count_over_time($selector[${DIAG_LOG_WINDOW_SECONDS}s])"
+    query_filtered="count_over_time($correlation [${DIAG_LOG_WINDOW_SECONDS}s])"
+    echo "    validator LogQL:  $correlation"
+    echo "    diagnostic LogQL: $query_all"
+    echo "                      $query_filtered"
+    labels=$(diag_run curl -sf --max-time 10 "$LOKI_URL/loki/api/v1/labels" 2>/dev/null |
+        jq -r '.data // [] | join(", ")' 2>/dev/null || true)
+    values=$(diag_run curl -sf --max-time 10 \
+        "$LOKI_URL/loki/api/v1/label/service_name/values" 2>/dev/null |
+        jq -r '.data // [] | join(", ")' 2>/dev/null || true)
+    echo "    stream labels Loki knows: ${labels:-<none or unreachable>}"
+    echo "    service_name values:      ${values:-<none or unreachable>}"
+    if total=$(diag_loki_count "$query_all" "$now"); then
+        echo "    entries matching the selector:            $total"
+    else
+        echo "    entries matching the selector:            unavailable (Loki did not answer)"
+    fi
+    if matching=$(diag_loki_count "$query_filtered" "$now"); then
+        echo "    entries also matching the line filter:    $matching"
+    else
+        echo "    entries also matching the line filter:    unavailable (Loki did not answer)"
+    fi
+    sample=$(diag_run curl -sfG --max-time 10 "$LOKI_URL/loki/api/v1/query_range" \
+        --data-urlencode "query=$correlation" \
+        --data-urlencode "start=$((now - DIAG_LOG_WINDOW_SECONDS))000000000" \
+        --data-urlencode "end=${now}000000000" \
+        --data-urlencode "limit=1" --data-urlencode "direction=backward" 2>/dev/null |
+        jq -r '.data.result[0].values[0][1] // empty' 2>/dev/null || true)
+    if [ -n "$sample" ]; then
+        printf '    sample entry: %.200s\n' "$sample"
+    fi
+}
+
+# Run every leg, isolated. Each subshell disables errexit and nounset so a leg
+# that trips still prints what it had, and its failure is reported rather than
+# propagated. Always returns success.
+run_log_correlation_diagnostics() {
+    local leg
+    echo ""
+    echo "--- Log-trace correlation diagnostics (non-fatal, for attribution only) ---"
+    for leg in diag_node_logs diag_collector_mount diag_collector_pipeline diag_loki_stream; do
+        (
+            set +e +u +o pipefail
+            "$leg"
+        ) || warn "$leg: the diagnostic itself failed — ignored"
+    done
+    echo ""
+    return 0
+}
+
 # ---------------------------------------------------------------------------
 # Step 5: Run telemetry validation suite
 # ---------------------------------------------------------------------------
@@ -546,6 +864,15 @@ if [ "$VALIDATION_EXIT" -eq 0 ]; then
 else
     fail "Some telemetry validation checks failed (exit $VALIDATION_EXIT)"
 fi
+
+# Only when the log-correlation checks actually ran: with --skip-loki there is
+# no result to attribute. Runs whether they passed or failed — the same numbers
+# that explain a failure are what proves a pass was not a coincidence. Placed
+# after the suite so its Loki counts are never earlier than the checks' own.
+if [ "$SKIP_LOKI" != true ]; then
+    run_log_correlation_diagnostics
+fi
+
 fold_exit "$VALIDATION_EXIT"
 
 # ---------------------------------------------------------------------------
diff --git a/docker/telemetry/workload/validate_telemetry.py b/docker/telemetry/workload/validate_telemetry.py
index 7f67118d8a..ef990dd8c3 100644
--- a/docker/telemetry/workload/validate_telemetry.py
+++ b/docker/telemetry/workload/validate_telemetry.py
@@ -48,6 +48,16 @@ import aiohttp
 # someone re-runs it later to investigate a result.
 LOG_QUERY_WINDOW_SECONDS = 4 * 60 * 60
 
+# Loki's OTLP ingestion promotes service.name to the stream label
+# `service_name`; a `job` attribute arrives as structured metadata, which a
+# stream selector cannot match (see otel-collector-config.yaml). Declared once
+# so both log checks and the diagnostic below report on the same query -- a
+# diagnostic that queried something else would describe a different failure
+# than the one being investigated.
+LOG_STREAM_SELECTOR = '{service_name="xrpld"}'
+LOG_TRACE_LINE_FILTER = '|= "trace_id="'
+LOG_CORRELATION_QUERY = f"{LOG_STREAM_SELECTOR} {LOG_TRACE_LINE_FILTER}"
+
 logger = logging.getLogger("validate_telemetry")
 
 # ---------------------------------------------------------------------------
@@ -1290,6 +1300,118 @@ async def _check_metric_label(
 # ---------------------------------------------------------------------------
 
 
+async def _loki_json(
+    session: aiohttp.ClientSession,
+    loki_url: str,
+    path: str,
+    params: dict[str, str] | None = None,
+) -> dict[str, Any] | None:
+    """GET one Loki API path and decode it, or None if that failed.
+
+    Diagnostic-only helper: every failure is a warning, so one unreachable
+    endpoint does not suppress the rest of the diagnostic.
+
+    Args:
+        session:  aiohttp client session.
+        loki_url: Loki API base URL.
+        path:     API path, including its leading slash.
+        params:   Optional query parameters.
+
+    Returns:
+        The decoded response body, or None when the request or decode failed.
+    """
+    try:
+        async with session.get(f"{loki_url}{path}", params=params or {}) as resp:
+            return await resp.json()
+    except Exception as exc:  # noqa: BLE001 - diagnostic must never raise
+        logger.warning("Loki diagnostic: GET %s failed: %s", path, exc)
+        return None
+
+
+async def _loki_count(
+    session: aiohttp.ClientSession, loki_url: str, query: str
+) -> int | None:
+    """Sum an instant LogQL count over every stream it returns.
+
+    Returns None rather than 0 when Loki could not be queried. "Loki holds
+    none" and "Loki did not answer" are different findings, and printing the
+    same number for both is how a diagnostic misleads.
+
+    Args:
+        session:  aiohttp client session.
+        loki_url: Loki API base URL.
+        query:    An instant LogQL metric query.
+
+    Returns:
+        Total entry count, or None when the query could not be evaluated.
+    """
+    params = {"query": query, "time": str(int(time.time()))}
+    data = await _loki_json(session, loki_url, "/loki/api/v1/query", params)
+    if data is None:
+        return None
+    try:
+        results = data.get("data", {}).get("result", [])
+        return sum(int(float(entry["value"][1])) for entry in results)
+    except Exception as exc:  # noqa: BLE001 - diagnostic must never raise
+        logger.warning("Loki diagnostic: could not read count for %s: %s", query, exc)
+        return None
+
+
+async def _log_loki_diagnostics(session: aiohttp.ClientSession, loki_url: str) -> None:
+    """Log what Loki holds for the stream the correlation checks select.
+
+    Same role as _log_prometheus_metric_names on the metric side, and the same
+    contract: diagnostic only, every failure a warning, never a check result.
+    A failed correlation check says a log line was not found; these numbers say
+    which of the three Loki-side reasons applies — nothing was ingested at all,
+    entries exist but none carry a trace id, or entries arrived under a label
+    set this selector cannot match (which the label inventory exposes).
+
+    Counted over exactly LOG_QUERY_WINDOW_SECONDS, the window the checks use.
+    A wider window here would report entries the checks cannot see, including a
+    previous run's, and describe a failure that is not the one at hand.
+
+    Args:
+        session:  aiohttp client session.
+        loki_url: Loki API base URL.
+    """
+    window = f"[{LOG_QUERY_WINDOW_SECONDS}s]"
+    logger.info("Loki diagnostic: the checks below query %s", LOG_CORRELATION_QUERY)
+
+    labels = await _loki_json(session, loki_url, "/loki/api/v1/labels")
+    if labels is not None:
+        names = labels.get("data") or []
+        logger.info(
+            "Loki diagnostic: stream labels Loki knows: %s",
+            ", ".join(names) or "(none)",
+        )
+
+    values = await _loki_json(
+        session, loki_url, "/loki/api/v1/label/service_name/values"
+    )
+    if values is not None:
+        found = values.get("data") or []
+        logger.info(
+            "Loki diagnostic: service_name values: %s", ", ".join(found) or "(none)"
+        )
+
+    for label, query in (
+        ("selector only", f"count_over_time({LOG_STREAM_SELECTOR}{window})"),
+        (
+            "selector + line filter",
+            f"count_over_time({LOG_CORRELATION_QUERY} {window})",
+        ),
+    ):
+        total = await _loki_count(session, loki_url, query)
+        logger.info(
+            "Loki diagnostic: %s = %s entries in the last %ds  [%s]",
+            label,
+            "unavailable" if total is None else total,
+            LOG_QUERY_WINDOW_SECONDS,
+            query,
+        )
+
+
 async def validate_log_trace_correlation(
     session: aiohttp.ClientSession,
     loki_url: str,
@@ -1310,13 +1432,14 @@ async def validate_log_trace_correlation(
     """
     logger.info("--- Log-Trace Correlation Validation (Loki) ---")
 
+    # Numbers first, so a failure below is read next to the state that caused
+    # it. Never raises and never adds a check (see _log_loki_diagnostics).
+    await _log_loki_diagnostics(session, loki_url)
+
     # Check 1: Any logs with trace_id exist.
     try:
         params = {
-            # Loki's OTLP ingestion promotes service.name to the label
-            # `service_name`. A `job` attribute is structured metadata, which a
-            # stream selector cannot match — see otel-collector-config.yaml.
-            "query": '{service_name="xrpld"} |= "trace_id="',
+            "query": LOG_CORRELATION_QUERY,
             "limit": 5,
             "direction": "backward",
             **_log_query_window(),
@@ -1360,7 +1483,7 @@ async def validate_log_trace_correlation(
     # written to a log must resolve to a trace that was actually exported.
     try:
         loki_params = {
-            "query": '{service_name="xrpld"} |= "trace_id="',
+            "query": LOG_CORRELATION_QUERY,
             "limit": 5,
             "direction": "backward",
             **_log_query_window(),
diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md
index 071677c6ac..de70a911d3 100644
--- a/docs/telemetry-runbook.md
+++ b/docs/telemetry-runbook.md
@@ -2774,7 +2774,7 @@ The sampled check is normally satisfied on a self-rooted consensus round — hea
 
 With all four satisfied, `info` is the minimum level at which the `log.trace_id_present` and `log.trace_id_cross_reference` checks pass by construction, and it is what the correlation-checking harnesses generate: the cfgs written by [run-full-validation.sh](../docker/telemetry/workload/run-full-validation.sh) and [integration-test.sh](../docker/telemetry/integration-test.sh) each set `enabled=1`, `trace_consensus=1` and `log_level info` together. `benchmark.sh` deliberately does not — it stays at `warning` to keep log I/O out of the overhead measurement, and it runs no correlation check. At `warning` and above that pair is suppressed and correlation becomes incidental — dependent on a `warn`-or-worse line happening to fire inside some active span.
 
-> **CI does not exercise either check.** `log.trace_id_present` and `log.trace_id_cross_reference` never run in CI — see [Coverage gap](#ci-workflow) for why. Cover them locally, without `--skip-loki`, after any change to log formatting, span activation, the `filelog` receiver or the Loki exporter:
+> **CI exercises both checks.** `log.trace_id_present` and `log.trace_id_cross_reference` are gated on every CI run — see [CI workflow](#ci-workflow) for the invocation and the per-leg diagnostics printed alongside them. Run the same thing locally after any change to log formatting, span activation, the `filelog` receiver or the Loki exporter:
 >
 > ```bash
 > docker/telemetry/workload/run-full-validation.sh --xrpld .build/xrpld
@@ -3602,7 +3602,7 @@ Harness options (`run-full-validation.sh`):
 | `--xrpld PATH`      | `.build/xrpld`    | Binary to run. Also settable via the `XRPLD` env var.                                                                                     |
 | `--nodes NUM`       | `5`               | Size of the local validator cluster.                                                                                                      |
 | `--profile NAME`    | `full-validation` | Load profile from `workload-profiles.json` (`full-validation`, `quick-smoke`, `stress`). This is the **only** thing that sets load shape. |
-| `--skip-loki`       | off               | Skip the log-trace correlation checks. CI always passes this.                                                                             |
+| `--skip-loki`       | off               | Skip the log-trace correlation checks and their per-leg diagnostics. Local exploration only; CI does not pass this.                       |
 | `--skip-regression` | off               | Skip timing capture and the baseline comparison. Local exploration only.                                                                  |
 | `--with-benchmark`  | off               | Also run `benchmark.sh` (telemetry-off vs telemetry-on overhead) after validation.                                                        |
 | `--cleanup`         | —                 | Tear everything down and exit.                                                                                                            |
@@ -3625,9 +3625,9 @@ stands today.
 | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | Spans            | Every **required** entry in `expected_spans.json` — 41 span types at the time of writing: 25 required, 16 marked `"optional": true`                                                  | Span name found in Tempo carrying its `required_attributes`, plus the declared parent-child relationships. An `"optional": true` entry that does not fire is recorded as a skip, not a failure — it needs traffic the harness may not generate (HTTP/JSON-RPC client, gRPC client, missing-ledger fetch, mode transitions) or that it deliberately no longer generates (path-finding RPC — see "Pathfinding is not exercised" in [the workload README](../docker/telemetry/workload/README.md)).                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
 | Metrics          | Every entry in every asserted category of `expected_metrics.json` — 84 checks across 25 asserting categories at the time of writing: 79 metric names plus 5 `required_labels` checks | SpanMetrics, `beast::insight` gauges/counters exported over OTLP, and the `MetricsRegistry` OTLP metrics. Each must have > 0 Prometheus series; none are optional. A category may also declare `required_labels`, and each label there becomes one additional check that at least one of that category's series carries it with a non-empty value (matched as `<label>!=""`, because Prometheus cannot distinguish an absent label from an empty one). Those labels were declared but never actually read until the check was generalised, so they were documented as required while going unverified; `spanmetrics` contributes 4 and `job_queue` 1. The separate `not_asserted` group lists metrics deliberately left out of the gate because they are workload-gated or defect-gated; it has neither a `metrics` nor a `required_labels` key, so the validator skips it entirely.                                                                                                                                                                                                                                                           |
-| Logs             | 2 checks                                                                                                                                                                             | `trace_id`/`span_id` present in Loki, and a Tempo trace id resolves in Loki. Skipped in CI, which runs `--skip-loki`.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                          |
+| Logs             | 2 checks                                                                                                                                                                             | `trace_id`/`span_id` present in Loki, and a logged trace id resolves in Tempo. Gated in CI. `run-full-validation.sh` prints a four-leg diagnostic (node, mount, collector, Loki) after the suite whenever these run, so a failure names the leg that broke.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
 | Parity           | 10 checks                                                                                                                                                                            | 6 span attributes the external-parity dashboard panels read, plus 4 metric value-sanity bounds.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                |
-| Dashboards       | Every uid in `expected_metrics.json` under `grafana_dashboards.uids` — currently all 15 provisioned dashboards                                                                       | Each listed dashboard loads and reports a panel count. This is a provisioning check only: it does **not** execute the panels' queries, so a dashboard can pass while individual panels render empty. `log-derived-insights` is Loki-backed, so under `--skip-loki` only its provisioning is meaningfully covered.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
+| Dashboards       | Every uid in `expected_metrics.json` under `grafana_dashboards.uids` — currently all 15 provisioned dashboards                                                                       | Each listed dashboard loads and reports a panel count. This is a provisioning check only: it does **not** execute the panels' queries, so a dashboard can pass while individual panels render empty. `log-derived-insights` is Loki-backed, so only its provisioning is covered here; its data path is covered by the two log checks instead.                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
 | Reverse coverage | 2 checks — `metric.reverse_coverage` and `span.reverse_coverage`                                                                                                                     | The only checks that run in the opposite direction: they read the full emitted inventory (the Prometheus `__name__` label values, the Tempo `span.name` tag values) and name everything the contract never mentions, sorted and one per line in the log. **Warn only — `passed` is hardcoded `True` in `_reverse_coverage_result`, so these can never fail CI.** Downstream branches legitimately add telemetry an upstream contract has not seen, and a hard failure would redden all of them. A metric family is accounted for by a `metrics` entry, by a `not_asserted.metrics_excluded` key, or by an anchored regex under the top-level `accounted_patterns` list — which exists for families whose membership is derived mechanically from a table in the code (the per-job-type job-queue instruments, the overlay per-category traffic cross product) plus the Prometheus scrape plumbing that is not xrpld telemetry. Histogram `_bucket`/`_count`/`_sum` names fold onto their base family before matching. Spans need no pattern list: the check reuses the forward matcher, so `rpc.command.*` covers every command it expands to. |
 
 ### Running Individual Tools
@@ -3808,20 +3808,23 @@ container as the main CI, so Conan and ccache hit the shared caches), and
   the workflow file, `docker/telemetry/**`, and the telemetry sources under
   `include/xrpl/telemetry/**`, `src/libxrpl/telemetry/**` and
   `src/xrpld/telemetry/**`. There is no cron schedule.
-- **Invocation**: `run-full-validation.sh --xrpld <binary> --skip-loki`, so the
-  default `full-validation` profile is used and the Loki checks are skipped.
-- **Coverage gap — log-trace correlation is never checked in CI**: the
-  `--skip-loki` above is hardcoded at
-  [telemetry-validation.yml:237](../.github/workflows/telemetry-validation.yml#L237),
-  and `validate_telemetry.py` creates `log.trace_id_present` and
-  `log.trace_id_cross_reference` only inside an `if not skip_loki` branch
-  ([:1657](../docker/telemetry/workload/validate_telemetry.py#L1657)), so those
-  two checks are not merely skipped — they are never constructed and never appear
-  in the report. `docker/telemetry/integration-test.sh` (which has its own
-  `check_log_correlation()`) is run by no workflow at all. A green
-  `Telemetry Validation` therefore carries no evidence about correlation; run it
-  locally without the flag to cover it:
-  `docker/telemetry/workload/run-full-validation.sh --xrpld .build/xrpld`.
+- **Invocation**: `run-full-validation.sh --xrpld <binary>`, so the default
+  `full-validation` profile is used and no category is skipped.
+- **Log-trace correlation is gated**: `--skip-loki` is not passed
+  ([telemetry-validation.yml:237](../.github/workflows/telemetry-validation.yml#L237)),
+  so `log.trace_id_present` and `log.trace_id_cross_reference` are constructed and
+  can fail the job. Correlation spans four independent legs — node, mount,
+  collector, Loki — and a failed check names none of them, so
+  `run-full-validation.sh` prints a per-leg diagnostic after the suite whenever
+  these checks are enabled: per-node counts of `debug.log` lines carrying the
+  injected `trace_id`/`span_id` shape plus the severity mix, the container-side
+  listing of `/var/log/xrpld` taken with the collector's own mounts and uid, the
+  `filelog` receiver's watched files, logs-pipeline warnings and internal
+  log-record counters, and Loki's entry counts for the stream selector with and
+  without the line filter. The diagnostics are non-fatal by construction: each
+  leg is isolated and a missing container or unreachable endpoint prints a note.
+  `docker/telemetry/integration-test.sh` (which has its own
+  `check_log_correlation()`) is still run by no workflow.
 - **Inputs**: only `run_benchmark` changes behaviour. `rpc_rate`, `rpc_duration`,
   `tx_tps` and `tx_duration` are inert, as noted in their descriptions.
 - **Results**: reports are uploaded as the `telemetry-validation-reports`

From 3836078a78aafc55f87d6a2fbc15d20c7316824e Mon Sep 17 00:00:00 2001
From: Pratik Mankawde <3397372+pratikmankawde@users.noreply.github.com>
Date: Tue, 25 Aug 2026 18:21:35 +0100
Subject: [PATCH 38/38] fix(telemetry): stop gating ledger.validate p95 and
 p99, which vary too much

The regression gate has been red on runs with no code change. Only two of
the 25 gated keys ever tripped, both on the same span and never together:
run 32862589645 failed p99 at 25.8750 ms against a 1.0600 ms baseline
(+2341%), run 32867433073 failed p95 at 0.7500 ms against 0.2404 ms
(+212%), and in each run the other quantile sat well inside its own bound.
A real slowdown would move both. This is variance, not a defect.

Measured across four CI runs:

  span.ledger.validate.p50   0.0484 to 0.0778 ms    1.6x spread   kept
  span.ledger.validate.p95   0.1281 to 0.7500 ms    5.9x spread   excluded
  span.ledger.validate.p99   0.3875 to 25.8750 ms  66.8x spread   excluded

Both excluded quantiles reach past their trip point on a healthy run. The
mechanism is arrival timing, not slow code: the span opens only once a
quorum-completing validation arrives (LedgerMaster.cpp:987, inside
checkAccept, past the early return) and wraps the promotion work that
follows, so one slow consensus round dominates the tail of a 3m rate
window and which round that is differs every run.

Widening is not available and must not be attempted later: tolerating
25.8750 ms against a 1.0600 ms baseline needs a bound of about 24.8 ms,
which gates nothing. A bound admitting every healthy run's worst case
admits every regression too. p50 stays gated; it is stable.

THE GENERAL RULE, recorded so this does not recur: an absolute bound
derived as hi_next minus baseline comes from the histogram ladder, so it
budgets for quantization noise and for nothing else. It knows nothing about
how far a metric moves between runs on identical code. Before gating any
key, check its observed maximum across several runs against its trip point
and gate it only with margin. Spread alone proves nothing: tx.apply.p50
swings 364x and never fires, because its 5 ms trip point absorbs the range.
Of the 23 keys still gated the worst reaches 0.67 of its trip point.

Mechanism: spans.names lists span names while _quantiles is shared, so
dropping two quantiles of one span cannot be expressed by deleting a name.
regression-metrics.json gains an excluded_keys map from a flat key to the
reason it is not gated, subtracted by both prom_queries.py (so the key is
never queried) and check_regression_bounds.py rule A. A per-name quantile
override was rejected: a typo there leaves the key gating, whereas a typo
in an exclusion subtracts nothing and new rule F rejects it, along with an
empty reason, a leftover threshold override and a leftover baseline value.

Derived figures recomputed from the committed baseline: 25 gated keys to
23, detection floor 2.02x-9.43x to 2.02x-9.42x, weakly guarded keys ten to
nine, bound over baseline 102%-843% to 102%-842%. The baseline edit is a
deletion of two entries only, with no value rewritten.

Verified: both previously failing runs replay to zero regressions and exit
0; a tenfold increase injected into each of the 23 remaining keys in turn
is still caught in all 23 cases; rule F was confirmed load-bearing by
stubbing it out, which lets a stale exclusion pass.
---
 .../telemetry/check_regression_bounds.py      | 79 ++++++++++++++++++-
 .../telemetry/test_check_regression_bounds.py | 62 ++++++++++++++-
 docker/telemetry/workload/README.md           |  9 ++-
 docker/telemetry/workload/baselines/README.md | 76 ++++++++++++++++--
 .../workload/baselines/baseline-timings.json  |  8 --
 docker/telemetry/workload/prom_queries.py     | 37 +++++++--
 .../workload/regression-metrics.json          |  5 ++
 .../workload/regression-thresholds.json       | 14 +---
 docs/telemetry-runbook.md                     | 19 +++--
 9 files changed, 268 insertions(+), 41 deletions(-)

diff --git a/.github/scripts/telemetry/check_regression_bounds.py b/.github/scripts/telemetry/check_regression_bounds.py
index 94189fc04d..03d0288946 100644
--- a/.github/scripts/telemetry/check_regression_bounds.py
+++ b/.github/scripts/telemetry/check_regression_bounds.py
@@ -26,7 +26,7 @@ bucket ``(lo, hi]`` of its ladder, with ``hi_next`` the next edge above ``hi``,
     max_abs_increase_* == hi_next - baseline
 
 so the gate trips only when the reading clears the bucket *above* the
-baseline's own. Five rules are checked:
+baseline's own. Six rules are checked:
 
   A  the baseline's key set equals the surface ``regression-metrics.json``
      declares (a stale key left behind reads as covered but never gates);
@@ -38,7 +38,15 @@ baseline's own. Five rules are checked:
      stops being true;
   E  no baseline carries the ladder-floor signature ``quantile x first_edge``,
      which means every sample landed in the first bucket and the number is
-     interpolation arithmetic rather than a latency.
+     interpolation arithmetic rather than a latency;
+  F  every entry in ``excluded_keys`` names a key the surface would otherwise
+     declare, carries a reason, and has neither a threshold override nor a
+     baseline value left behind.
+
+Rule A subtracts ``excluded_keys`` before comparing, so a quantile removed from
+the gated set does not read as a missing baseline. Rule F is what keeps that
+subtraction honest: an exclusion is the one edit here that makes the gate cover
+LESS, so a stale or misspelt entry must fail rather than silently widen itself.
 
 A PLACEHOLDER baseline -- ``"placeholder": true`` or an empty ``metrics``
 object -- exits 0, because that is the documented bootstrap state and CI has to
@@ -119,6 +127,10 @@ def declared_keys(metrics_cfg):
     Deliberately reimplemented rather than imported from ``prom_queries.py``,
     which pulls in aiohttp; CI telemetry checks stay dependency-free. The key
     format is fixed by that file's own ``_key_format`` field.
+
+    ``excluded_keys`` is NOT subtracted here: rule F needs the full product to
+    tell a real exclusion from a misspelt one. Callers that want the gated
+    surface subtract it themselves.
     """
     keys = set()
     spans = metrics_cfg.get("spans", {})
@@ -154,6 +166,57 @@ def resolve_override(key, thresholds):
     return thresholds.get("overrides", {}).get(group, {}).get(quantile)
 
 
+def check_exclusions(metrics_cfg, thresholds, baseline_metrics, declared):
+    """Apply rule F to every entry in ``excluded_keys``.
+
+    An exclusion is the only edit to this config that makes the gate cover
+    LESS, so each entry has to prove it is deliberate and complete:
+
+      * it names a key the names x quantiles product would otherwise declare,
+        so a typo or a stale entry surviving a surface change is caught rather
+        than silently subtracting nothing;
+      * it carries a non-empty reason, because "why is this not gated" is the
+        question a future maintainer will ask and prose is the only answer;
+      * no threshold override and no baseline value are left behind, since
+        either would read as gated to anyone grepping for the key.
+
+    Args:
+        metrics_cfg:      Parsed regression-metrics.json.
+        thresholds:       Parsed regression-thresholds.json.
+        baseline_metrics: The baseline's ``metrics`` map.
+        declared:         Output of declared_keys(), before exclusions come off.
+
+    Returns:
+        A list of failure strings, empty when every entry is well formed.
+    """
+    failures = []
+    for key, reason in sorted(metrics_cfg.get("excluded_keys", {}).items()):
+        if key not in declared:
+            failures.append(
+                f"{key}: listed in excluded_keys but not produced by the "
+                f"names x quantiles product, so it subtracts nothing -- fix the "
+                f"spelling or drop the entry (rule F)"
+            )
+            continue
+        if not isinstance(reason, str) or not reason.strip():
+            failures.append(
+                f"{key}: excluded with no reason. Record why it is not gated, "
+                f"with the measurement behind it (rule F)"
+            )
+        if resolve_override(key, thresholds) is not None:
+            failures.append(
+                f"{key}: excluded but still has a threshold override, which "
+                f"reads as gated -- remove it from {THRESHOLDS} (rule F)"
+            )
+        if key in baseline_metrics:
+            failures.append(
+                f"{key}: excluded but still has a baseline value, so rule A "
+                f"would pass while nothing gates it -- remove it from "
+                f"{BASELINE} (rule F)"
+            )
+    return failures
+
+
 def check_key(key, entry, thresholds, ladders):
     """Apply rules B, C, D and E to one gated key. Returns a list of failures."""
     value, unit = entry.get("value"), entry.get("unit", "")
@@ -242,6 +305,10 @@ def main():
     failures = []
 
     declared = declared_keys(metrics_cfg)
+    failures.extend(check_exclusions(metrics_cfg, thresholds, gated, declared))
+    # Rule A compares against the GATED surface, so a deliberately excluded
+    # quantile is not reported as a baseline that was never captured.
+    declared -= set(metrics_cfg.get("excluded_keys", {}))
     for key in sorted(set(gated) - declared):
         failures.append(
             f"{key}: in the baseline but not declared by {METRICS}, so it is "
@@ -258,11 +325,19 @@ def main():
             failures.extend(check_key(key, gated[key], thresholds, ladders))
 
     if not failures:
+        excluded = metrics_cfg.get("excluded_keys", {})
         print(
             f"OK: {len(gated)} gated key(s); every absolute bound equals "
             f"hi_next - baseline, every key has an override, and the absolute "
             f"bound is the operative half of the AND for all of them"
         )
+        # Printed, not silent: an exclusion narrows the gate, so the count
+        # belongs in the CI log where a reviewer sees it without opening a file.
+        if excluded:
+            print(
+                f"    {len(excluded)} declared key(s) deliberately not gated: "
+                f"{', '.join(sorted(excluded))} (see excluded_keys in {METRICS})"
+            )
         return 0
 
     print(
diff --git a/.github/scripts/telemetry/test_check_regression_bounds.py b/.github/scripts/telemetry/test_check_regression_bounds.py
index 4c7a4000cb..ba025b0036 100644
--- a/.github/scripts/telemetry/test_check_regression_bounds.py
+++ b/.github/scripts/telemetry/test_check_regression_bounds.py
@@ -13,7 +13,7 @@ Two groups:
   is the documented bootstrap state, while a missing, unreadable or malformed
   input must FAIL. A checker that returns success without having checked
   anything is the failure this whole gate exists to prevent;
-* one case per rule (A to E), so a rule that stops flagging is caught.
+* one case per rule (A to F), so a rule that stops flagging is caught.
 
 stdlib unittest only; the repo installs no third-party runner for CI.
 """
@@ -193,6 +193,66 @@ class TestRules(CheckerCase):
         self.assertEqual(code, 1, out)
         self.assertIn("(rule D)", out)
 
+    def test_rule_a_ignores_an_excluded_key(self):
+        """An excluded key must not read as a baseline that was never captured.
+
+        The unmodified tree already exercises this — span.ledger.validate p95
+        and p99 are declared by the names x quantiles product, excluded, and
+        absent from the baseline — so this asserts the subtraction is what makes
+        it pass, by naming the keys in the reported exclusion line.
+        """
+        code, out = self.run_checker()
+        self.assertEqual(code, 0, out)
+        self.assertIn("span.ledger.validate.p95", out)
+        self.assertIn("deliberately not gated", out)
+
+    def test_rule_f_flags_exclusion_that_subtracts_nothing(self):
+        """A misspelt or stale exclusion silently narrows nothing — catch it."""
+        self.edit_json(
+            METRICS,
+            lambda d: d["excluded_keys"].update({"span.ledger.validate.p97": "typo"}),
+        )
+        code, out = self.run_checker()
+        self.assertEqual(code, 1, out)
+        self.assertIn("(rule F)", out)
+        self.assertIn("subtracts nothing", out)
+
+    def test_rule_f_flags_exclusion_without_a_reason(self):
+        self.edit_json(
+            METRICS,
+            lambda d: d["excluded_keys"].update({"span.ledger.validate.p95": "   "}),
+        )
+        code, out = self.run_checker()
+        self.assertEqual(code, 1, out)
+        self.assertIn("(rule F)", out)
+        self.assertIn("no reason", out)
+
+    def test_rule_f_flags_override_left_behind(self):
+        """An excluded key still carrying a bound reads as gated."""
+        self.edit_json(
+            THRESHOLDS,
+            lambda d: d["overrides"]["span.ledger.validate"].update(
+                {"p95": {"max_pct_increase": 50.0, "max_abs_increase_ms": 0.25}}
+            ),
+        )
+        code, out = self.run_checker()
+        self.assertEqual(code, 1, out)
+        self.assertIn("(rule F)", out)
+        self.assertIn("still has a threshold override", out)
+
+    def test_rule_f_flags_baseline_value_left_behind(self):
+        """Excluded but still in the baseline: rule A passes, nothing gates."""
+        self.edit_json(
+            BASELINE,
+            lambda d: d["metrics"].update(
+                {"span.ledger.validate.p95": {"unit": "ms", "value": 0.24}}
+            ),
+        )
+        code, out = self.run_checker()
+        self.assertEqual(code, 1, out)
+        self.assertIn("(rule F)", out)
+        self.assertIn("still has a baseline value", out)
+
     def test_rule_e_flags_ladder_floor_signature(self):
         """ledger.store's quantiles were the ladder floor times the quantile."""
         store = {"p50": 0.005, "p95": 0.0095, "p99": 0.0099}
diff --git a/docker/telemetry/workload/README.md b/docker/telemetry/workload/README.md
index 5c10527978..eebd106e44 100644
--- a/docker/telemetry/workload/README.md
+++ b/docker/telemetry/workload/README.md
@@ -239,7 +239,8 @@ How it runs inside the validation pipeline:
 
 1. `run-full-validation.sh` executes the normal workload and validation suite.
 2. After validation, `capture_timings.py` queries Prometheus for every
-   metric in `regression-metrics.json` and writes `reports/timings.json`.
+   metric `regression-metrics.json` declares and does not list in
+   `excluded_keys`, then writes `reports/timings.json`.
 3. `compare_to_baseline.py` reads `timings.json`,
    `baselines/baseline-timings.json`, and `regression-thresholds.json`,
    then either:
@@ -269,6 +270,12 @@ Per-run tuning:
   top of the next bucket up — so refreshing the baseline obliges you to
   re-derive the bounds. See `_absolute_bound_derivation` in that file;
   `.github/scripts/telemetry/check_regression_bounds.py` enforces it in CI.
+- That bound budgets for **quantization** noise only, so a key whose run-to-run
+  variance is larger than it cannot be gated at all — `span.ledger.validate.p95`
+  and `.p99` are excluded for that reason and are listed, with the measurements,
+  in `excluded_keys` in `regression-metrics.json`. Check a key's observed maximum
+  across runs against `baseline + bound` before gating it; widening the bound is
+  not the fix. See `baselines/README.md`.
 
 See [`baselines/README.md`](./baselines/README.md) for the baseline
 lifecycle and refresh process.
diff --git a/docker/telemetry/workload/baselines/README.md b/docker/telemetry/workload/baselines/README.md
index 119bef5e5f..eaa3c8e9ca 100644
--- a/docker/telemetry/workload/baselines/README.md
+++ b/docker/telemetry/workload/baselines/README.md
@@ -25,9 +25,9 @@ was invoked with. Capture and comparison are profile-agnostic — they only read
 Prometheus — so all existing profiles (`full-validation`, `quick-smoke`, `stress`)
 continue to work unchanged.
 
-## Current state: 25 metrics gate, on a baseline captured 2026-08-24
+## Current state: 23 metrics gate, on a baseline captured 2026-08-24
 
-`baseline-timings.json` holds real captured values for the 25 keys the harness gates. The
+`baseline-timings.json` holds real captured values for the 23 keys the harness gates. The
 previous entries, captured on 2026-06-05, were voided into a placeholder first: they predated the
 spanmetrics ladder re-cut of 2026-08-04 (`3860c93db2`), which made every sub-millisecond quantile
 in that capture bucket-edge arithmetic rather than a latency (a p95 of `0.95` ms is `0.95 × 1 ms`).
@@ -64,7 +64,15 @@ Two earlier generations of this bound were wrong, in opposite directions:
 | ------------------------------ | -------------------------------------------- | --------------------- | ---------------------------------------- |
 | flat                           | 10 ms `p50`/`p95`, 15 ms `p99`, 20000 us job | 5 / 28 keys           | 2 / 25 keys                              |
 | 2 × enclosing bucket width     | per metric                                   | 28 / 28 keys          | **21 / 25 keys**                         |
-| `hi_next − baseline` (current) | per metric                                   | 25 / 25 keys          | **0 / 25 keys**                          |
+| `hi_next − baseline` (current) | per metric                                   | 23 / 23 keys          | **0 / 23 keys**                          |
+
+The first two rows were measured when 28 and 25 keys were gated; the current row was re-measured
+over today's 23 by injecting a 10x regression into each gated key in turn against a real CI
+`timings.json`, which the gate flagged in all 23 cases. The zero in the last column is by
+construction rather than by sampling: rule C in
+[`check_regression_bounds.py`](../../../../.github/scripts/telemetry/check_regression_bounds.py)
+fails the build unless every trip point is exactly `hi_next`, and a trip point at a bucket edge
+cannot be crossed by interpolation inside that bucket.
 
 The flat bound was calibrated for a 5-25 ms band the spans do not occupy: 18 of the 28
 quantiles gated at the time sat below 1 ms, so it sat 1.15x to 2000x above the metric it guarded,
@@ -86,11 +94,10 @@ signature described below.
 
 The guarantee costs sensitivity where the ladder is coarse: the detection floor is
 `hi_next / baseline`, so a baseline sitting just above an edge is guarded loosely. Measured over
-the current baseline the floor ranges 2.02x to 9.43x. Do **not** read these as guarded:
+the current baseline the floor ranges 2.02x to 9.42x. Do **not** read these as guarded:
 
 | key                               | baseline   | fires at  | floor |
 | --------------------------------- | ---------- | --------- | ----- |
-| `span.ledger.validate.p99`        | 1.0600 ms  | 10 ms     | 9.43x |
 | `span.ledger.build.p50`           | 1.0612 ms  | 10 ms     | 9.42x |
 | `span.tx.process.p95`             | 0.7240 ms  | 5 ms      | 6.91x |
 | `span.tx.apply.p50`               | 0.7917 ms  | 5 ms      | 6.32x |
@@ -103,7 +110,7 @@ the current baseline the floor ranges 2.02x to 9.43x. Do **not** read these as g
 
 `span.ledger.build.p50` is the one that matters most: ledger construction is the hot path this
 gate exists to guard, and at a 9.42x floor it could get almost ten times slower and still pass.
-All ten are limited by two 5x-wide
+All nine are limited by two 5x-wide
 ladder steps, 1 ms → 5 ms and 5000 us → 25000 us. The fix is a 2 ms edge (ideally 3 ms as well) in
 the collector's spanmetrics `buckets` list plus the matching entries in `kMillisecondBuckets`, and
 a 10000 us edge in `kMicrosecondBuckets`. That work belongs to the branch that owns the ladders.
@@ -129,6 +136,58 @@ presence is still asserted by `../expected_spans.json` and `docker/telemetry/int
 and its rate is still on the ledger-operations dashboard; only the latency gate drops it.
 `check_regression_bounds.py` rule E fails the build if a key with this signature is gated again.
 
+## Known exclusion: `ledger.validate` p95 and p99 vary more than any bound can absorb
+
+`span.ledger.validate.p95` and `.p99` are **not** gated. `p50` still is. They are the first
+exclusion at _quantile_ rather than _span_ granularity, which is why
+[`../regression-metrics.json`](../regression-metrics.json) grew an `excluded_keys` map — `spans.names`
+lists span names and `_quantiles` is shared across all of them, so removing two quantiles of one
+span cannot be expressed by deleting a name.
+
+Measured across four CI runs:
+
+| key                               | baseline  | trip point | observed min | observed max | spread |
+| --------------------------------- | --------- | ---------- | ------------ | ------------ | ------ |
+| `span.ledger.validate.p50` (kept) | 0.0779 ms | 0.25 ms    | 0.0484 ms    | 0.0778 ms    | 1.6x   |
+| `span.ledger.validate.p95`        | 0.2404 ms | 0.5 ms     | 0.1281 ms    | 0.7500 ms    | 5.9x   |
+| `span.ledger.validate.p99`        | 1.0600 ms | 10 ms      | 0.3875 ms    | 25.8750 ms   | 66.8x  |
+
+Both excluded quantiles reach past their trip point on an ordinary run, so CI reddened twice with
+no code change: run `32867433073` read `p95` = 0.7500 ms (+212%) and run `32862589645` read
+`p99` = 25.8750 ms (+2341%). The two failures landed on **different** quantiles in different runs
+while the other quantile stayed well inside its bound in the same run — the signature of variance,
+not of a regression.
+
+The mechanism is arrival timing, not slow code. The span opens only once a quorum-completing
+validation arrives ([`LedgerMaster.cpp:987`](../../../../src/xrpld/app/ledger/detail/LedgerMaster.cpp#L987),
+inside `checkAccept`, past the `tvc < minVal` early return) and wraps the promotion work that
+follows — `setValidated`, `setFull`, `setValidLedger`, `pendSaveValidated`. Its duration therefore
+tracks when peer validations arrive in a 5-node cluster and what promotion then schedules, so a
+single slow consensus round dominates the tail of a 3 m rate window, and which round that is
+differs every run.
+
+**Widening the bound is not an option and must not be attempted.** Tolerating 25.8750 ms against a
+1.0600 ms baseline needs a bound of ~24.8 ms, i.e. a gate that fires at nothing a regression could
+plausibly reach. A bound that admits every healthy run's worst case admits every regression too.
+`check_regression_bounds.py` rule F fails the build if either key is re-gated with a bound while
+still listed in `excluded_keys`, and the per-key reasons in that map record this in full.
+
+### The general rule this exposed
+
+`hi_next − baseline` is derived from the **ladder**, so it budgets for **quantization** noise — one
+bucket of interpolation headroom — and for nothing else. It knows nothing about how far the metric
+itself moves between runs on identical code. Where run-to-run workload variance is the larger term,
+the bound is simply the wrong size and the gate reddens on a healthy run.
+
+**Before gating any key, check its observed maximum across several runs against its trip point
+(`baseline + bound`), and gate it only if the maximum stays below that with margin.** Spread alone
+proves nothing: `span.tx.apply.p50` swings 364x across runs (0.0064 → 2.3378 ms) and never fires,
+because its 5 ms trip point absorbs the whole range. It is spread **relative to the trip point**
+that decides. Measured over the runs behind this baseline, the worst surviving key reaches 0.67 of
+its trip point (`span.consensus.ledger_close.p95`) and the other 22 sit at or below 0.60 — the two
+excluded keys, at 1.50 and 2.59, were the only ones above 1.0. A key that fails this test is not
+fixed by widening its bound; exclude it and say why.
+
 ## Bootstrapping the baseline
 
 1. Merge a CI run with a `"placeholder": true` baseline. The telemetry-validation
@@ -158,6 +217,11 @@ Refreshing the baseline also obliges you to re-derive the absolute bounds in
 A value that moves into a different bucket needs a different bound, and a bound left behind
 either stops catching regressions or starts firing on quantization noise.
 
+It also obliges you to re-check each key's run-to-run spread against its new trip point, per
+[The general rule this exposed](#the-general-rule-this-exposed). A refreshed baseline can land in a
+bucket whose `hi_next` no longer clears the metric's own variance, which turns the key into a
+recurring false positive — the failure that excluded `ledger.validate` p95 and p99.
+
 ## The baseline is only valid at the log level it was captured at
 
 Every timing here is coupled to the `log_level` that `run-full-validation.sh` writes into
diff --git a/docker/telemetry/workload/baselines/baseline-timings.json b/docker/telemetry/workload/baselines/baseline-timings.json
index c02ca2b558..07b289c864 100644
--- a/docker/telemetry/workload/baselines/baseline-timings.json
+++ b/docker/telemetry/workload/baselines/baseline-timings.json
@@ -58,14 +58,6 @@
       "unit": "ms",
       "value": 0.07787769784172663
     },
-    "span.ledger.validate.p95": {
-      "unit": "ms",
-      "value": 0.2404310344827586
-    },
-    "span.ledger.validate.p99": {
-      "unit": "ms",
-      "value": 1.060000000000033
-    },
     "span.rpc.ws_message.p50": {
       "unit": "ms",
       "value": 0.15990942879094971
diff --git a/docker/telemetry/workload/prom_queries.py b/docker/telemetry/workload/prom_queries.py
index a26a2b5459..08c1afc64c 100644
--- a/docker/telemetry/workload/prom_queries.py
+++ b/docker/telemetry/workload/prom_queries.py
@@ -68,13 +68,24 @@ def _build_simple_entries(
     cfg: dict,
     prefix: str,
     window: str,
+    excluded: frozenset[str] = frozenset(),
 ) -> list[QueryEntry]:
-    """Build QueryEntry list for a single-template category (spans, rpc)."""
+    """Build QueryEntry list for a single-template category (spans, rpc).
+
+    ``excluded`` holds flat keys the surface declares but does not gate (see
+    ``excluded_keys`` in regression-metrics.json). They are dropped here rather
+    than filtered later, so a key that no longer gates is never queried and
+    never appears in ``timings.json`` — a captured key with no baseline reports
+    as "new metric (not in baseline)", which reads as coverage while being
+    unable to gate.
+    """
     tmpl = cfg.get("_query_template", "")
     unit = cfg.get("_unit", "ms")
     entries: list[QueryEntry] = []
     for name in cfg.get("names", []):
         for q in cfg.get("_quantiles", []):
+            if f"{prefix}.{name}.p{_quantile_label(q)}" in excluded:
+                continue
             expr = (
                 tmpl.replace("{quantile}", _format_quantile(q))
                 .replace("{name}", name)
@@ -90,8 +101,14 @@ def _build_simple_entries(
     return entries
 
 
-def _build_job_entries(cfg: dict, window: str) -> list[QueryEntry]:
-    """Build QueryEntry list for the job_queue category (multi-phase)."""
+def _build_job_entries(
+    cfg: dict, window: str, excluded: frozenset[str] = frozenset()
+) -> list[QueryEntry]:
+    """Build QueryEntry list for the job_queue category (multi-phase).
+
+    ``excluded`` is applied for the same reason as in _build_simple_entries; the
+    key format is flat, so one exclusion list covers both categories.
+    """
     unit = cfg.get("_unit", "us")
     phases = cfg.get("_phases", ["queued", "running"])
     tmpl_map = {
@@ -105,6 +122,8 @@ def _build_job_entries(cfg: dict, window: str) -> list[QueryEntry]:
             if not tmpl:
                 continue
             for q in cfg.get("_quantiles", []):
+                if f"job.{name}.{phase}.p{_quantile_label(q)}" in excluded:
+                    continue
                 expr = (
                     tmpl.replace("{quantile}", _format_quantile(q))
                     .replace("{name}", name)
@@ -131,15 +150,19 @@ def build_query_plan(metrics_path: str | Path, window: str = "3m") -> list[Query
                       ``regression`` workload profile.
 
     Returns:
-        A list of ``QueryEntry`` values, one per (metric × quantile).
+        A list of ``QueryEntry`` values, one per (metric × quantile), less any
+        key listed in the config's ``excluded_keys`` map.
     """
     with open(metrics_path) as f:
         cfg = json.load(f)
 
+    excluded = frozenset(cfg.get("excluded_keys", {}))
     plan: list[QueryEntry] = []
-    plan.extend(_build_simple_entries(cfg.get("spans", {}), "span", window))
-    plan.extend(_build_simple_entries(cfg.get("rpc_methods", {}), "rpc", window))
-    plan.extend(_build_job_entries(cfg.get("job_queue", {}), window))
+    plan.extend(_build_simple_entries(cfg.get("spans", {}), "span", window, excluded))
+    plan.extend(
+        _build_simple_entries(cfg.get("rpc_methods", {}), "rpc", window, excluded)
+    )
+    plan.extend(_build_job_entries(cfg.get("job_queue", {}), window, excluded))
     return plan
 
 
diff --git a/docker/telemetry/workload/regression-metrics.json b/docker/telemetry/workload/regression-metrics.json
index 911868b595..dbc0ba055b 100644
--- a/docker/telemetry/workload/regression-metrics.json
+++ b/docker/telemetry/workload/regression-metrics.json
@@ -3,6 +3,11 @@
   "_key_format": "{category}.{name}.p{quantile}  (e.g. span.tx.process.p99, job.transaction.queued.p95). Only the categories defined below are captured; there is no rpc_methods group, so no rpc.* key is produced or gated (FU-4).",
   "_excluded_spans": "rpc.process is deliberately absent from spans.names. It is created only in ServerHandler::processRequest() on the HTTP/JSON-RPC path, which the workload load generators, being WebSocket-only, never reach, so its quantiles were captured as null every run and could never gate. (The harness shell scripts do issue a few HTTP JSON-RPC health polls, far too few to produce a meaningful quantile.) See baselines/README.md.",
   "_excluded_ledger_store": "ledger.store is deliberately absent from spans.names too, for a different reason: it is below the ladder's resolution. The 2026-08-24 capture returned p50/p95/p99 of exactly 0.005/0.0095/0.0099 ms, which is 0.5/0.95/0.99 x the ladder's first edge of 0.01 ms — the signature of every sample landing in the first bucket, so the numbers are interpolation arithmetic on the bucket floor rather than latencies. That is physically plausible: LedgerMaster.cpp:463 wraps an in-memory ledgerHistory_.insert, which completes in single-digit microseconds. While all mass stays under 10 us the reported quantile cannot move materially, so NO absolute bound can gate it — every ledger.store slowing from 2 us to 9 us, 4.5x, leaves the reported value unchanged. Three keys that read as covered but cannot fire are worse than no keys (the same argument that excluded rpc.process), so they were removed rather than left in with a bound that looks derived. Restoring the key needs sub-10us edges on the collector's spanmetrics ladder (for example 0.001ms and 0.005ms) plus the matching entries in HistogramBuckets.h — that is the ladder's branch, not this file. ledger.store presence is still asserted by expected_spans.json and docker/telemetry/integration-test.sh, and its rate is still on the ledger-operations dashboard; only the latency gate drops it.",
+  "_excluded_quantiles": "A THIRD KIND OF EXCLUSION, and the only one that deleting a name cannot express. spans.names lists span NAMES while _quantiles is shared across all of them, so the declared surface is the names x quantiles product and dropping ONE quantile of ONE span needs a subtraction. excluded_keys below is that subtraction: a flat {category}.{name}.p{quantile} key, exactly as _key_format defines it, mapped to the reason it is not gated. It can only ever remove a key, never add one, so a typo cannot silently start gating something new -- and check_regression_bounds.py rule F rejects an entry that would not otherwise be declared, an entry with an empty reason, and an entry that still carries a threshold override or a baseline value, so the exclusion cannot rot into dead config. Both prom_queries.py (which builds the capture plan) and check_regression_bounds.py (rule A) subtract it, so an excluded key is not queried, never reaches timings.json, and is not expected in the baseline. NOTHING ELSE CHANGES: the quantile is still computable from Prometheus with the _query_template above, the span is still asserted by expected_spans.json, and its rate is still on the ledger-operations dashboard. Only the latency gate drops it.",
+  "excluded_keys": {
+    "span.ledger.validate.p95": "Run-to-run variance is larger than the bound this ladder can derive. Measured across four CI runs the value spans 0.1281 to 0.7500 ms, a 5.9x spread, against a baseline of 0.2404 ms whose trip point is the next ladder edge at 0.5 ms -- so an ordinary run clears the trip point with nothing having regressed. Run 32867433073 read 0.7500 ms, +212%, and turned CI red. The derived bound models QUANTIZATION noise only (hi_next - baseline is one bucket of headroom); the dominant noise term for this span is peer-validation arrival timing in a 5-node cluster, and that term was never measured before the key was gated. Widening is not available: a bound that tolerated 0.7500 ms would reach past the 1 ms edge and leave the key gating nothing. This is a variance limit, not a defect and not a missing bound -- do NOT re-gate it by widening.",
+    "span.ledger.validate.p99": "The same mechanism as p95, two orders of magnitude worse. Across the same four runs the value spans 0.3875 to 25.8750 ms, a 66.8x spread, against a baseline of 1.0600 ms and a 10 ms trip point; run 32862589645 read 25.8750 ms, +2341%. The span opens only once a quorum-completing validation arrives (LedgerMaster.cpp:987, inside checkAccept, past the tvc < minVal early return) and wraps the promotion work that follows -- setValidated, setFull, setValidLedger, pendSaveValidated -- so its duration tracks peer-validation arrival timing and what promotion then triggers. One slow consensus round therefore dominates the tail of a 3m rate window, and which round that is differs every run. A bound tolerating 25.8750 ms would be ~24.8 ms against a 1.0600 ms baseline, which gates nothing at all. Note that the two CI failures landed on DIFFERENT quantiles in different runs while the other quantile stayed well inside its bound in the same run: that asymmetry is the signature of variance, not of a regression."
+  },
   "spans": {
     "_query_template": "histogram_quantile({quantile}, sum by (le) (rate(span_duration_milliseconds_bucket{span_name=\"{name}\"}[{window}])))",
     "_unit": "ms",
diff --git a/docker/telemetry/workload/regression-thresholds.json b/docker/telemetry/workload/regression-thresholds.json
index ada4245ff5..7ea8672996 100644
--- a/docker/telemetry/workload/regression-thresholds.json
+++ b/docker/telemetry/workload/regression-thresholds.json
@@ -1,8 +1,8 @@
 {
   "_description": "Per-metric regression thresholds. A metric regresses when current - baseline exceeds BOTH the percentage and absolute bounds (AND, not OR \u2014 this tolerates small-value noise). Defaults apply unless a per-metric override exists.",
   "_bucket_note": "SpanMetrics latency histograms use explicit buckets [0.01,0.05,0.1,0.25,0.5,1,5,10,25,50,100,250,500]ms then [1,2,3,4,5,10,30]s (20 edges; docker/telemetry/otel-collector-config.yaml is the authoritative list). Second-scale consensus spans have 2s/3s/4s boundaries, so their quantiles quantize to ~1s widths there \u2014 the ladder is NOT uniformly 2x-or-coarser, which matters for _percentage_bound_note. The native job_queue histograms are microsecond-valued on the ladder [1,2,5,10,25,50,100,250,500,1000,5000,25000,100000,500000]us then [1,5,10,30,60]s (19 edges; include/xrpl/telemetry/HistogramBuckets.h is authoritative). NOTE: BOTH ladders were re-cut, and a baseline captured before its own ladder changed is an interpolation artefact, not a latency. The job_queue floor moved 100us \u2192 1us. The span ladder was re-cut on 2026-08-04 in 3860c93db2, moving the floor 1ms \u2192 0.01ms; so any sub-millisecond span quantile captured before that date is equally void \u2014 a p95 reading 0.95ms is 0.95 \u00d7 the old 1ms first edge, not a measurement. An earlier note asserted that the surviving span baselines were unaffected by the ladder work; that is wrong for every span quantile below 1ms. Only the band from 1ms to 1s is safe: those edges are byte-identical across the two ladders. The re-cut also ADDED edges above 1s (2s/3s/4s/10s/30s), so a span whose quantiles land in the second-scale range \u2014 consensus.round ~3.9s, consensus.establish ~1.9s, the ledger.acquire tail \u2014 is distorted just as much, and any pre-2026-08-04 baseline for it is equally void. Do not read this note as licensing a stale second-scale baseline.",
-  "_absolute_bound_derivation": "HOW EVERY max_abs_increase_* NUMBER BELOW WAS OBTAINED. Rule: locate the baseline value in the half-open bucket (lo, hi] of its own ladder, take hi_next = the next edge above hi, and set the bound to (hi_next - baseline). The trip point is therefore exactly hi_next: the gate fires only when the reported value EXCEEDS the top of the bucket above the baseline's own bucket. WHY THAT AND NOT A MULTIPLE OF THE BUCKET WIDTH: histogram_quantile returns a value interpolated inside whichever bucket the true quantile falls in, so a reading taken while the true quantile sits anywhere in the baseline's bucket OR anywhere in the one immediately above is at most hi_next and cannot fire. Firing requires the true quantile to have moved at least two buckets up. A multiple of the ENCLOSING width cannot deliver that, because once the quantile crosses hi the interpolation happens across the NEXT bucket, which on this ladder is up to 8x wider \u2014 (0.5,1] has width 0.5 and (1,5] has width 4 \u2014 so the reading's excursion is not bounded by any multiple of the enclosing width. Worked example: span.tx.process.p99 has baseline 0.9945ms in bucket (0.5, 1], hi_next = 5, so its bound is 4.0055ms and the gate fires only above 5ms. Bounds are stored as exact doubles rather than rounded figures so that rounding cannot break the guarantee and so check_regression_bounds.py can assert each one against the ladder to within a 1e-12 relative tolerance -- tight enough that a bound rounded for readability, such as 4.0055 for 4.005485184848892, is rejected; _derivation_table below shows the arithmetic for each one. Measured over the committed baseline this rule yields a detection floor of 2.02x to 9.43x of baseline, per key. WHAT THIS REPLACED, IN TWO GENERATIONS: (1) a single flat pair of bounds (10ms for span p50/p95, 15ms for span p99, 20000us for job_queue p95) justified as 'roughly two bucket widths in the 5-25ms band where most span quantiles actually sit'. The 2026-08-24 capture falsifies that premise \u2014 18 of the 28 quantiles gated at that time sat below 1ms \u2014 so the absolute bound sat 1.15x to 2000x above the metric it guarded and, because the rule is an AND, the percentage bound could never carry a regression on its own; a 10x regression injected into each key in turn was caught on only 5 of 28, and a 100x regression injected into span.ledger.store.p95 produced 0 regressions and exit 0. (2) a first correction to 2 \u00d7 the ENCLOSING bucket width, which caught 10x on 28 of 28 but placed the trip point INSIDE the adjacent bucket -- and so left a single-crossing false positive reachable -- on 21 of the 25 keys gated today, 4 of them tripping on a tail-mass shift under 1.5% of samples. That is the assumption this rule removes. RE-DERIVE THESE NUMBERS whenever baseline-timings.json is refreshed or either ladder changes: a refreshed baseline can land in a different bucket, which changes hi_next. .github/scripts/telemetry/check_regression_bounds.py enforces the rule in CI so a stale bound cannot survive a baseline refresh. LIMITATION \u2014 WHICH KEYS ARE ONLY WEAKLY GUARDED: the guarantee costs sensitivity wherever the ladder is coarse, and the detection floor is hi_next/baseline, so a baseline sitting just above an edge is guarded loosely. span.ledger.build.p50 (baseline 1.0612ms, fires at 10ms, 9.42x) and span.ledger.validate.p99 (1.0600ms, 9.43x) are NOT meaningfully guarded: ledger.build p50 fires only at 10ms, so it could get up to 9.4x slower -- 1.06ms to just under 10ms -- and still pass. span.tx.process.p95 (6.91x), span.tx.apply.p50 (6.32x), span.rpc.ws_message.p95 (5.92x), job.acceptLedger.running.p95 (5.74x), span.consensus.accept.p50 (5.74x), span.consensus.ledger_close.p99 (5.37x), span.rpc.ws_message.p99 (5.06x) and span.tx.process.p99 (5.03x) are weak. All ten are limited by two 5x-wide ladder steps, 1ms\u21925ms and 5000us\u219225000us. The fix is a 2ms edge (and ideally 3ms) in the collector's spanmetrics ladder plus the matching edges in kMillisecondBuckets, and a 10000us edge in kMicrosecondBuckets \u2014 that work belongs to the branch that owns the ladders, not here. Until then do not read these keys as guarded. span.ledger.store is absent from the overrides below because it was removed from the gated surface entirely: its quantiles were the ladder floor times the quantile, so no bound could gate it. See _excluded_ledger_store in regression-metrics.json.",
-  "_percentage_bound_note": "For every key gated today the absolute bound is the binding half of the AND and the percentage bound never decides the outcome: measured, (bound / baseline) ranges from 102% (span.tx.apply.p99) to 843% (span.ledger.validate.p99), all above the 50% and 5% percentage bounds configured here, and the minimum trip multiple of all 25 keys is set by the absolute bound. THIS IS NOT A GENERAL GUARANTEE, and an earlier version of this note wrongly claimed it was, on the false premise that 'every step of both ladders is at least a factor of 2'. The span ladder breaks that three times at the top: 2s->3s is 1.5x, 3s->4s is 1.33x, 4s->5s is 1.25x, so second-scale consensus quantiles quantize to ~1s widths there. Because the bound is (hi_next - baseline), a baseline between about 2667ms and 3000ms, or between about 3334ms and 4000ms, gets an absolute bound worth less than 50% of itself and the PERCENTAGE bound becomes the operative one -- at which point the metric fires on a 50% move that is smaller than one bucket width, and the single-crossing guarantee in _absolute_bound_derivation is lost. That band is not hypothetical: the collector config names consensus.round (~3.9s) as a reason those edges exist, and 3900ms sits in the second sub-band with an absolute bound of 5000 - 3900 = 1100, only 28.2% of baseline. Whoever gates a key whose baseline lands in either sub-band MUST lower its max_pct_increase below (bound / baseline) for that key, or state explicitly that the metric is percentage-gated and the bucket guarantee does not hold for it. check_regression_bounds.py enforces this as rule D so the trap cannot be walked into silently. The percentage entries are required and still meaningful regardless: compare_to_baseline.py treats a missing max_pct_increase as 'no threshold configured' and would stop gating the metric entirely; they record the intended relative tolerance (consensus spans 5%, everything else 50%); and they are the operative bound on the defaults path (see _defaults_note).",
+  "_absolute_bound_derivation": "HOW EVERY max_abs_increase_* NUMBER BELOW WAS OBTAINED. Rule: locate the baseline value in the half-open bucket (lo, hi] of its own ladder, take hi_next = the next edge above hi, and set the bound to (hi_next - baseline). The trip point is therefore exactly hi_next: the gate fires only when the reported value EXCEEDS the top of the bucket above the baseline's own bucket. WHY THAT AND NOT A MULTIPLE OF THE BUCKET WIDTH: histogram_quantile returns a value interpolated inside whichever bucket the true quantile falls in, so a reading taken while the true quantile sits anywhere in the baseline's bucket OR anywhere in the one immediately above is at most hi_next and cannot fire. Firing requires the true quantile to have moved at least two buckets up. A multiple of the ENCLOSING width cannot deliver that, because once the quantile crosses hi the interpolation happens across the NEXT bucket, which on this ladder is up to 8x wider \u2014 (0.5,1] has width 0.5 and (1,5] has width 4 \u2014 so the reading's excursion is not bounded by any multiple of the enclosing width. Worked example: span.tx.process.p99 has baseline 0.9945ms in bucket (0.5, 1], hi_next = 5, so its bound is 4.0055ms and the gate fires only above 5ms. Bounds are stored as exact doubles rather than rounded figures so that rounding cannot break the guarantee and so check_regression_bounds.py can assert each one against the ladder to within a 1e-12 relative tolerance -- tight enough that a bound rounded for readability, such as 4.0055 for 4.005485184848892, is rejected; _derivation_table below shows the arithmetic for each one. Measured over the committed baseline this rule yields a detection floor of 2.02x to 9.42x of baseline, per key. WHAT THIS RULE DOES NOT COVER, AND THE ONE CHECK TO RUN BEFORE GATING ANY KEY: hi_next - baseline is derived from the LADDER, so it budgets for QUANTIZATION noise -- one bucket of interpolation headroom -- and for nothing else. It knows nothing about how much the metric itself moves between runs on identical code. Where run-to-run workload variance is the larger term the bound is simply the wrong size, and the gate reddens on a healthy run. So before adding a key here, capture it over several runs and check its OBSERVED MAXIMUM against its trip point (baseline + bound); gate it only if the observed maximum stays below that trip point with margin. Spread on its own proves nothing -- span.tx.apply.p50 swings 364x across runs and never fires, because its 5ms trip point absorbs the whole range -- it is spread RELATIVE TO THE TRIP POINT that decides. Measured across the runs behind this baseline, the worst surviving key reaches 0.67 of its trip point (span.consensus.ledger_close.p95), and the only two that exceeded it were span.ledger.validate p95 and p99, now excluded. A key that fails this test is not fixed by widening its bound: see excluded_keys in regression-metrics.json. WHAT THIS REPLACED, IN TWO GENERATIONS: (1) a single flat pair of bounds (10ms for span p50/p95, 15ms for span p99, 20000us for job_queue p95) justified as 'roughly two bucket widths in the 5-25ms band where most span quantiles actually sit'. The 2026-08-24 capture falsifies that premise \u2014 18 of the 28 quantiles gated at that time sat below 1ms \u2014 so the absolute bound sat 1.15x to 2000x above the metric it guarded and, because the rule is an AND, the percentage bound could never carry a regression on its own; a 10x regression injected into each key in turn was caught on only 5 of 28, and a 100x regression injected into span.ledger.store.p95 produced 0 regressions and exit 0. (2) a first correction to 2 \u00d7 the ENCLOSING bucket width, which caught 10x on 28 of 28 but placed the trip point INSIDE the adjacent bucket -- and so left a single-crossing false positive reachable -- on 21 of the 25 keys gated at the time, 4 of them tripping on a tail-mass shift under 1.5% of samples. That is the assumption this rule removes. RE-DERIVE THESE NUMBERS whenever baseline-timings.json is refreshed or either ladder changes: a refreshed baseline can land in a different bucket, which changes hi_next. .github/scripts/telemetry/check_regression_bounds.py enforces the rule in CI so a stale bound cannot survive a baseline refresh. LIMITATION \u2014 WHICH KEYS ARE ONLY WEAKLY GUARDED: the guarantee costs sensitivity wherever the ladder is coarse, and the detection floor is hi_next/baseline, so a baseline sitting just above an edge is guarded loosely. span.ledger.build.p50 (baseline 1.0612ms, fires at 10ms, 9.42x) is NOT meaningfully guarded: it fires only at 10ms, so it could get up to 9.4x slower -- 1.06ms to just under 10ms -- and still pass. span.tx.process.p95 (6.91x), span.tx.apply.p50 (6.32x), span.rpc.ws_message.p95 (5.92x), job.acceptLedger.running.p95 (5.74x), span.consensus.accept.p50 (5.74x), span.consensus.ledger_close.p99 (5.37x), span.rpc.ws_message.p99 (5.06x) and span.tx.process.p99 (5.03x) are weak. All nine are limited by two 5x-wide ladder steps, 1ms\u21925ms and 5000us\u219225000us. The fix is a 2ms edge (and ideally 3ms) in the collector's spanmetrics ladder plus the matching edges in kMillisecondBuckets, and a 10000us edge in kMicrosecondBuckets \u2014 that work belongs to the branch that owns the ladders, not here. Until then do not read these keys as guarded. span.ledger.store is absent from the overrides below because it was removed from the gated surface entirely: its quantiles were the ladder floor times the quantile, so no bound could gate it. See _excluded_ledger_store in regression-metrics.json.",
+  "_percentage_bound_note": "For every key gated today the absolute bound is the binding half of the AND and the percentage bound never decides the outcome: measured, (bound / baseline) ranges from 102% (span.tx.apply.p99) to 842% (span.ledger.build.p50), all above the 50% and 5% percentage bounds configured here, and the minimum trip multiple of all 23 keys is set by the absolute bound. THIS IS NOT A GENERAL GUARANTEE, and an earlier version of this note wrongly claimed it was, on the false premise that 'every step of both ladders is at least a factor of 2'. The span ladder breaks that three times at the top: 2s->3s is 1.5x, 3s->4s is 1.33x, 4s->5s is 1.25x, so second-scale consensus quantiles quantize to ~1s widths there. Because the bound is (hi_next - baseline), a baseline between about 2667ms and 3000ms, or between about 3334ms and 4000ms, gets an absolute bound worth less than 50% of itself and the PERCENTAGE bound becomes the operative one -- at which point the metric fires on a 50% move that is smaller than one bucket width, and the single-crossing guarantee in _absolute_bound_derivation is lost. That band is not hypothetical: the collector config names consensus.round (~3.9s) as a reason those edges exist, and 3900ms sits in the second sub-band with an absolute bound of 5000 - 3900 = 1100, only 28.2% of baseline. Whoever gates a key whose baseline lands in either sub-band MUST lower its max_pct_increase below (bound / baseline) for that key, or state explicitly that the metric is percentage-gated and the bucket guarantee does not hold for it. check_regression_bounds.py enforces this as rule D so the trap cannot be walked into silently. The percentage entries are required and still meaningful regardless: compare_to_baseline.py treats a missing max_pct_increase as 'no threshold configured' and would stop gating the metric entirely; they record the intended relative tolerance (consensus spans 5%, everything else 50%); and they are the operative bound on the defaults path (see _defaults_note).",
   "_defaults_note": "A MISSING OVERRIDE IS DETECTED BY CI, NOT BY THESE DEFAULTS. .github/scripts/telemetry/check_regression_bounds.py fails the build at lint time, naming the key and the exact value its bound should have, before the workload ever runs. That is the mechanism; the defaults below are only a runtime backstop for the case where that check is bypassed. The defaults carry the FLOOR of each ladder as their absolute bound \u2014 0.01ms for spans, 1us for job_queue \u2014 deliberately too small to bind for any real metric, which leaves max_pct_increase (50%) as the operative bound on this path. Measured: a metric with no override and a baseline of 3900ms passes at +49% and fires at +51%; a job metric with a baseline of 5000us behaves the same. The backstop is honestly imperfect and the earlier version of this note oversold it. At 50% relative it CAN false-fire: a metric whose baseline is 1.06ms inside the 4ms-wide (1,5] bucket fires on a single-bucket-width move (measured: 1.06 \u2192 5.06ms, +377%, regressed). An earlier note called that 'the intended signal that the override is missing', which was wrong \u2014 CI prints REGRESSION and a reader cannot tell it from a real one, and rejecting a tighter alternative for exactly that cries-wolf risk while shipping it here would be inconsistent. The check is what makes the signal legible. The backstop is kept only because a metric silently not gated at all is the worse of the two failures.",
   "_derivation_table": {
     "_format": "override key: <quantile> <baseline> in <bucket> -> hi_next - baseline = <bound>",
@@ -13,7 +13,7 @@
     "span.consensus.accept": "p50 1.7435897435897438 in (1,5] -> hi_next 10 - baseline = 8.256410256410255 | p95 8.9296875 in (5,10] -> hi_next 25 - baseline = 16.0703125 | p99 15.150000000000082 in (10,25] -> hi_next 50 - baseline = 34.849999999999916",
     "span.consensus.ledger_close": "p50 0.15142857142857144 in (0.1,0.25] -> hi_next 0.5 - baseline = 0.34857142857142853 | p95 0.49328358208955236 in (0.25,0.5] -> hi_next 1 - baseline = 0.5067164179104476 | p99 0.9314285714285726 in (0.5,1] -> hi_next 5 - baseline = 4.068571428571428",
     "span.ledger.build": "p50 1.0612244897959187 in (1,5] -> hi_next 10 - baseline = 8.938775510204081 | p95 4.679591836734694 in (1,5] -> hi_next 10 - baseline = 5.320408163265306 | p99 5.075000000000041 in (5,10] -> hi_next 25 - baseline = 19.924999999999958",
-    "span.ledger.validate": "p50 0.07787769784172663 in (0.05,0.1] -> hi_next 0.25 - baseline = 0.17212230215827337 | p95 0.2404310344827586 in (0.1,0.25] -> hi_next 0.5 - baseline = 0.2595689655172414 | p99 1.060000000000033 in (1,5] -> hi_next 10 - baseline = 8.939999999999968",
+    "span.ledger.validate": "p50 0.07787769784172663 in (0.05,0.1] -> hi_next 0.25 - baseline = 0.17212230215827337   (p95 and p99 are NOT gated -- see excluded_keys in regression-metrics.json: their run-to-run spread, 5.9x and 66.8x over four CI runs, exceeds any bound this rule can derive)",
     "span.rpc.ws_message": "p50 0.15990942879094971 in (0.1,0.25] -> hi_next 0.5 - baseline = 0.3400905712090503 | p95 0.8443074003795064 in (0.5,1] -> hi_next 5 - baseline = 4.155692599620494 | p99 0.9878368121442125 in (0.5,1] -> hi_next 5 - baseline = 4.012163187855787",
     "span.tx.apply": "p50 0.791666666666667 in (0.5,1] -> hi_next 5 - baseline = 4.208333333333333 | p95 4.627368421052632 in (1,5] -> hi_next 10 - baseline = 5.372631578947368 | p99 4.959157894736842 in (1,5] -> hi_next 10 - baseline = 5.040842105263158",
     "span.tx.process": "p50 0.34281067382135394 in (0.25,0.5] -> hi_next 1 - baseline = 0.6571893261786461 | p95 0.7239865112994349 in (0.5,1] -> hi_next 5 - baseline = 4.276013488700565 | p99 0.9945148151511076 in (0.5,1] -> hi_next 5 - baseline = 4.005485184848892"
@@ -111,14 +111,6 @@
       "p50": {
         "max_pct_increase": 50.0,
         "max_abs_increase_ms": 0.17212230215827337
-      },
-      "p95": {
-        "max_pct_increase": 50.0,
-        "max_abs_increase_ms": 0.2595689655172414
-      },
-      "p99": {
-        "max_pct_increase": 50.0,
-        "max_abs_increase_ms": 8.939999999999968
       }
     },
     "span.rpc.ws_message": {
diff --git a/docs/telemetry-runbook.md b/docs/telemetry-runbook.md
index de70a911d3..17303ac9d3 100644
--- a/docs/telemetry-runbook.md
+++ b/docs/telemetry-runbook.md
@@ -3729,12 +3729,21 @@ Key properties:
   metric it guarded, and a 10x regression injected into each key in turn was
   caught on only 5 of 28.
 - **The detection floor is `hi_next / baseline`, so some keys are only weakly
-  guarded.** It ranges 2.02x to 9.43x over the current baseline;
-  `span.ledger.build.p50` and `span.ledger.validate.p99` are effectively
-  not guarded at 9.4x. `baselines/README.md` lists all ten weak keys and the
-  ladder edges that would fix them.
+  guarded.** It ranges 2.02x to 9.42x over the current baseline;
+  `span.ledger.build.p50` is effectively not guarded at 9.4x.
+  `baselines/README.md` lists all nine weak keys and the ladder edges that would
+  fix them.
+- **The bound covers quantization noise only, so a key whose run-to-run variance
+  exceeds it cannot be gated.** `span.ledger.validate.p95` and `.p99` are
+  excluded for exactly that reason — measured spreads of 5.9x and 66.8x across
+  four CI runs, both reaching past their trip points on healthy runs, because
+  the span's duration follows peer-validation arrival timing rather than code
+  speed. Widening their bounds would gate nothing, so they are listed in
+  `excluded_keys` in `regression-metrics.json` and `check_regression_bounds.py`
+  rule F keeps that exclusion honest. Before gating any key, check its observed
+  maximum across runs against `baseline + bound`; see `baselines/README.md`.
 - **For every currently gated metric the absolute bound decides; the percentage
-  bound does not.** Measured, the bound is 102%-843% of its own baseline, above
+  bound does not.** Measured, the bound is 102%-842% of its own baseline, above
   both configured percentage bounds. This is _not_ a general property: the span
   ladder's top steps are only 1.25x-1.5x apart, so a baseline between about
   2667-3000 ms or 3334-4000 ms gets an absolute bound worth under 50% of itself