ARCHIVE/rippled
1
0
Fork 0
mirror of https://github.com/XRPLF/rippled.git synced 2026-08-22 23:00:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

fix(telemetry): correct job latency units and name dashboard rate nouns

Browse Source 
The four job-latency panels on node-health declared milliseconds while
querying `job_running_us` / `job_queued_us`, which record microseconds
(MetricsRegistry records the raw value, and the instrument description
says microseconds). Every reading was therefore a thousand times too
large: the p95 for acceptLedger, 241495us, rendered as "241 sec"
instead of 241ms. job-queue.json already read these same metrics as
microseconds, so the two dashboards disagreed by 1000x on identical
data. Switch node-health to microseconds to match.

Also replace the generic `ops` and `cps` units, which Grafana renders
as the literal "ops/s" and "counts/s", with custom-suffix units naming
what each panel counts -- messages, fetches, calls, mismatches.

Two panels plot more than one quantity on a single axis, which no
single unit can describe. Give each series its own unit through field
overrides: reads per second beside two queue depths on NuDB Read
Pressure, and ledgers beside fetches on Ledger Close Rate, the latter
on a right-hand axis.

State Duration Rate plots a seconds-per-second time share, which can
exceed 1.0 and so is not a percentage; label it as the ratio it is.
The normalised share already exists as its own panel.

Queries are unchanged; the values were already correct.

Alongside, bring the touched panels up to the dashboard guidelines and
hoist the stat panels above the fold.
This commit is contained in:
Pratik Mankawde
2026-07-30 18:06:23 +01:00
parent 94ca33214c
commit 2095a3d2f1
6 changed files with 168 additions and 97 deletions
Download Patch File Download Diff File Expand all files Collapse all files

40
docs/telemetry-runbook.md
View File

@@ -718,24 +718,24 @@ Requires `trace_peer=1` in the `[telemetry]` config section.
### Node Health -- System Metrics (`node-health`)
| Panel | Type | PromQL | Labels Used |
| -------------------------------------- | ---------- | ---------------------------------------------------------- | ----------- |
| Validated Ledger Age | stat | `ledgermaster_validated_ledger_age` | — |
| Published Ledger Age | stat | `ledgermaster_published_ledger_age` | — |
| Operating Mode (Time Share) | timeseries | `rate(state_accounting_X_duration) / sum(rate(all modes))` | — |
| Operating Mode Transitions | timeseries | `state_accounting_*_transitions` | — |
| I/O Latency | timeseries | `histogram_quantile(0.95, ios_latency_bucket)` | — |
| Job Queue Depth | timeseries | `job_count` | — |
| Ledger Fetch Rate | stat | `rate(ledger_fetches[5m])` | — |
| Ledger History Mismatches | stat | `rate(ledger_history_mismatch[5m])` | — |
| Key Jobs Execution Time | timeseries | `acceptledger{quantile="$quantile"}` (+ 10 more key jobs) | `quantile` |
| Key Jobs Dequeue Wait Time | timeseries | `acceptledger_q{quantile="$quantile"}` (+ 10 more) | `quantile` |
| FullBelowCache Size | timeseries | `node_family_full_below_cache_size` | — |
| FullBelowCache Hit Rate | gauge | `node_family_full_below_cache_hit_rate` | — |
| Ledger Publish Gap | stat | `Published_Ledger_Age - Validated_Ledger_Age` | — |
| State Duration Rate (Full vs Tracking) | timeseries | `rate(state_accounting_full_duration[5m]) / 1000000` | — |
| All Jobs Execution Time (Detail) | timeseries | `{__name__=~"<all_jobs>", quantile="$quantile"}` | `quantile` |
| All Jobs Dequeue Wait (Detail) | timeseries | `{__name__=~"<all_jobs>_q", quantile="$quantile"}` | `quantile` |
| Panel | Type | PromQL | Labels Used |
| -------------------------------------- | ---------- | --------------------------------------------------------------------------------- | ----------- |
| Validated Ledger Age | stat | `ledgermaster_validated_ledger_age` | — |
| Published Ledger Age | stat | `ledgermaster_published_ledger_age` | — |
| Operating Mode (Time Share) | timeseries | `rate(state_accounting_X_duration) / sum(rate(all modes))` | — |
| Operating Mode Transitions | timeseries | `state_accounting_*_transitions` | — |
| I/O Latency | timeseries | `histogram_quantile(0.95, ios_latency_bucket)` | — |
| Job Queue Depth | timeseries | `job_count` | — |
| Ledger Fetch Rate | stat | `rate(ledger_fetches_total[$__rate_interval])` | — |
| Ledger History Mismatches | stat | `rate(ledger_history_mismatch_total[$__rate_interval])` | — |
| Key Jobs Execution Time | timeseries | `acceptledger{quantile="$quantile"}` (+ 10 more key jobs) | `quantile` |
| Key Jobs Dequeue Wait Time | timeseries | `acceptledger_q{quantile="$quantile"}` (+ 10 more) | `quantile` |
| FullBelowCache Size | timeseries | `node_family_full_below_cache_size` | — |
| FullBelowCache Hit Rate | gauge | `node_family_full_below_cache_hit_rate` | — |
| Ledger Publish Gap | stat | `Published_Ledger_Age - Validated_Ledger_Age` | — |
| State Duration Rate (Full vs Tracking) | timeseries | `rate(state_accounting_full_duration[5m]) / 1000000` | — |
| All Jobs Execution Time (Detail) | timeseries | `histogram_quantile($quantile, rate(job_running_us_bucket[5m])) by job_type` — µs | `quantile` |
| All Jobs Dequeue Wait (Detail) | timeseries | `histogram_quantile($quantile, rate(job_queued_us_bucket[5m])) by job_type` — µs | `quantile` |
### Network Traffic -- System Metrics (`network-traffic`)
@@ -762,8 +762,8 @@ Requires `trace_peer=1` in the `[telemetry]` config section.
| RPC Response Time Heatmap | heatmap | `rpc_time_bucket` | — |
| Pathfinding Fast Duration | timeseries | `histogram_quantile(0.95, pathfind_fast_bucket)` | — |
| Pathfinding Full Duration | timeseries | `histogram_quantile(0.95, pathfind_full_bucket)` | — |
| Resource Warnings Rate | stat | `rate(warn[5m])` | — |
| Resource Drops Rate | stat | `rate(drop[5m])` | — |
| Resource Warnings Rate | stat | `rate(warn_total[$__rate_interval])` | — |
| Resource Drops Rate | stat | `rate(drop_total[$__rate_interval])` | — |
### Span → Metric → Dashboard Summary