Files
rippled/include/xrpl/core/Job.h
Pratik Mankawde 15596f5b8d feat(telemetry): pinpoint root cause of slow TMGetObjectByHash service
Slowness on the peer object-fetch path could be observed but not
attributed. Job duration metrics carry only `job_type`, and both
`RcvGetLedger` and `RcvGetObjByHash` report as `ledgerRequest`, so a
queue-wait spike could not be traced to a handler. Nothing measured
NodeStore cost, request size, or the differential charge.

Latency now decomposes into three additive parts, each separately
measurable:

    end-to-end = queue wait + NodeStore lookup + everything else

- `handler` label on job_queued_total/_started_total/_finished_total and
  job_queued_us/job_running_us. The value is sanitised: a name passes
  through only if non-empty and all ASCII letters, else "other". Two job
  names embed a ledger sequence, so a raw label would mint one series
  per ledger; the rule bounds the domain at 43 names plus "other".
- getobject_lookup_us, _request_objects, _lookups_total{result},
  _rejected_total{reason} and _charge, recorded at their call sites.
  All three histograms get explicit bucket views: the SDK default stops
  at 10,000, which every one of them exceeds.
- Per-job-type waiting/running/deferred gauges for the 35 non-special
  job types. `deferred` is the leading indicator, since addJob never
  rejects -- it defers, so backpressure otherwise shows up only as
  latency after the fact.

`JobQueue::collect()` snapshots the counters under the queue lock and
publishes gauges after releasing it. Writing them while holding the lock
would invert a lock order against the collector's own lock, which the
collector's flush thread already holds when it calls this hook.

Tests assert exact values, including that the charge is priced on the
requested count rather than the capped iteration count.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-25 11:59:16 +01:00

158 lines
5.2 KiB
C++

#pragma once
#include <xrpl/basics/CountedObject.h>
#include <xrpl/core/ClosureCounter.h>
#include <xrpl/core/LoadEvent.h>
#include <xrpl/core/LoadMonitor.h>
#include <chrono>
#include <cstdint>
#include <functional>
#include <memory>
#include <string>
namespace xrpl {
// Note that this queue should only be used for CPU-bound jobs
// It is primarily intended for signature checking
// Protocol-wide
// NOLINTNEXTLINE(cppcoreguidelines-use-enum-class)
enum JobType {
// Special type indicating an invalid job - will go away soon.
JtInvalid = -1,
// Job types - the position in this enum indicates the job priority with
// earlier jobs having lower priority than later jobs. If you wish to
// insert a job at a specific priority, simply add it at the right location.
JtPack, // Make a fetch pack for a peer
JtPuboldledger, // An old ledger has been accepted
JtClient, // A placeholder for the priority of all jtCLIENT jobs
JtClientSubscribe, // A websocket subscription by a client
JtClientFeeChange, // Subscription for fee change by a client
JtClientConsensus, // Subscription for consensus state change by a client
JtClientAcctHist, // Subscription for account history by a client
JtClientRpc, // Client RPC request
JtClientWebsocket, // Client websocket request
JtRpc, // A websocket command from the client
JtSweep, // Sweep for stale structures
JtValidationUt, // A validation from an untrusted source
JtManifest, // A validator's manifest
JtUpdatePf, // Update pathfinding requests
JtTransactionL, // A local transaction
JtReplayReq, // Peer request a ledger delta or a skip list
JtLedgerReq, // Peer request ledger/txnset data
JtProposalUt, // A proposal from an untrusted source
JtReplayTask, // A Ledger replay task/subtask
JtTransaction, // A transaction received from the network
JtMissingTxn, // Request missing transactions
JtRequestedTxn, // Reply with requested transactions
JtBatch, // Apply batched transactions
JtLedgerData, // Received data for a ledger we're acquiring
JtAdvance, // Advance validated/acquired ledgers
JtPubledger, // Publish a fully-accepted ledger
JtTxnData, // Fetch a proposed set
JtWal, // Write-ahead logging
JtValidationT, // A validation from a trusted source
JtWrite, // Write out hashed objects
JtAccept, // Accept a consensus ledger
JtProposalT, // A proposal from a trusted source
JtNetopCluster, // NetworkOPs cluster peer report
JtNetopTimer, // NetworkOPs net timer processing
JtAdmin, // An administrative operation
// Special job types which are not dispatched by the job pool
JtPeer,
JtDisk,
JtTxnProc,
JtObSetup,
JtPathFind,
JtHoRead,
JtHoWrite,
JtGeneric, // Used just to measure time
// Node store monitoring
JtNsSyncRead,
JtNsAsyncRead,
JtNsWrite,
};
class Job : public CountedObject<Job>
{
public:
using clock_type = std::chrono::steady_clock;
/**
* Default constructor.
*
* Allows Job to be used as a container type.
*
* This is used to allow things like jobMap [key] = value.
*/
// VFALCO NOTE I'd prefer not to have a default constructed object.
// What is the semantic meaning of a Job with no associated
// function? Having the invariant "all Job objects refer to
// a job" would reduce the number of states.
//
Job();
Job(JobType type, std::uint64_t index);
// VFALCO TODO try to remove the dependency on LoadMonitor.
Job(JobType type,
std::string const& name,
std::uint64_t index,
LoadMonitor& lm,
std::function<void()> const& job);
[[nodiscard]] JobType
getType() const;
/**
* Returns the job name supplied to JobQueue::addJob.
*
* Several job types have more than one producer (for example both
* RcvGetLedger and RcvGetObjByHash run as JtLedgerReq), so the name is
* the only thing that tells them apart. It is read by the JobQueue
* PerfLog hooks and exported as the `handler` metric label.
*
* Empty for the default and index-only constructors, which carry no
* name; callers must treat an empty name as "unknown".
*/
[[nodiscard]] std::string const&
getName() const;
/**
* Returns the time when the job was queued.
*/
[[nodiscard]] clock_type::time_point const&
queueTime() const;
void
doJob();
// These comparison operators make the jobs sort in priority order
// in the job set
bool
operator<(Job const& j) const;
bool
operator>(Job const& j) const;
bool
operator<=(Job const& j) const;
bool
operator>=(Job const& j) const;
private:
JobType type_;
std::uint64_t jobIndex_;
std::function<void()> job_;
std::shared_ptr<LoadEvent> loadEvent_;
std::string name_;
clock_type::time_point queueTime_;
};
using JobCounter = ClosureCounter<void>;
} // namespace xrpl