rippled/OpenTelemetryPlan/Phase5_IntegrationTest_taskList.md

Phase 5: Integration Test Task List

Goal: End-to-end verification of the complete telemetry pipeline using a 6-node consensus network. Proves that RPC, transaction, and consensus spans flow through the observability stack (otel-collector, Jaeger, Prometheus, Grafana) under realistic conditions.

Scope: Integration test script, manual testing plan, 6-node local network setup, Jaeger/Prometheus/Grafana verification.

Branch: pratik/otel-phase5-docs-deployment

Related Plan Documents

Document	Relevance
07-observability-backends.md	Jaeger, Grafana, Prometheus setup
05-configuration-reference.md	Collector config, Docker Compose
06-implementation-phases.md	Phase 5 tasks, definition of done
Phase5_taskList.md	Phase 5 main task list (5.6 = integration)

---

Task IT.1: Create Integration Test Script

Objective: Automated bash script that stands up a 6-node xrpld network with telemetry, exercises all span categories, and verifies data in Jaeger/Prometheus.

What to do:

• Create docker/telemetry/integration-test.sh:
  • Prerequisites check (docker, xrpld binary, curl, jq)
  • Start observability stack via docker compose
  • Generate 6 validator key pairs via temp standalone xrpld
  • Generate 6 node configs + shared validators.txt
  • Start 6 xrpld nodes in consensus mode (--start, no -a)
  • Wait for all nodes to reach "proposing" state (120s timeout)

Key new file: docker/telemetry/integration-test.sh

Verification:

• Script starts without errors
• All 6 nodes reach "proposing" state
• Observability stack is healthy (otel-collector, Jaeger, Prometheus, Grafana)

---

Task IT.2: RPC Span Verification (Phase 2)

Objective: Verify RPC spans flow through the telemetry pipeline.

What to do:

• Send server_info, server_state, ledger RPCs to node1 (port 5005)
• Wait for batch export (5s)
• Query Jaeger API for:
  • rpc.request spans (ServerHandler::onRequest)
  • rpc.process spans (ServerHandler::processRequest)
  • rpc.command.server_info spans (callMethod)
  • rpc.command.server_state spans (callMethod)
  • rpc.command.ledger spans (callMethod)
• Verify xrpl.rpc.command attribute present on rpc.command.* spans

Verification:

• Jaeger shows rpc.request traces
• Jaeger shows rpc.process traces
• Jaeger shows rpc.command.* traces with correct attributes

---

Task IT.3: Transaction Span Verification (Phase 3)

Objective: Verify transaction spans flow through the telemetry pipeline.

What to do:

• Get genesis account sequence via account_info RPC
• Submit Payment transaction using genesis seed (snoPBrXtMeMyMHUVTgbuqAfg1SUTb)
• Wait for consensus inclusion (10s)
• Query Jaeger API for:
  • tx.process spans (NetworkOPsImp::processTransaction) on submitting node
  • tx.receive spans (PeerImp::handleTransaction) on peer nodes
• Verify xrpl.tx.hash attribute on tx.process spans
• Verify xrpl.peer.id attribute on tx.receive spans

Verification:

• Jaeger shows tx.process traces with xrpl.tx.hash
• Jaeger shows tx.receive traces with xrpl.peer.id

---

Task IT.4: Consensus Span Verification (Phase 4)

Objective: Verify consensus spans flow through the telemetry pipeline.

What to do:

• Consensus runs automatically in 6-node network
• Query Jaeger API for:
  • consensus.proposal.send (Adaptor::propose)
  • consensus.ledger_close (Adaptor::onClose)
  • consensus.accept (Adaptor::onAccept)
  • consensus.validation.send (Adaptor::validate)
• Verify attributes:
  • xrpl.consensus.mode on consensus.ledger_close
  • xrpl.consensus.proposers on consensus.accept
  • xrpl.consensus.ledger.seq on consensus.validation.send

Verification:

• Jaeger shows consensus.ledger_close traces with xrpl.consensus.mode
• Jaeger shows consensus.accept traces with xrpl.consensus.proposers
• Jaeger shows consensus.proposal.send traces
• Jaeger shows consensus.validation.send traces

---

Task IT.5: Spanmetrics Verification (Phase 5)

Objective: Verify spanmetrics connector derives RED metrics from spans.

What to do:

• Query Prometheus for traces_span_metrics_calls_total
• Query Prometheus for traces_span_metrics_duration_milliseconds_count
• Verify Grafana loads at http://localhost:3000

Verification:

• Prometheus returns non-empty results for traces_span_metrics_calls_total
• Prometheus returns non-empty results for duration histogram
• Grafana UI accessible with dashboards visible

---

Task IT.6: Manual Testing Plan

Objective: Document how to run tests manually for future reference.

What to do:

• Create docker/telemetry/TESTING.md with:
  • Prerequisites section
  • Single-node standalone test (quick verification)
  • 6-node consensus test (full verification)
  • Expected span catalog (all 12 span names with attributes)
  • Verification queries (Jaeger API, Prometheus API)
  • Troubleshooting guide

Key new file: docker/telemetry/TESTING.md

Verification:

• Document covers both single-node and multi-node testing
• All 12 span names documented with source file and attributes
• Troubleshooting section covers common failure modes

---

Task IT.7: Run and Verify

Objective: Execute the integration test and validate results.

What to do:

• Run docker/telemetry/integration-test.sh locally
• Debug any failures
• Leave stack running for manual verification
• Share URLs:
  • Jaeger: http://localhost:16686
  • Grafana: http://localhost:3000
  • Prometheus: http://localhost:9090

Verification:

• Script completes with all checks passing
• Jaeger UI shows rippled service with all expected span names
• Grafana dashboards load and show data

---

Task IT.8: Commit

Objective: Commit all new files to Phase 5 branch.

What to do:

• Run pcc (pre-commit checks)
• Commit 3 new files to pratik/otel-phase5-docs-deployment

Verification:

• pcc passes
• Commit created on Phase 5 branch

---

Summary

Task	Description	New Files	Depends On
IT.1	Integration test script	1	Phase 5
IT.2	RPC span verification	0	IT.1
IT.3	Transaction span verification	0	IT.1
IT.4	Consensus span verification	0	IT.1
IT.5	Spanmetrics verification	0	IT.1
IT.6	Manual testing plan	1	--
IT.7	Run and verify	0	IT.1-IT.6
IT.8	Commit	0	IT.7

Exit Criteria:

• All 6 xrpld nodes reach "proposing" state
• All 11 expected span names visible in Jaeger
• Spanmetrics available in Prometheus
• Grafana dashboards show data
• Manual testing plan document complete