Compare commits

..

36 Commits

Author SHA1 Message Date
cyan317
8b0e68f48e chore: Add counter for total messages waiting to be sent (#1691) 2024-10-18 16:11:38 +01:00
Alex Kremer
189098d092 fix: Static linkage (#1551)
Fixes #1507
2024-10-18 16:11:38 +01:00
Alex Kremer
854749a05e fix: Add upper bound to limit 2024-09-11 15:15:17 +01:00
cyan317
f57706be3d fix: no restriction on type field (#1644)
'type' should not matter if 'full' or 'accounts' is false. Relax the
restriction for 'type'
2024-09-11 14:44:20 +01:00
cyan317
bb0d912f2b fix: Add more restrictions to admin fields (#1643) 2024-09-10 15:05:23 +01:00
cyan317
d02d6affdb fix: Don't forward ledger API if 'full' is a string (#1640)
Fix #1635
2024-09-09 11:21:42 +01:00
cyan317
0054e4b64c fix: not forward admin API (#1629)
To merge to 2.2.3 branch.
It is different from the #1628 . I think this branch still forward
feature API to rippled.
2024-09-06 10:37:08 +01:00
Sergey Kuznetsov
9fe9e7c9d2 fix: Subscription source bugs fix (#1626)
For #1620.

- Add timeouts for websocket operations for connections to rippled.
Without these timeouts if connection hangs for some reason, clio
wouldn't know the connection is hanging.
- Fix potential data race in choosing new subscription source which will
forward messages to users.
- Optimise switching between subscription sources.
2024-09-05 14:58:06 +01:00
Alex Kremer
2e2740d4c5 feat: Published subscription message counters (#1618)
This PR adds counters to track the amount of published messages for each
subscription stream.
2024-08-29 16:48:04 +01:00
Sergey Kuznetsov
5004dc4e15 fix: Fix logging in SubscriptionSource (#1617)
For #1616. Later should be ported to develop as well.
2024-08-29 15:59:02 +01:00
cyan317
665fab183a fix: Add more account check (#1543)
Make sure all char is alphanumeric for account
2024-07-18 15:38:24 +01:00
Alex Kremer
b65ac67d17 fix: Relax error when full or accounts set to false (#1540)
Fixes #1537
2024-07-18 15:20:46 +01:00
Sergey Kuznetsov
7b18e28c47 fix: Fix extra brackets in warnings (#1519)
Fixes #1518
2024-07-05 12:05:14 +01:00
cyan317
4940d463dc Fix empty currency (#1481) 2024-06-21 13:01:14 +01:00
Peter Chen
c795cf371a Fix base_asset value in getAggregatePrice (#1467)
Fixes #1372
2024-06-18 09:04:33 -04:00
Peter Chen
e135aa49d5 Create generate free port class to avoid conflicting ports (#1439)
Fixes #1317
2024-06-18 11:29:05 +01:00
cyan317
5ba08b1d26 Improve etl check (#1465)
Fix #1444
2024-06-17 11:52:39 +01:00
github-actions[bot]
37cd79ceb0 [CI] clang-tidy auto fixes (#1464)
Fixes #1463. Please review and commit clang-tidy fixes.

Co-authored-by: kuznetsss <kuznetsss@users.noreply.github.com>
2024-06-17 09:18:35 +01:00
Sergey Kuznetsov
1334bd05d9 Add forwarding timeout option (#1462)
Fixes #1454.
2024-06-14 16:53:08 +01:00
Peter Chen
437ea7bf98 Fix quoteAsset value in getAggregatePrice (#1449)
Fixes #1373
2024-06-12 11:16:11 -04:00
github-actions[bot]
f9f3bc928e [CI] clang-tidy auto fixes (#1459)
Fixes #1458. Please review and commit clang-tidy fixes.

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Co-authored-by: Sergey Kuznetsov <skuznetsov@ripple.com>
2024-06-12 11:25:16 +01:00
cyan317
aa1f3efda2 Add trouble shooting md (#1455)
Fix #1284
2024-06-12 10:34:49 +01:00
github-actions[bot]
a6d21c1a02 [CI] clang-tidy auto fixes (#1457)
Fixes #1456. Please review and commit clang-tidy fixes.

Co-authored-by: kuznetsss <kuznetsss@users.noreply.github.com>
2024-06-12 10:34:13 +01:00
cyan317
49b80c7ad8 Support string type integer for oracle_document_id (#1448)
Fix #1420
2024-06-12 10:31:32 +01:00
Sergey Kuznetsov
56ab943be5 Add option to set X-User header value for forwarded requests (#1425)
Fixes #1422.
2024-06-11 17:59:10 +01:00
Sergey Kuznetsov
9d3b4f0313 Add assertion to process method (#1453)
Also adjust clangd so it doesn't bother us with header files about `expected`.
2024-06-11 13:10:08 +01:00
Alex Kremer
42c970a2a3 Forward feature RPC (#1440)
Fixes #1436 

This is a temporary implementation of the `feature` RPC that will always
return `noPermission` iff `vetoed` is set.
If `vetoed` isn't specified, Clio will always forward the request to
`rippled` instead.

In the future, #1131 will implement a Clio-native `feature` RPC. This
requires specific support from `libxrpl` side and that is not going to
be available till at least 2.2.1, hence the temporary forwarding.

It would be great to review the error message and code so that we pick
the right one from the start.

Co-authored-by: Sergey Kuznetsov <skuznetsov@ripple.com>
2024-06-11 10:51:03 +01:00
Peter Chen
1125b09611 Allow tlsv13 in Clio (#1447)
Fixes #1419

Co-authored-by: Sergey Kuznetsov <skuznetsov@ripple.com>
2024-06-10 11:52:53 -04:00
Sergey Kuznetsov
ce94f0f513 Update libxrpl to 2.2.0 (#1446) 2024-06-10 14:52:35 +01:00
Peter Chen
d39fb20022 Update build documentation to address encountered errors (#1442) 2024-06-04 10:05:27 -04:00
Peter Chen
967b85ca33 Change ledgerInfo and replace with ledgerHeader (#1426)
Fixes [#1396](https://github.com/XRPLF/clio/issues/1396)
2024-06-03 13:11:31 -04:00
Peter Chen
55b8134e6d Comment out precommit hook for Doc (#1432)
Temporary Fix for #1431
2024-05-30 20:17:18 +01:00
Alex Kremer
66e8a65732 Flow to check new libXRPL version (#1433) 2024-05-29 17:31:15 +01:00
Peter Chen
067dd72aed Move NameGenerator to util (#1428)
Fixes for #876
2024-05-29 09:08:27 -04:00
Peter Chen
da5bf5c441 Fix invalid syntax in example-config.json (#1423)
remove "." fixes syntax for example-config; runs with ./clio_server
2024-05-24 09:22:08 -04:00
Sergey Kuznetsov
ff4bc5b0aa Load tool (#1421)
Simple tool to put a specific load on clio.
2024-05-24 13:01:36 +01:00
164 changed files with 3467 additions and 1432 deletions

View File

@@ -5,4 +5,6 @@ Diagnostics:
UnusedIncludes: Strict
MissingIncludes: Strict
Includes:
IgnoreHeader: ".*/(detail|impl)/.*"
IgnoreHeader:
- ".*/(detail|impl)/.*"
- ".*expected.*"

View File

@@ -3,4 +3,6 @@
# This script is intended to be run from the root of the repository.
source .githooks/check-format
source .githooks/check-docs
#source .githooks/check-docs
# TODO: Fix Doxygen issue with reference links. See https://github.com/XRPLF/clio/issues/1431

View File

@@ -7,6 +7,14 @@ inputs:
body:
description: Issue body
required: true
labels:
description: Comma-separated list of labels
required: true
default: 'bug'
assignees:
description: Comma-separated list of assignees
required: true
default: 'cindyyan317,godexsoft,kuznetsss'
outputs:
created_issue_id:
description: Created issue id
@@ -19,7 +27,7 @@ runs:
shell: bash
run: |
echo -e '${{ inputs.body }}' > issue.md
gh issue create --assignee 'cindyyan317,godexsoft,kuznetsss' --label bug --title '${{ inputs.title }}' --body-file ./issue.md > create_issue.log
gh issue create --assignee '${{ inputs.assignees }}' --label '${{ inputs.labels }}' --title '${{ inputs.title }}' --body-file ./issue.md > create_issue.log
created_issue=$(cat create_issue.log | sed 's|.*/||')
echo "created_issue=$created_issue" >> $GITHUB_OUTPUT
rm create_issue.log issue.md

28
.github/scripts/update-libxrpl-version vendored Executable file
View File

@@ -0,0 +1,28 @@
#!/bin/bash
# Note: This script is intended to be run from the root of the repository.
#
# This script modifies conanfile.py such that the specified version of libXRPL is used.
if [[ -z "$1" ]]; then
cat <<EOF
ERROR
-----------------------------------------------------------------------------
Version should be passed as first argument to the script.
-----------------------------------------------------------------------------
EOF
exit 1
fi
VERSION=$1
GNU_SED=$(sed --version 2>&1 | grep -q 'GNU' && echo true || echo false)
echo "+ Updating required libXRPL version to $VERSION"
if [[ "$GNU_SED" == "false" ]]; then
sed -i '' -E "s|'xrpl/[a-zA-Z0-9\\.\\-]+'|'xrpl/$VERSION'|g" conanfile.py
else
sed -i -E "s|'xrpl/[a-zA-Z0-9\\.\\-]+'|'xrpl/$VERSION'|g" conanfile.py
fi

91
.github/workflows/check_libxrpl.yml vendored Normal file
View File

@@ -0,0 +1,91 @@
name: Check new libXRPL
on:
repository_dispatch:
types: [check_libxrpl]
jobs:
build:
name: Build Clio / `libXRPL ${{ github.event.client_payload.version }}`
runs-on: [self-hosted, heavy]
container:
image: rippleci/clio_ci:latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Update libXRPL version requirement
shell: bash
run: |
./.github/scripts/update-libxrpl-version ${{ github.event.client_payload.version }}
- name: Prepare runner
uses: ./.github/actions/prepare_runner
with:
disable_ccache: true
- name: Setup conan
uses: ./.github/actions/setup_conan
id: conan
with:
conan_profile: gcc
- name: Run conan and cmake
uses: ./.github/actions/generate
with:
conan_profile: ${{ steps.conan.outputs.conan_profile }}
conan_cache_hit: ${{ steps.restore_cache.outputs.conan_cache_hit }}
build_type: Release
- name: Build Clio
uses: ./.github/actions/build_clio
- name: Strip tests
run: strip build/clio_tests
- name: Upload clio_tests
uses: actions/upload-artifact@v4
with:
name: clio_tests_libxrpl-${{ github.event.client_payload.version }}
path: build/clio_tests
run_tests:
name: Run tests
needs: build
runs-on: [self-hosted, heavy]
container:
image: rippleci/clio_ci:latest
steps:
- uses: actions/download-artifact@v4
with:
name: clio_tests_libxrpl-${{ github.event.client_payload.version }}
- name: Run clio_tests
run: |
chmod +x ./clio_tests
./clio_tests
create_issue_on_failure:
name: Create an issue on failure
needs: [build, run_tests]
if: ${{ always() && contains(needs.*.result, 'failure') }}
runs-on: ubuntu-20.04
permissions:
contents: write
issues: write
steps:
- uses: actions/checkout@v4
- name: Create an issue
uses: ./.github/actions/create_issue
env:
GH_TOKEN: ${{ github.token }}
with:
labels: 'compatibility,bug'
title: 'Proposed libXRPL check failed'
body: >
Clio build or tests failed against `libXRPL ${{ github.event.client_payload.version }}`.
Workflow: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}/

View File

@@ -13,12 +13,15 @@ jobs:
include:
- os: macos14
build_type: Release
static: false
- os: heavy
build_type: Release
static: true
container:
image: rippleci/clio_ci:latest
- os: heavy
build_type: Debug
static: true
container:
image: rippleci/clio_ci:latest
runs-on: [self-hosted, "${{ matrix.os }}"]
@@ -50,6 +53,7 @@ jobs:
conan_profile: ${{ steps.conan.outputs.conan_profile }}
conan_cache_hit: ${{ steps.restore_cache.outputs.conan_cache_hit }}
build_type: ${{ matrix.build_type }}
static: ${{ matrix.static }}
- name: Build Clio
uses: ./.github/actions/build_clio

View File

@@ -36,6 +36,7 @@ Below are some useful docs to learn more about Clio.
- [How to configure Clio and rippled](./docs/configure-clio.md)
- [How to run Clio](./docs/run-clio.md)
- [Logging](./docs/logging.md)
- [Troubleshooting guide](./docs/trouble_shooting.md)
**General reference material:**

View File

@@ -28,7 +28,7 @@ class Clio(ConanFile):
'protobuf/3.21.9',
'grpc/1.50.1',
'openssl/1.1.1u',
'xrpl/2.2.0-rc3',
'xrpl/2.2.0',
'libbacktrace/cci.20210118'
]

View File

@@ -36,6 +36,8 @@ compiler.version=15
compiler.libcxx=libc++
build_type=Release
compiler.cppstd=20
[conf]
tools.build:cxxflags+=["-DBOOST_ASIO_DISABLE_CONCEPTS"]
```
> Linux example:
@@ -91,6 +93,9 @@ If successful, `conan install` will find the required packages and `cmake` will
> [!TIP]
> To generate a Code Coverage report, include `-o coverage=True` in the `conan install` command above, along with `-o tests=True` to enable tests. After running the `cmake` commands, execute `make clio_tests-ccov`. The coverage report will be found at `clio_tests-llvm-cov/index.html`.
> [!NOTE]
> If you've built Clio before and the build is now failing, it's likely due to updated dependencies. Try deleting the build folder and then rerunning the Conan and CMake commands mentioned above.
### Generating API docs for Clio
The API documentation for Clio is generated by [Doxygen](https://www.doxygen.nl/index.html). If you want to generate the API documentation when building Clio, make sure to install Doxygen on your system.

View File

@@ -27,7 +27,7 @@ If you're running Clio and `rippled` on separate machines, in addition to uncomm
2. Open a public, unencrypted WebSocket port on your `rippled` server.
3. In the `rippled` config, change the IP specified for `secure_gateway`, under the `port_grpc` section, to the IP of your Clio server. This entry can take the form of a comma-separated list if you are running multiple Clio nodes.
3. In the `rippled` config, change the IP specified for `secure_gateway`, under the `port_grpc` and websocket server sections, to the IP of your Clio server. This entry can take the form of a comma-separated list if you are running multiple Clio nodes.
## Ledger sequence

View File

@@ -35,7 +35,10 @@
"grpc_port": "50051"
}
],
"forwarding_cache_timeout": 0.250, // in seconds, could be 0, which means no cache
"forwarding": {
"cache_timeout": 0.250, // in seconds, could be 0, which means no cache
"request_timeout": 10.0 // time for Clio to wait for rippled to reply on a forwarded request (default is 10 seconds)
},
"dos_guard": {
// Comma-separated list of IPs to exclude from rate limiting
"whitelist": [
@@ -64,10 +67,10 @@
"admin_password": "xrp",
// If local_admin is true, Clio will consider requests come from 127.0.0.1 as admin requests
// It's true by default unless admin_password is set,'local_admin' : true and 'admin_password' can not be set at the same time
"local_amdin": false
"local_admin": false
},
// Time in seconds for graceful shutdown. Defaults to 10 seconds. Not fully implemented yet.
"graceful_period": 10.,
"graceful_period": 10.0,
// Overrides log level on a per logging channel.
// Defaults to global "log_level" for each unspecified channel.
"log_channels": [

47
docs/trouble_shooting.md Normal file
View File

@@ -0,0 +1,47 @@
# Troubleshooting Guide
This guide will help you troubleshoot common issues of Clio.
## Can't connect to DB
If you see the error log message `Could not connect to Cassandra: No hosts available`, this means that Clio can't connect to the database. Check the following:
- Make sure the database is running at the specified address and port.
- Make sure the database is accessible from the machine where Clio is running.
You can use [cqlsh](https://pypi.org/project/cqlsh/) to check the connection to the database.
If you would like to run a local ScyllaDB, you can call:
```sh
docker run --rm -p 9042:9042 --name clio-scylla -d scylladb/scylla
```
## Check the server status of Clio
To check if Clio is syncing with rippled:
```sh
curl -v -d '{"method":"server_info", "params":[{}]}' 127.0.0.1:51233|python3 -m json.tool|grep seq
```
If Clio is syncing with rippled, the `seq` value will be increasing.
## Clio fails to start
If you see the error log message `Failed to fetch ETL state from...`, this means the configured rippled node is not reachable. Check the following:
- Make sure the rippled node is running at the specified address and port.
- Make sure the rippled node is accessible from the machine where Clio is running.
If you would like to run Clio without an avaliable rippled node, you can add below setting to Clio's configuration file:
```
"allow_no_etl": true
```
## Clio is not added to secure_gateway in rippled's config
If you see the warning message `AsyncCallData is_unlimited is false.`, this means that Clio is not added to the `secure_gateway` of `port_grpc` session in the rippled configuration file. It will slow down the sync process. Please add Clio's IP to the `secure_gateway` in the rippled configuration file for both grpc and ws port.
## Clio is slow
To speed up the response time, Clio has a cache inside. However, cache can take time to warm up. If you see slow response time, you can firstly check if cache is still loading.
You can check the cache status by calling:
```sh
curl -v -d '{"method":"server_info", "params":[{}]}' 127.0.0.1:51233|python3 -m json.tool|grep is_full
curl -v -d '{"method":"server_info", "params":[{}]}' 127.0.0.1:51233|python3 -m json.tool|grep is_enabled
```
If `is_full` is false, it means the cache is still loading. Normally, the Clio can respond quicker if cache finishs loading. If `is_enabled` is false, it means the cache is disabled in the configuration file or there is data corruption in the database.
## Receive error message `Too many requests`
If client sees the error message `Too many requests`, this means that the client is blocked by Clio's DosGuard protection. You may want to add the client's IP to the whitelist in the configuration file, Or update other your DosGuard settings.

View File

@@ -206,13 +206,13 @@ public:
}
void
writeLedger(ripple::LedgerHeader const& ledgerInfo, std::string&& blob) override
writeLedger(ripple::LedgerHeader const& ledgerHeader, std::string&& blob) override
{
executor_.write(schema_->insertLedgerHeader, ledgerInfo.seq, std::move(blob));
executor_.write(schema_->insertLedgerHeader, ledgerHeader.seq, std::move(blob));
executor_.write(schema_->insertLedgerHash, ledgerInfo.hash, ledgerInfo.seq);
executor_.write(schema_->insertLedgerHash, ledgerHeader.hash, ledgerHeader.seq);
ledgerSequence_ = ledgerInfo.seq;
ledgerSequence_ = ledgerHeader.seq;
}
std::optional<std::uint32_t>

View File

@@ -47,7 +47,7 @@ struct ETLState {
fetchETLStateFromSource(Forward& source) noexcept
{
auto const serverInfoRippled = data::synchronous([&source](auto yield) {
return source.forwardToRippled({{"command", "server_info"}}, std::nullopt, yield);
return source.forwardToRippled({{"command", "server_info"}}, std::nullopt, {}, yield);
});
if (serverInfoRippled)

View File

@@ -25,7 +25,6 @@
#include "etl/Source.hpp"
#include "feed/SubscriptionManagerInterface.hpp"
#include "util/Assert.hpp"
#include "util/Constants.hpp"
#include "util/Random.hpp"
#include "util/log/Logger.hpp"
@@ -38,7 +37,6 @@
#include <algorithm>
#include <chrono>
#include <cmath>
#include <cstddef>
#include <cstdint>
#include <memory>
@@ -77,11 +75,9 @@ LoadBalancer::LoadBalancer(
SourceFactory sourceFactory
)
{
auto const forwardingCacheTimeout = config.valueOr<float>("forwarding_cache_timeout", 0.f);
auto const forwardingCacheTimeout = config.valueOr<float>("forwarding.cache_timeout", 0.f);
if (forwardingCacheTimeout > 0.f) {
forwardingCache_ = impl::ForwardingCache{std::chrono::milliseconds{
std::lroundf(forwardingCacheTimeout * static_cast<float>(util::MILLISECONDS_PER_SECOND))
}};
forwardingCache_ = impl::ForwardingCache{Config::toMilliseconds(forwardingCacheTimeout)};
}
static constexpr std::uint32_t MAX_DOWNLOAD = 256;
@@ -103,6 +99,7 @@ LoadBalancer::LoadBalancer(
}
};
auto const forwardingTimeout = Config::toMilliseconds(config.valueOr<float>("forwarding.request_timeout", 10.));
for (auto const& entry : config.array("etl_sources")) {
auto source = sourceFactory(
entry,
@@ -110,22 +107,27 @@ LoadBalancer::LoadBalancer(
backend,
subscriptions,
validatedLedgers,
forwardingTimeout,
[this]() {
if (not hasForwardingSource_)
if (not hasForwardingSource_.lock().get())
chooseForwardingSource();
},
[this]() { chooseForwardingSource(); },
[this]() { forwardingCache_->invalidate(); }
[this](bool wasForwarding) {
if (wasForwarding)
chooseForwardingSource();
},
[this]() {
if (forwardingCache_.has_value())
forwardingCache_->invalidate();
}
);
// checking etl node validity
auto const stateOpt = ETLState::fetchETLStateFromSource(*source);
if (!stateOpt) {
checkOnETLFailure(fmt::format(
"Failed to fetch ETL state from source = {} Please check the configuration and network",
source->toString()
));
LOG(log_.warn()) << "Failed to fetch ETL state from source = " << source->toString()
<< " Please check the configuration and network";
} else if (etlState_ && etlState_->networkID && stateOpt->networkID &&
etlState_->networkID != stateOpt->networkID) {
checkOnETLFailure(fmt::format(
@@ -141,6 +143,9 @@ LoadBalancer::LoadBalancer(
LOG(log_.info()) << "Added etl source - " << sources_.back()->toString();
}
if (!etlState_)
checkOnETLFailure("Failed to fetch ETL state from any source. Please check the configuration and network");
if (sources_.empty())
checkOnETLFailure("No ETL sources configured. Please check the configuration");
@@ -213,6 +218,7 @@ std::optional<boost::json::object>
LoadBalancer::forwardToRippled(
boost::json::object const& request,
std::optional<std::string> const& clientIp,
bool isAdmin,
boost::asio::yield_context yield
)
{
@@ -227,9 +233,11 @@ LoadBalancer::forwardToRippled(
auto numAttempts = 0u;
auto xUserValue = isAdmin ? ADMIN_FORWARDING_X_USER_VALUE : USER_FORWARDING_X_USER_VALUE;
std::optional<boost::json::object> response;
while (numAttempts < sources_.size()) {
if (auto res = sources_[sourceIdx]->forwardToRippled(request, clientIp, yield)) {
if (auto res = sources_[sourceIdx]->forwardToRippled(request, clientIp, xUserValue, yield)) {
response = std::move(res);
break;
}
@@ -309,11 +317,13 @@ LoadBalancer::getETLState() noexcept
void
LoadBalancer::chooseForwardingSource()
{
hasForwardingSource_ = false;
LOG(log_.info()) << "Choosing a new source to forward subscriptions";
auto hasForwardingSourceLock = hasForwardingSource_.lock();
hasForwardingSourceLock.get() = false;
for (auto& source : sources_) {
if (not hasForwardingSource_ and source->isConnected()) {
if (not hasForwardingSourceLock.get() and source->isConnected()) {
source->setForwarding(true);
hasForwardingSource_ = true;
hasForwardingSourceLock.get() = true;
} else {
source->setForwarding(false);
}

View File

@@ -20,11 +20,12 @@
#pragma once
#include "data/BackendInterface.hpp"
#include "etl/ETLHelpers.hpp"
#include "etl/ETLState.hpp"
#include "etl/NetworkValidatedLedgersInterface.hpp"
#include "etl/Source.hpp"
#include "etl/impl/ForwardingCache.hpp"
#include "feed/SubscriptionManagerInterface.hpp"
#include "util/Mutex.hpp"
#include "util/config/Config.hpp"
#include "util/log/Logger.hpp"
@@ -38,13 +39,12 @@
#include <org/xrpl/rpc/v1/ledger.pb.h>
#include <ripple/proto/org/xrpl/rpc/v1/xrp_ledger.grpc.pb.h>
#include <atomic>
#include <chrono>
#include <cstdint>
#include <memory>
#include <optional>
#include <string>
#include <utility>
#include <string_view>
#include <vector>
namespace etl {
@@ -68,13 +68,28 @@ private:
util::Logger log_{"ETL"};
// Forwarding cache must be destroyed after sources because sources have a callback to invalidate cache
std::optional<impl::ForwardingCache> forwardingCache_;
std::optional<std::string> forwardingXUserValue_;
std::vector<SourcePtr> sources_;
std::optional<ETLState> etlState_;
std::uint32_t downloadRanges_ =
DEFAULT_DOWNLOAD_RANGES; /*< The number of markers to use when downloading initial ledger */
std::atomic_bool hasForwardingSource_{false};
// Using mutext instead of atomic_bool because choosing a new source to
// forward messages should be done with a mutual exclusion otherwise there will be a race condition
util::Mutex<bool> hasForwardingSource_{false};
public:
/**
* @brief Value for the X-User header when forwarding admin requests
*/
static constexpr std::string_view ADMIN_FORWARDING_X_USER_VALUE = "clio_admin";
/**
* @brief Value for the X-User header when forwarding user requests
*/
static constexpr std::string_view USER_FORWARDING_X_USER_VALUE = "clio_user";
/**
* @brief Create an instance of the load balancer.
*
@@ -167,6 +182,7 @@ public:
*
* @param request JSON-RPC request to forward
* @param clientIp The IP address of the peer, if known
* @param isAdmin Whether the request is from an admin
* @param yield The coroutine context
* @return Response received from rippled node as JSON object on success; nullopt on failure
*/
@@ -174,6 +190,7 @@ public:
forwardToRippled(
boost::json::object const& request,
std::optional<std::string> const& clientIp,
bool isAdmin,
boost::asio::yield_context yield
);

View File

@@ -30,6 +30,7 @@
#include <boost/asio/io_context.hpp>
#include <chrono>
#include <memory>
#include <string>
#include <utility>
@@ -43,6 +44,7 @@ make_Source(
std::shared_ptr<BackendInterface> backend,
std::shared_ptr<feed::SubscriptionManagerInterface> subscriptions,
std::shared_ptr<NetworkValidatedLedgersInterface> validatedLedgers,
std::chrono::steady_clock::duration forwardingTimeout,
SourceBase::OnConnectHook onConnect,
SourceBase::OnDisconnectHook onDisconnect,
SourceBase::OnLedgerClosedHook onLedgerClosed
@@ -52,7 +54,7 @@ make_Source(
auto const wsPort = config.valueOr<std::string>("ws_port", {});
auto const grpcPort = config.valueOr<std::string>("grpc_port", {});
impl::ForwardingSource forwardingSource{ip, wsPort};
impl::ForwardingSource forwardingSource{ip, wsPort, forwardingTimeout};
impl::GrpcSource grpcSource{ip, grpcPort, std::move(backend)};
auto subscriptionSource = std::make_unique<impl::SubscriptionSource>(
ioc,

View File

@@ -32,11 +32,13 @@
#include <grpcpp/support/status.h>
#include <org/xrpl/rpc/v1/get_ledger.pb.h>
#include <chrono>
#include <cstdint>
#include <functional>
#include <memory>
#include <optional>
#include <string>
#include <string_view>
#include <utility>
#include <vector>
@@ -49,7 +51,7 @@ namespace etl {
class SourceBase {
public:
using OnConnectHook = std::function<void()>;
using OnDisconnectHook = std::function<void()>;
using OnDisconnectHook = std::function<void(bool)>;
using OnLedgerClosedHook = std::function<void()>;
virtual ~SourceBase() = default;
@@ -127,6 +129,7 @@ public:
*
* @param request The request to forward
* @param forwardToRippledClientIp IP of the client forwarding this request if known
* @param xUserValue Value of the X-User header
* @param yield The coroutine context
* @return Response wrapped in an optional on success; nullopt otherwise
*/
@@ -134,6 +137,7 @@ public:
forwardToRippled(
boost::json::object const& request,
std::optional<std::string> const& forwardToRippledClientIp,
std::string_view xUserValue,
boost::asio::yield_context yield
) const = 0;
};
@@ -146,6 +150,7 @@ using SourceFactory = std::function<SourcePtr(
std::shared_ptr<BackendInterface> backend,
std::shared_ptr<feed::SubscriptionManagerInterface> subscriptions,
std::shared_ptr<NetworkValidatedLedgersInterface> validatedLedgers,
std::chrono::steady_clock::duration forwardingTimeout,
SourceBase::OnConnectHook onConnect,
SourceBase::OnDisconnectHook onDisconnect,
SourceBase::OnLedgerClosedHook onLedgerClosed
@@ -159,6 +164,7 @@ using SourceFactory = std::function<SourcePtr(
* @param backend BackendInterface implementation
* @param subscriptions Subscription manager
* @param validatedLedgers The network validated ledgers data structure
* @param forwardingTimeout The timeout for forwarding to rippled
* @param onConnect The hook to call on connect
* @param onDisconnect The hook to call on disconnect
* @param onLedgerClosed The hook to call on ledger closed. This is called when a ledger is closed and the source is set
@@ -172,6 +178,7 @@ make_Source(
std::shared_ptr<BackendInterface> backend,
std::shared_ptr<feed::SubscriptionManagerInterface> subscriptions,
std::shared_ptr<NetworkValidatedLedgersInterface> validatedLedgers,
std::chrono::steady_clock::duration forwardingTimeout,
SourceBase::OnConnectHook onConnect,
SourceBase::OnDisconnectHook onDisconnect,
SourceBase::OnLedgerClosedHook onLedgerClosed

View File

@@ -34,6 +34,7 @@
#include <optional>
#include <stdexcept>
#include <string>
#include <string_view>
#include <utility>
namespace etl::impl {
@@ -41,9 +42,12 @@ namespace etl::impl {
ForwardingSource::ForwardingSource(
std::string ip,
std::string wsPort,
std::chrono::steady_clock::duration forwardingTimeout,
std::chrono::steady_clock::duration connectionTimeout
)
: log_(fmt::format("ForwardingSource[{}:{}]", ip, wsPort)), connectionBuilder_(std::move(ip), std::move(wsPort))
: log_(fmt::format("ForwardingSource[{}:{}]", ip, wsPort))
, connectionBuilder_(std::move(ip), std::move(wsPort))
, forwardingTimeout_{forwardingTimeout}
{
connectionBuilder_.setConnectionTimeout(connectionTimeout)
.addHeader(
@@ -55,6 +59,7 @@ std::optional<boost::json::object>
ForwardingSource::forwardToRippled(
boost::json::object const& request,
std::optional<std::string> const& forwardToRippledClientIp,
std::string_view xUserValue,
boost::asio::yield_context yield
) const
{
@@ -64,18 +69,21 @@ ForwardingSource::forwardToRippled(
{boost::beast::http::field::forwarded, fmt::format("for={}", *forwardToRippledClientIp)}
);
}
connectionBuilder.addHeader({"X-User", std::string{xUserValue}});
auto expectedConnection = connectionBuilder.connect(yield);
if (not expectedConnection) {
return std::nullopt;
}
auto& connection = expectedConnection.value();
auto writeError = connection->write(boost::json::serialize(request), yield);
auto writeError = connection->write(boost::json::serialize(request), yield, forwardingTimeout_);
if (writeError) {
return std::nullopt;
}
auto response = connection->read(yield);
auto response = connection->read(yield, forwardingTimeout_);
if (not response) {
return std::nullopt;
}

View File

@@ -28,19 +28,22 @@
#include <chrono>
#include <optional>
#include <string>
#include <string_view>
namespace etl::impl {
class ForwardingSource {
util::Logger log_;
util::requests::WsConnectionBuilder connectionBuilder_;
std::chrono::steady_clock::duration forwardingTimeout_;
static constexpr std::chrono::seconds CONNECTION_TIMEOUT{3};
public:
ForwardingSource(
std::string ip_,
std::string wsPort_,
std::string ip,
std::string wsPort,
std::chrono::steady_clock::duration forwardingTimeout,
std::chrono::steady_clock::duration connectionTimeout = CONNECTION_TIMEOUT
);
@@ -49,6 +52,7 @@ public:
*
* @param request The request to forward
* @param forwardToRippledClientIp IP of the client forwarding this request if known
* @param xUserValue Optional value for X-User header
* @param yield The coroutine context
* @return Response wrapped in an optional on success; nullopt otherwise
*/
@@ -56,6 +60,7 @@ public:
forwardToRippled(
boost::json::object const& request,
std::optional<std::string> const& forwardToRippledClientIp,
std::string_view xUserValue,
boost::asio::yield_context yield
) const;
};

View File

@@ -46,7 +46,7 @@
namespace etl::impl {
GrpcSource::GrpcSource(std::string const& ip, std::string const& grpcPort, std::shared_ptr<BackendInterface> backend)
: log_(fmt::format("ETL_Grpc[{}:{}]", ip, grpcPort)), backend_(std::move(backend))
: log_(fmt::format("GrpcSource[{}:{}]", ip, grpcPort)), backend_(std::move(backend))
{
try {
boost::asio::ip::tcp::endpoint const endpoint{boost::asio::ip::make_address(ip), std::stoi(grpcPort)};

View File

@@ -34,6 +34,7 @@
#include <memory>
#include <optional>
#include <string>
#include <string_view>
#include <utility>
#include <vector>
namespace etl::impl {
@@ -202,6 +203,7 @@ public:
*
* @param request The request to forward
* @param forwardToRippledClientIp IP of the client forwarding this request if known
* @param xUserValue Optional value of the X-User header
* @param yield The coroutine context
* @return Response wrapped in an optional on success; nullopt otherwise
*/
@@ -209,10 +211,11 @@ public:
forwardToRippled(
boost::json::object const& request,
std::optional<std::string> const& forwardToRippledClientIp,
std::string_view xUserValue,
boost::asio::yield_context yield
) const final
{
return forwardingSource_.forwardToRippled(request, forwardToRippledClientIp, yield);
return forwardingSource_.forwardToRippled(request, forwardToRippledClientIp, xUserValue, yield);
}
};

View File

@@ -24,6 +24,8 @@
#include "rpc/JS.hpp"
#include "util/Retry.hpp"
#include "util/log/Logger.hpp"
#include "util/prometheus/Label.hpp"
#include "util/prometheus/Prometheus.hpp"
#include "util/requests/Types.hpp"
#include <boost/algorithm/string/classification.hpp>
@@ -66,22 +68,28 @@ SubscriptionSource::SubscriptionSource(
OnConnectHook onConnect,
OnDisconnectHook onDisconnect,
OnLedgerClosedHook onLedgerClosed,
std::chrono::steady_clock::duration const connectionTimeout,
std::chrono::steady_clock::duration const wsTimeout,
std::chrono::steady_clock::duration const retryDelay
)
: log_(fmt::format("GrpcSource[{}:{}]", ip, wsPort))
: log_(fmt::format("SubscriptionSource[{}:{}]", ip, wsPort))
, wsConnectionBuilder_(ip, wsPort)
, validatedLedgers_(std::move(validatedLedgers))
, subscriptions_(std::move(subscriptions))
, strand_(boost::asio::make_strand(ioContext))
, wsTimeout_(wsTimeout)
, retry_(util::makeRetryExponentialBackoff(retryDelay, RETRY_MAX_DELAY, strand_))
, onConnect_(std::move(onConnect))
, onDisconnect_(std::move(onDisconnect))
, onLedgerClosed_(std::move(onLedgerClosed))
, lastMessageTimeSecondsSinceEpoch_(PrometheusService::gaugeInt(
"subscription_source_last_message_time",
util::prometheus::Labels({{"source", fmt::format("{}:{}", ip, wsPort)}}),
"Seconds since epoch of the last message received from rippled subscription streams"
))
{
wsConnectionBuilder_.addHeader({boost::beast::http::field::user_agent, "clio-client"})
.addHeader({"X-User", "clio-client"})
.setConnectionTimeout(connectionTimeout);
.setConnectionTimeout(wsTimeout_);
}
SubscriptionSource::~SubscriptionSource()
@@ -133,6 +141,7 @@ void
SubscriptionSource::setForwarding(bool isForwarding)
{
isForwarding_ = isForwarding;
LOG(log_.info()) << "Forwarding set to " << isForwarding_;
}
std::chrono::steady_clock::time_point
@@ -166,20 +175,22 @@ SubscriptionSource::subscribe()
}
wsConnection_ = std::move(connection).value();
isConnected_ = true;
onConnect_();
auto const& subscribeCommand = getSubscribeCommandJson();
auto const writeErrorOpt = wsConnection_->write(subscribeCommand, yield);
auto const writeErrorOpt = wsConnection_->write(subscribeCommand, yield, wsTimeout_);
if (writeErrorOpt) {
handleError(writeErrorOpt.value(), yield);
return;
}
isConnected_ = true;
LOG(log_.info()) << "Connected";
onConnect_();
retry_.reset();
while (!stop_) {
auto const message = wsConnection_->read(yield);
auto const message = wsConnection_->read(yield, wsTimeout_);
if (not message) {
handleError(message.error(), yield);
return;
@@ -224,10 +235,11 @@ SubscriptionSource::handleMessage(std::string const& message)
auto validatedLedgers = boost::json::value_to<std::string>(result.at(JS(validated_ledgers)));
setValidatedRange(std::move(validatedLedgers));
}
LOG(log_.info()) << "Received a message on ledger subscription stream. Message : " << object;
LOG(log_.debug()) << "Received a message on ledger subscription stream. Message: " << object;
} else if (object.contains(JS(type)) && object.at(JS(type)) == JS_LedgerClosed) {
LOG(log_.info()) << "Received a message on ledger subscription stream. Message : " << object;
LOG(log_.debug()) << "Received a message of type 'ledgerClosed' on ledger subscription stream. Message: "
<< object;
if (object.contains(JS(ledger_index))) {
ledgerIndex = object.at(JS(ledger_index)).as_int64();
}
@@ -245,10 +257,13 @@ SubscriptionSource::handleMessage(std::string const& message)
// 2 - Validated transaction
// Only forward proposed transaction, validated transactions are sent by Clio itself
if (object.contains(JS(transaction)) and !object.contains(JS(meta))) {
LOG(log_.debug()) << "Forwarding proposed transaction: " << object;
subscriptions_->forwardProposedTransaction(object);
} else if (object.contains(JS(type)) && object.at(JS(type)) == JS_ValidationReceived) {
LOG(log_.debug()) << "Forwarding validation: " << object;
subscriptions_->forwardValidation(object);
} else if (object.contains(JS(type)) && object.at(JS(type)) == JS_ManifestReceived) {
LOG(log_.debug()) << "Forwarding manifest: " << object;
subscriptions_->forwardManifest(object);
}
}
@@ -261,7 +276,7 @@ SubscriptionSource::handleMessage(std::string const& message)
return std::nullopt;
} catch (std::exception const& e) {
LOG(log_.error()) << "Exception in handleMessage : " << e.what();
LOG(log_.error()) << "Exception in handleMessage: " << e.what();
return util::requests::RequestError{fmt::format("Error handling message: {}", e.what())};
}
}
@@ -270,16 +285,14 @@ void
SubscriptionSource::handleError(util::requests::RequestError const& error, boost::asio::yield_context yield)
{
isConnected_ = false;
isForwarding_ = false;
bool const wasForwarding = isForwarding_.exchange(false);
if (not stop_) {
onDisconnect_();
LOG(log_.info()) << "Disconnected";
onDisconnect_(wasForwarding);
}
if (wsConnection_ != nullptr) {
auto const err = wsConnection_->close(yield);
if (err) {
LOG(log_.error()) << "Error closing websocket connection: " << err->message();
}
wsConnection_->close(yield);
wsConnection_.reset();
}
@@ -306,7 +319,11 @@ SubscriptionSource::logError(util::requests::RequestError const& error) const
void
SubscriptionSource::setLastMessageTime()
{
lastMessageTime_.lock().get() = std::chrono::steady_clock::now();
lastMessageTimeSecondsSinceEpoch_.get().set(
std::chrono::duration_cast<std::chrono::seconds>(std::chrono::system_clock::now().time_since_epoch()).count()
);
auto lock = lastMessageTime_.lock();
lock.get() = std::chrono::steady_clock::now();
}
void

View File

@@ -19,12 +19,13 @@
#pragma once
#include "etl/ETLHelpers.hpp"
#include "etl/NetworkValidatedLedgersInterface.hpp"
#include "etl/Source.hpp"
#include "feed/SubscriptionManagerInterface.hpp"
#include "util/Mutex.hpp"
#include "util/Retry.hpp"
#include "util/log/Logger.hpp"
#include "util/prometheus/Gauge.hpp"
#include "util/requests/Types.hpp"
#include "util/requests/WsConnection.hpp"
@@ -37,6 +38,7 @@
#include <atomic>
#include <chrono>
#include <cstdint>
#include <functional>
#include <future>
#include <memory>
#include <optional>
@@ -71,6 +73,8 @@ private:
boost::asio::strand<boost::asio::io_context::executor_type> strand_;
std::chrono::steady_clock::duration wsTimeout_;
util::Retry retry_;
OnConnectHook onConnect_;
@@ -83,9 +87,11 @@ private:
util::Mutex<std::chrono::steady_clock::time_point> lastMessageTime_;
std::reference_wrapper<util::prometheus::GaugeInt> lastMessageTimeSecondsSinceEpoch_;
std::future<void> runFuture_;
static constexpr std::chrono::seconds CONNECTION_TIMEOUT{30};
static constexpr std::chrono::seconds WS_TIMEOUT{30};
static constexpr std::chrono::seconds RETRY_MAX_DELAY{30};
static constexpr std::chrono::seconds RETRY_DELAY{1};
@@ -103,7 +109,7 @@ public:
* @param onNewLedger The onNewLedger hook. Called when a new ledger is received
* @param onLedgerClosed The onLedgerClosed hook. Called when the ledger is closed but only if the source is
* forwarding
* @param connectionTimeout The connection timeout. Defaults to 30 seconds
* @param wsTimeout A timeout for websocket operations. Defaults to 30 seconds
* @param retryDelay The retry delay. Defaults to 1 second
*/
SubscriptionSource(
@@ -115,7 +121,7 @@ public:
OnConnectHook onConnect,
OnDisconnectHook onDisconnect,
OnLedgerClosedHook onLedgerClosed,
std::chrono::steady_clock::duration const connectionTimeout = CONNECTION_TIMEOUT,
std::chrono::steady_clock::duration const wsTimeout = WS_TIMEOUT,
std::chrono::steady_clock::duration const retryDelay = RETRY_DELAY
);

View File

@@ -104,13 +104,17 @@ ProposedTransactionFeed::pub(boost::json::object const& receivedTxJson)
boost::asio::post(strand_, [this, pubMsg = std::move(pubMsg), affectedAccounts = std::move(affectedAccounts)]() {
notified_.clear();
signal_.emit(pubMsg);
// Prevent the same connection from receiving the same message twice if it is subscribed to multiple accounts
// However, if the same connection subscribe both stream and account, it will still receive the message twice.
// notified_ can be cleared before signal_ emit to improve this, but let's keep it as is for now, since rippled
// acts like this.
notified_.clear();
for (auto const& account : affectedAccounts)
accountSignal_.emit(account, pubMsg);
++pubCount_.get();
});
}

View File

@@ -24,6 +24,7 @@
#include "feed/impl/TrackableSignalMap.hpp"
#include "feed/impl/Util.hpp"
#include "util/log/Logger.hpp"
#include "util/prometheus/Counter.hpp"
#include "util/prometheus/Gauge.hpp"
#include <boost/asio/io_context.hpp>
@@ -54,6 +55,7 @@ class ProposedTransactionFeed {
boost::asio::strand<boost::asio::io_context::executor_type> strand_;
std::reference_wrapper<util::prometheus::GaugeInt> subAllCount_;
std::reference_wrapper<util::prometheus::GaugeInt> subAccountCount_;
std::reference_wrapper<util::prometheus::CounterInt> pubCount_;
TrackableSignalMap<ripple::AccountID, Subscriber, std::shared_ptr<std::string>> accountSignal_;
TrackableSignal<Subscriber, std::shared_ptr<std::string>> signal_;
@@ -67,7 +69,7 @@ public:
: strand_(boost::asio::make_strand(ioContext))
, subAllCount_(getSubscriptionsGaugeInt("tx_proposed"))
, subAccountCount_(getSubscriptionsGaugeInt("account_proposed"))
, pubCount_(getPublishedMessagesCounterInt("tx_proposed"))
{
}

View File

@@ -36,7 +36,10 @@
namespace feed::impl {
SingleFeedBase::SingleFeedBase(boost::asio::io_context& ioContext, std::string const& name)
: strand_(boost::asio::make_strand(ioContext)), subCount_(getSubscriptionsGaugeInt(name)), name_(name)
: strand_(boost::asio::make_strand(ioContext))
, subCount_(getSubscriptionsGaugeInt(name))
, pubCount_(getPublishedMessagesCounterInt(name))
, name_(name)
{
}
@@ -70,6 +73,7 @@ SingleFeedBase::pub(std::string msg) const
boost::asio::post(strand_, [this, msg = std::move(msg)]() mutable {
auto const msgPtr = std::make_shared<std::string>(std::move(msg));
signal_.emit(msgPtr);
++pubCount_.get();
});
}

View File

@@ -22,6 +22,7 @@
#include "feed/Types.hpp"
#include "feed/impl/TrackableSignal.hpp"
#include "util/log/Logger.hpp"
#include "util/prometheus/Counter.hpp"
#include "util/prometheus/Gauge.hpp"
#include <boost/asio/io_context.hpp>
@@ -40,6 +41,7 @@ namespace feed::impl {
class SingleFeedBase {
boost::asio::strand<boost::asio::io_context::executor_type> strand_;
std::reference_wrapper<util::prometheus::GaugeInt> subCount_;
std::reference_wrapper<util::prometheus::CounterInt> pubCount_;
TrackableSignal<Subscriber, std::shared_ptr<std::string> const&> signal_;
util::Logger logger_{"Subscriptions"};
std::string name_;

View File

@@ -284,23 +284,29 @@ TransactionFeed::pub(
affectedBooks = std::move(affectedBooks)]() {
notified_.clear();
signal_.emit(allVersionsMsgs);
// clear the notified set. If the same connection subscribes both transactions + proposed_transactions,
// rippled SENDS the same message twice
notified_.clear();
txProposedsignal_.emit(allVersionsMsgs);
notified_.clear();
// check duplicate for account and proposed_account, this prevents sending the same message multiple times
// if it affects multiple accounts watched by the same connection
for (auto const& account : affectedAccounts) {
accountSignal_.emit(account, allVersionsMsgs);
accountProposedSignal_.emit(account, allVersionsMsgs);
}
notified_.clear();
// check duplicate for books, this prevents sending the same message multiple times if it affects multiple
// books watched by the same connection
for (auto const& book : affectedBooks) {
bookSignal_.emit(book, allVersionsMsgs);
}
++pubCount_.get();
}
);
}

View File

@@ -26,6 +26,7 @@
#include "feed/impl/TrackableSignalMap.hpp"
#include "feed/impl/Util.hpp"
#include "util/log/Logger.hpp"
#include "util/prometheus/Counter.hpp"
#include "util/prometheus/Gauge.hpp"
#include <boost/asio/io_context.hpp>
@@ -67,6 +68,7 @@ class TransactionFeed {
std::reference_wrapper<util::prometheus::GaugeInt> subAllCount_;
std::reference_wrapper<util::prometheus::GaugeInt> subAccountCount_;
std::reference_wrapper<util::prometheus::GaugeInt> subBookCount_;
std::reference_wrapper<util::prometheus::CounterInt> pubCount_;
TrackableSignalMap<ripple::AccountID, Subscriber, AllVersionTransactionsType const&> accountSignal_;
TrackableSignalMap<ripple::Book, Subscriber, AllVersionTransactionsType const&> bookSignal_;
@@ -89,6 +91,7 @@ public:
, subAllCount_(getSubscriptionsGaugeInt("tx"))
, subAccountCount_(getSubscriptionsGaugeInt("account"))
, subBookCount_(getSubscriptionsGaugeInt("book"))
, pubCount_(getPublishedMessagesCounterInt("tx"))
{
}

View File

@@ -19,6 +19,7 @@
#pragma once
#include "util/prometheus/Counter.hpp"
#include "util/prometheus/Gauge.hpp"
#include "util/prometheus/Label.hpp"
#include "util/prometheus/Prometheus.hpp"
@@ -38,4 +39,14 @@ getSubscriptionsGaugeInt(std::string const& counterName)
fmt::format("Current subscribers number on the {} stream", counterName)
);
}
inline util::prometheus::CounterInt&
getPublishedMessagesCounterInt(std::string const& counterName)
{
return PrometheusService::counterInt(
"subscriptions_published_count",
util::prometheus::Labels({util::prometheus::Label{"stream", counterName}}),
fmt::format("Total published messages on the {} stream", counterName)
);
}
} // namespace feed::impl

View File

@@ -11,18 +11,16 @@ target_sources(clio_server PRIVATE Main.cpp)
target_link_libraries(clio_server PRIVATE clio)
if (static)
target_link_options(clio_server PRIVATE -static)
if (is_gcc AND NOT san)
if (san)
message(FATAL_ERROR "Static linkage not allowed when using sanitizers")
elseif (is_appleclang)
message(FATAL_ERROR "Static linkage not supported on AppleClang")
else ()
target_link_options(
# For now let's assume that we only using libstdc++ under gcc.
# Note: -static-libstdc++ can statically link both libstdc++ and libc++
clio_server PRIVATE -static-libstdc++ -static-libgcc
)
endif ()
if (is_appleclang)
message(FATAL_ERROR "Static linkage not supported on AppleClang")
endif ()
endif ()
set_target_properties(clio_server PROPERTIES RUNTIME_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR})

View File

@@ -136,7 +136,7 @@ parseCerts(Config const& config)
readKey.close();
std::string key = contents.str();
ssl::context ctx{ssl::context::tlsv12};
ssl::context ctx{ssl::context::tls_server};
ctx.set_options(ssl::context::default_workarounds | ssl::context::no_sslv2);
ctx.use_certificate_chain(buffer(cert.data(), cert.size()));
ctx.use_private_key(buffer(key.data(), key.size()), ssl::context::file_format::pem);

View File

@@ -25,6 +25,7 @@ target_sources(
handlers/BookChanges.cpp
handlers/BookOffers.cpp
handlers/DepositAuthorized.cpp
handlers/Feature.cpp
handlers/GatewayBalances.cpp
handlers/GetAggregatePrice.cpp
handlers/Ledger.cpp

View File

@@ -22,6 +22,7 @@
#include "data/BackendInterface.hpp"
#include "rpc/Counters.hpp"
#include "rpc/Errors.hpp"
#include "rpc/RPCHelpers.hpp"
#include "rpc/WorkQueue.hpp"
#include "rpc/common/HandlerProvider.hpp"
#include "rpc/common/Types.hpp"
@@ -134,8 +135,13 @@ public:
Result
buildResponse(web::Context const& ctx)
{
if (forwardingProxy_.shouldForward(ctx))
if (forwardingProxy_.shouldForward(ctx)) {
// Disallow forwarding of the admin api, only user api is allowed for security reasons.
if (isAdminCmd(ctx.method, ctx.params))
return Result{Status{RippledError::rpcNO_PERMISSION}};
return forwardingProxy_.forward(ctx);
}
if (backend_->isTooBusy()) {
LOG(log_.error()) << "Database is too busy. Rejecting request";

View File

@@ -24,6 +24,7 @@
#include "rpc/Errors.hpp"
#include "rpc/JS.hpp"
#include "rpc/common/Types.hpp"
#include "util/AccountUtils.hpp"
#include "util/Profiler.hpp"
#include "util/log/Logger.hpp"
#include "web/Context.hpp"
@@ -35,6 +36,7 @@
#include <boost/json/array.hpp>
#include <boost/json/object.hpp>
#include <boost/json/parse.hpp>
#include <boost/json/serialize.hpp>
#include <boost/json/string.hpp>
#include <boost/json/value.hpp>
#include <boost/json/value_to.hpp>
@@ -48,6 +50,7 @@
#include <ripple/basics/chrono.h>
#include <ripple/basics/strHex.h>
#include <ripple/beast/utility/Zero.h>
#include <ripple/json/json_reader.h>
#include <ripple/json/json_value.h>
#include <ripple/protocol/AccountID.h>
#include <ripple/protocol/Book.h>
@@ -186,14 +189,14 @@ accountFromStringStrict(std::string const& account)
if (blob && ripple::publicKeyType(ripple::makeSlice(*blob))) {
publicKey = ripple::PublicKey(ripple::Slice{blob->data(), blob->size()});
} else {
publicKey = ripple::parseBase58<ripple::PublicKey>(ripple::TokenType::AccountPublic, account);
publicKey = util::parseBase58Wrapper<ripple::PublicKey>(ripple::TokenType::AccountPublic, account);
}
std::optional<ripple::AccountID> result;
if (publicKey) {
result = ripple::calcAccountID(*publicKey);
} else {
result = ripple::parseBase58<ripple::AccountID>(account);
result = util::parseBase58Wrapper<ripple::AccountID>(account);
}
return result;
@@ -358,7 +361,7 @@ toJson(ripple::LedgerHeader const& lgrInfo, bool const binary, std::uint32_t con
{
boost::json::object header;
if (binary) {
header[JS(ledger_data)] = ripple::strHex(ledgerInfoToBlob(lgrInfo));
header[JS(ledger_data)] = ripple::strHex(ledgerHeaderToBlob(lgrInfo));
} else {
header[JS(account_hash)] = ripple::strHex(lgrInfo.accountHash);
header[JS(close_flags)] = lgrInfo.closeFlags;
@@ -396,7 +399,7 @@ parseStringAsUInt(std::string const& value)
}
std::variant<Status, ripple::LedgerHeader>
ledgerInfoFromRequest(std::shared_ptr<data::BackendInterface const> const& backend, web::Context const& ctx)
ledgerHeaderFromRequest(std::shared_ptr<data::BackendInterface const> const& backend, web::Context const& ctx)
{
auto hashValue = ctx.params.contains("ledger_hash") ? ctx.params.at("ledger_hash") : nullptr;
@@ -444,9 +447,9 @@ ledgerInfoFromRequest(std::shared_ptr<data::BackendInterface const> const& backe
return *lgrInfo;
}
// extract ledgerInfoFromRequest's parameter from context
// extract ledgerHeaderFromRequest's parameter from context
std::variant<Status, ripple::LedgerHeader>
getLedgerInfoFromHashOrSeq(
getLedgerHeaderFromHashOrSeq(
BackendInterface const& backend,
boost::asio::yield_context yield,
std::optional<std::string> ledgerHash,
@@ -479,7 +482,7 @@ getLedgerInfoFromHashOrSeq(
}
std::vector<unsigned char>
ledgerInfoToBlob(ripple::LedgerHeader const& info, bool includeHash)
ledgerHeaderToBlob(ripple::LedgerHeader const& info, bool includeHash)
{
ripple::Serializer s;
s.add32(info.seq);
@@ -799,7 +802,7 @@ getAccountsFromTransaction(boost::json::object const& transaction)
auto inObject = getAccountsFromTransaction(value.as_object());
accounts.insert(accounts.end(), inObject.begin(), inObject.end());
} else if (value.is_string()) {
auto const account = ripple::parseBase58<ripple::AccountID>(boost::json::value_to<std::string>(value));
auto const account = util::parseBase58Wrapper<ripple::AccountID>(boost::json::value_to<std::string>(value));
if (account) {
accounts.push_back(*account);
}
@@ -1272,6 +1275,31 @@ specifiesCurrentOrClosedLedger(boost::json::object const& request)
return false;
}
bool
isAdminCmd(std::string const& method, boost::json::object const& request)
{
if (method == JS(ledger)) {
auto const requestStr = boost::json::serialize(request);
Json::Value jv;
Json::Reader{}.parse(requestStr, jv);
// rippled considers string/non-zero int/non-empty array/ non-empty json as true.
// Use rippled's API asBool to get the same result.
// https://github.com/XRPLF/rippled/issues/5119
auto const isFieldSet = [&jv](auto const field) { return jv.isMember(field) and jv[field].asBool(); };
// According to doc
// https://xrpl.org/docs/references/http-websocket-apis/public-api-methods/ledger-methods/ledger,
// full/accounts/type are admin only, but type only works when full/accounts are set, so we don't need to check
// type.
if (isFieldSet(JS(full)) or isFieldSet(JS(accounts)))
return true;
}
if (method == JS(feature) and request.contains(JS(vetoed)))
return true;
return false;
}
std::variant<ripple::uint256, Status>
getNFTID(boost::json::object const& request)
{

View File

@@ -265,7 +265,7 @@ generatePubLedgerMessage(
* @return The ledger info or an error status
*/
std::variant<Status, ripple::LedgerHeader>
ledgerInfoFromRequest(std::shared_ptr<data::BackendInterface const> const& backend, web::Context const& ctx);
ledgerHeaderFromRequest(std::shared_ptr<data::BackendInterface const> const& backend, web::Context const& ctx);
/**
* @brief Get ledger info from hash or sequence
@@ -278,7 +278,7 @@ ledgerInfoFromRequest(std::shared_ptr<data::BackendInterface const> const& backe
* @return The ledger info or an error status
*/
std::variant<Status, ripple::LedgerHeader>
getLedgerInfoFromHashOrSeq(
getLedgerHeaderFromHashOrSeq(
BackendInterface const& backend,
boost::asio::yield_context yield,
std::optional<std::string> ledgerHash,
@@ -372,7 +372,7 @@ getAccountsFromTransaction(boost::json::object const& transaction);
* @return The blob
*/
std::vector<unsigned char>
ledgerInfoToBlob(ripple::LedgerHeader const& info, bool includeHash = false);
ledgerHeaderToBlob(ripple::LedgerHeader const& info, bool includeHash = false);
/**
* @brief Whether global frozen is set
@@ -557,6 +557,16 @@ parseIssue(boost::json::object const& issue);
bool
specifiesCurrentOrClosedLedger(boost::json::object const& request);
/**
* @brief Check whether a request requires administrative privileges on rippled side.
*
* @param method The method name to check
* @param request The request to check
* @return true if the request requires ADMIN role
*/
bool
isAdminCmd(std::string const& method, boost::json::object const& request);
/**
* @brief Get the NFTID from the request
*

View File

@@ -20,10 +20,8 @@
#pragma once
#include "rpc/Errors.hpp"
#include "rpc/common/Concepts.hpp"
#include "rpc/common/Specs.hpp"
#include "rpc/common/Types.hpp"
#include "rpc/common/Validators.hpp"
#include <boost/json/value.hpp>
#include <fmt/core.h>
@@ -146,10 +144,10 @@ public:
[[nodiscard]] MaybeError
verify(boost::json::value& value, std::string_view key) const
{
if (not value.is_object() or not value.as_object().contains(key.data()))
if (not value.is_object() or not value.as_object().contains(key))
return {}; // ignore. field does not exist, let 'required' fail instead
if (not rpc::validation::checkType<Type>(value.as_object().at(key.data())))
if (not rpc::validation::checkType<Type>(value.as_object().at(key)))
return {}; // ignore if type does not match
return processor_(value, key);
@@ -162,9 +160,10 @@ private:
/**
* @brief A meta-processor that wraps a validator and produces a custom error in case the wrapped validator fails.
*/
template <typename SomeRequirement>
template <typename RequirementOrModifierType>
requires SomeRequirement<RequirementOrModifierType> or SomeModifier<RequirementOrModifierType>
class WithCustomError final {
SomeRequirement requirement;
RequirementOrModifierType reqOrModifier;
Status error;
public:
@@ -172,10 +171,11 @@ public:
* @brief Constructs a validator that calls the given validator `req` and returns a custom error `err` in case `req`
* fails.
*
* @param req The requirement to validate against
* @param reqOrModifier The requirement to validate against
* @param err The custom error to return in case `req` fails
*/
WithCustomError(SomeRequirement req, Status err) : requirement{std::move(req)}, error{std::move(err)}
WithCustomError(RequirementOrModifierType reqOrModifier, Status err)
: reqOrModifier{std::move(reqOrModifier)}, error{std::move(err)}
{
}
@@ -188,8 +188,9 @@ public:
*/
[[nodiscard]] MaybeError
verify(boost::json::value const& value, std::string_view key) const
requires SomeRequirement<RequirementOrModifierType>
{
if (auto const res = requirement.verify(value, key); not res)
if (auto const res = reqOrModifier.verify(value, key); not res)
return Error{error};
return {};
@@ -205,12 +206,30 @@ public:
*/
[[nodiscard]] MaybeError
verify(boost::json::value& value, std::string_view key) const
requires SomeRequirement<RequirementOrModifierType>
{
if (auto const res = requirement.verify(value, key); not res)
if (auto const res = reqOrModifier.verify(value, key); not res)
return Error{error};
return {};
}
/**
* @brief Runs the stored modifier and produces a custom error if the wrapped modifier fails.
*
* @param value The JSON value representing the outer object. This value can be modified by the modifier.
* @param key The key used to retrieve the element from the outer object
* @return Possibly an error
*/
MaybeError
modify(boost::json::value& value, std::string_view key) const
requires SomeModifier<RequirementOrModifierType>
{
if (auto const res = reqOrModifier.modify(value, key); not res)
return Error{error};
return {};
}
};
} // namespace rpc::meta

View File

@@ -19,12 +19,16 @@
#pragma once
#include "rpc/Errors.hpp"
#include "rpc/common/Types.hpp"
#include "util/JsonUtils.hpp"
#include <boost/json/value.hpp>
#include <boost/json/value_to.hpp>
#include <ripple/protocol/ErrorCodes.h>
#include <exception>
#include <functional>
#include <string>
#include <string_view>
@@ -100,4 +104,75 @@ struct ToLower final {
}
};
/**
* @brief Convert input string to integer.
*
* Note: the conversion is only performed if the input value is a string.
*/
struct ToNumber final {
/**
* @brief Update the input string to integer if it can be converted to integer by stoi.
*
* @param value The JSON value representing the outer object
* @param key The key used to retrieve the modified value from the outer object
* @return Possibly an error
*/
[[nodiscard]] static MaybeError
modify(boost::json::value& value, std::string_view key)
{
if (not value.is_object() or not value.as_object().contains(key))
return {}; // ignore. field does not exist, let 'required' fail instead
if (not value.as_object().at(key).is_string())
return {}; // ignore for non-string types
auto const strInt = boost::json::value_to<std::string>(value.as_object().at(key));
if (strInt.find('.') != std::string::npos)
return Error{Status{RippledError::rpcINVALID_PARAMS}}; // maybe a float
try {
value.as_object()[key.data()] = std::stoi(strInt);
} catch (std::exception& e) {
return Error{Status{RippledError::rpcINVALID_PARAMS}};
}
return {};
}
};
/**
* @brief Customised modifier allowing user define how to modify input in provided callable.
*/
class CustomModifier final {
std::function<MaybeError(boost::json::value&, std::string_view)> modifier_;
public:
/**
* @brief Constructs a custom modifier from any supported callable.
*
* @tparam Fn The type of callable
* @param fn The callable/function object
*/
template <typename Fn>
requires std::invocable<Fn, boost::json::value&, std::string_view>
explicit CustomModifier(Fn&& fn) : modifier_{std::forward<Fn>(fn)}
{
}
/**
* @brief Modify the JSON value according to the custom modifier function stored.
*
* @param value The JSON value representing the outer object
* @param key The key used to retrieve the tested value from the outer object
* @return Any compatible user-provided error if modify/verify failed; otherwise no error is returned
*/
[[nodiscard]] MaybeError
modify(boost::json::value& value, std::string_view key) const
{
if (not value.is_object() or not value.as_object().contains(key))
return {}; // ignore. field does not exist, let 'required' fail instead
return modifier_(value.as_object().at(key.data()), key);
};
};
} // namespace rpc::modifiers

View File

@@ -94,7 +94,7 @@ struct ReturnType {
* @param warnings The warnings generated by the RPC call
*/
ReturnType(std::expected<boost::json::value, Status> result, boost::json::array warnings = {})
: result{std::move(result)}, warnings{std::move(warnings)}
: result{std::move(result)}, warnings(std::move(warnings))
{
}

View File

@@ -22,6 +22,7 @@
#include "rpc/Errors.hpp"
#include "rpc/RPCHelpers.hpp"
#include "rpc/common/Types.hpp"
#include "util/AccountUtils.hpp"
#include <boost/json/object.hpp>
#include <boost/json/value.hpp>
@@ -29,6 +30,7 @@
#include <fmt/core.h>
#include <ripple/basics/base_uint.h>
#include <ripple/protocol/AccountID.h>
#include <ripple/protocol/ErrorCodes.h>
#include <ripple/protocol/UintTypes.h>
#include <ripple/protocol/tokens.h>
@@ -45,7 +47,7 @@ namespace rpc::validation {
[[nodiscard]] MaybeError
Required::verify(boost::json::value const& value, std::string_view key)
{
if (not value.is_object() or not value.as_object().contains(key.data()))
if (not value.is_object() or not value.as_object().contains(key))
return Error{Status{RippledError::rpcINVALID_PARAMS, "Required field '" + std::string{key} + "' missing"}};
return {};
@@ -54,10 +56,10 @@ Required::verify(boost::json::value const& value, std::string_view key)
[[nodiscard]] MaybeError
CustomValidator::verify(boost::json::value const& value, std::string_view key) const
{
if (not value.is_object() or not value.as_object().contains(key.data()))
if (not value.is_object() or not value.as_object().contains(key))
return {}; // ignore. field does not exist, let 'required' fail instead
return validator_(value.as_object().at(key.data()), key);
return validator_(value.as_object().at(key), key);
}
[[nodiscard]] bool
@@ -113,7 +115,7 @@ CustomValidator AccountBase58Validator =
if (!value.is_string())
return Error{Status{RippledError::rpcINVALID_PARAMS, std::string(key) + "NotString"}};
auto const account = ripple::parseBase58<ripple::AccountID>(boost::json::value_to<std::string>(value));
auto const account = util::parseBase58Wrapper<ripple::AccountID>(boost::json::value_to<std::string>(value));
if (!account || account->isZero())
return Error{Status{ClioError::rpcMALFORMED_ADDRESS}};
@@ -140,8 +142,12 @@ CustomValidator CurrencyValidator =
if (!value.is_string())
return Error{Status{RippledError::rpcINVALID_PARAMS, std::string(key) + "NotString"}};
auto const currencyStr = boost::json::value_to<std::string>(value);
if (currencyStr.empty())
return Error{Status{RippledError::rpcINVALID_PARAMS, std::string(key) + "IsEmpty"}};
ripple::Currency currency;
if (!ripple::to_currency(currency, boost::json::value_to<std::string>(value)))
if (!ripple::to_currency(currency, currencyStr))
return Error{Status{ClioError::rpcMALFORMED_CURRENCY, "malformedCurrency"}};
return MaybeError{};

View File

@@ -402,6 +402,7 @@ public:
* @param fn The callable/function object
*/
template <typename Fn>
requires std::invocable<Fn, boost::json::value const&, std::string_view>
explicit CustomValidator(Fn&& fn) : validator_{std::forward<Fn>(fn)}
{
}

View File

@@ -55,17 +55,22 @@ public:
bool
shouldForward(web::Context const& ctx) const
{
auto const& request = ctx.params;
if (ctx.method == "subscribe" || ctx.method == "unsubscribe")
return false;
// TODO https://github.com/XRPLF/clio/issues/1131 - remove once clio-native feature is
// implemented fully. For now we disallow forwarding of the admin api, only user api is allowed.
if (ctx.method == "feature" and not request.contains("vetoed"))
return true;
if (handlerProvider_->isClioOnly(ctx.method))
return false;
if (isProxied(ctx.method))
return true;
auto const& request = ctx.params;
if (specifiesCurrentOrClosedLedger(request))
return true;
@@ -88,7 +93,7 @@ public:
auto toForward = ctx.params;
toForward["command"] = ctx.method;
auto res = balancer_->forwardToRippled(toForward, ctx.clientIp, ctx.yield);
auto res = balancer_->forwardToRippled(toForward, ctx.clientIp, ctx.isAdmin, ctx.yield);
if (not res) {
notifyFailedToForward(ctx.method);
return Result{Status{RippledError::rpcFAILED_TO_FORWARD}};

View File

@@ -36,6 +36,7 @@
#include "rpc/handlers/BookChanges.hpp"
#include "rpc/handlers/BookOffers.hpp"
#include "rpc/handlers/DepositAuthorized.hpp"
#include "rpc/handlers/Feature.hpp"
#include "rpc/handlers/GatewayBalances.hpp"
#include "rpc/handlers/GetAggregatePrice.hpp"
#include "rpc/handlers/Ledger.hpp"
@@ -85,6 +86,7 @@ ProductionHandlerProvider::ProductionHandlerProvider(
{"book_changes", {BookChangesHandler{backend}}},
{"book_offers", {BookOffersHandler{backend}}},
{"deposit_authorized", {DepositAuthorizedHandler{backend}}},
{"feature", {FeatureHandler{}}},
{"gateway_balances", {GatewayBalancesHandler{backend}}},
{"get_aggregate_price", {GetAggregatePriceHandler{backend}}},
{"ledger", {LedgerHandler{backend}}},

View File

@@ -94,14 +94,14 @@ AMMInfoHandler::process(AMMInfoHandler::Input input, Context const& ctx) const
return Error{Status{RippledError::rpcINVALID_PARAMS}};
auto const range = sharedPtrBackend_->fetchLedgerRange();
auto const lgrInfoOrStatus = getLedgerInfoFromHashOrSeq(
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, ctx.yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);
if (auto const status = std::get_if<Status>(&lgrInfoOrStatus))
return Error{*status};
auto const lgrInfo = std::get<LedgerInfo>(lgrInfoOrStatus);
auto const lgrInfo = std::get<LedgerHeader>(lgrInfoOrStatus);
if (input.accountID) {
auto keylet = keylet::account(*input.accountID);

View File

@@ -84,7 +84,7 @@ AccountChannelsHandler::Result
AccountChannelsHandler::process(AccountChannelsHandler::Input input, Context const& ctx) const
{
auto const range = sharedPtrBackend_->fetchLedgerRange();
auto const lgrInfoOrStatus = getLedgerInfoFromHashOrSeq(
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, ctx.yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);

View File

@@ -46,7 +46,7 @@ AccountCurrenciesHandler::Result
AccountCurrenciesHandler::process(AccountCurrenciesHandler::Input input, Context const& ctx) const
{
auto const range = sharedPtrBackend_->fetchLedgerRange();
auto const lgrInfoOrStatus = getLedgerInfoFromHashOrSeq(
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, ctx.yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);

View File

@@ -56,7 +56,7 @@ AccountInfoHandler::process(AccountInfoHandler::Input input, Context const& ctx)
return Error{Status{RippledError::rpcINVALID_PARAMS, ripple::RPC::missing_field_message(JS(account))}};
auto const range = sharedPtrBackend_->fetchLedgerRange();
auto const lgrInfoOrStatus = getLedgerInfoFromHashOrSeq(
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, ctx.yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);

View File

@@ -120,7 +120,7 @@ AccountLinesHandler::Result
AccountLinesHandler::process(AccountLinesHandler::Input input, Context const& ctx) const
{
auto const range = sharedPtrBackend_->fetchLedgerRange();
auto const lgrInfoOrStatus = getLedgerInfoFromHashOrSeq(
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, ctx.yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);

View File

@@ -52,7 +52,7 @@ AccountNFTsHandler::Result
AccountNFTsHandler::process(AccountNFTsHandler::Input input, Context const& ctx) const
{
auto const range = sharedPtrBackend_->fetchLedgerRange();
auto const lgrInfoOrStatus = getLedgerInfoFromHashOrSeq(
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, ctx.yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);

View File

@@ -52,7 +52,7 @@ AccountObjectsHandler::Result
AccountObjectsHandler::process(AccountObjectsHandler::Input input, Context const& ctx) const
{
auto const range = sharedPtrBackend_->fetchLedgerRange();
auto const lgrInfoOrStatus = getLedgerInfoFromHashOrSeq(
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, ctx.yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);

View File

@@ -68,7 +68,7 @@ AccountOffersHandler::Result
AccountOffersHandler::process(AccountOffersHandler::Input input, Context const& ctx) const
{
auto const range = sharedPtrBackend_->fetchLedgerRange();
auto const lgrInfoOrStatus = getLedgerInfoFromHashOrSeq(
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, ctx.yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);

View File

@@ -91,7 +91,7 @@ AccountTxHandler::process(AccountTxHandler::Input input, Context const& ctx) con
if (!input.ledgerIndexMax && !input.ledgerIndexMin) {
// mimic rippled, when both range and index specified, respect the range.
// take ledger from ledgerHash or ledgerIndex only when range is not specified
auto const lgrInfoOrStatus = getLedgerInfoFromHashOrSeq(
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, ctx.yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);
@@ -169,11 +169,11 @@ AccountTxHandler::process(AccountTxHandler::Input input, Context const& ctx) con
obj[JS(hash)] = obj[txKey].as_object()[JS(hash)];
obj[txKey].as_object().erase(JS(hash));
}
if (auto const ledgerInfo =
if (auto const ledgerHeader =
sharedPtrBackend_->fetchLedgerBySequence(txnPlusMeta.ledgerSequence, ctx.yield);
ledgerInfo) {
obj[JS(ledger_hash)] = ripple::strHex(ledgerInfo->hash);
obj[JS(close_time_iso)] = ripple::to_string_iso(ledgerInfo->closeTime);
ledgerHeader) {
obj[JS(ledger_hash)] = ripple::strHex(ledgerHeader->hash);
obj[JS(close_time_iso)] = ripple::to_string_iso(ledgerHeader->closeTime);
}
}
obj[JS(validated)] = true;

View File

@@ -39,7 +39,6 @@
#include <ripple/protocol/jss.h>
#include <cstdint>
#include <limits>
#include <memory>
#include <optional>
#include <string>
@@ -57,8 +56,8 @@ class AccountTxHandler {
std::shared_ptr<BackendInterface> sharedPtrBackend_;
public:
// no max limit
static auto constexpr LIMIT_MIN = 1;
static auto constexpr LIMIT_MAX = 1000;
static auto constexpr LIMIT_DEFAULT = 200;
/**
@@ -133,7 +132,7 @@ public:
{JS(limit),
validation::Type<uint32_t>{},
validation::Min(1u),
modifiers::Clamp<int32_t>{LIMIT_MIN, std::numeric_limits<int32_t>::max()}},
modifiers::Clamp<int32_t>{LIMIT_MIN, LIMIT_MAX}},
{JS(marker),
meta::WithCustomError{
validation::Type<boost::json::object>{},

View File

@@ -45,7 +45,7 @@ BookChangesHandler::Result
BookChangesHandler::process(BookChangesHandler::Input input, Context const& ctx) const
{
auto const range = sharedPtrBackend_->fetchLedgerRange();
auto const lgrInfoOrStatus = getLedgerInfoFromHashOrSeq(
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, ctx.yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);

View File

@@ -51,7 +51,7 @@ BookOffersHandler::process(Input input, Context const& ctx) const
// check ledger
auto const range = sharedPtrBackend_->fetchLedgerRange();
auto const lgrInfoOrStatus = getLedgerInfoFromHashOrSeq(
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, ctx.yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);

View File

@@ -46,7 +46,7 @@ DepositAuthorizedHandler::Result
DepositAuthorizedHandler::process(DepositAuthorizedHandler::Input input, Context const& ctx) const
{
auto const range = sharedPtrBackend_->fetchLedgerRange();
auto const lgrInfoOrStatus = getLedgerInfoFromHashOrSeq(
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, ctx.yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);

View File

@@ -0,0 +1,84 @@
//------------------------------------------------------------------------------
/*
This file is part of clio: https://github.com/XRPLF/clio
Copyright (c) 2024, the clio developers.
Permission to use, copy, modify, and distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
//==============================================================================
#include "rpc/handlers/Feature.hpp"
#include "rpc/Errors.hpp"
#include "rpc/JS.hpp"
#include "rpc/common/MetaProcessors.hpp"
#include "rpc/common/Specs.hpp"
#include "rpc/common/Types.hpp"
#include "rpc/common/Validators.hpp"
#include "util/Assert.hpp"
#include <boost/json/conversion.hpp>
#include <boost/json/value.hpp>
#include <ripple/protocol/jss.h>
#include <cstdint>
#include <string>
namespace rpc {
FeatureHandler::Result
FeatureHandler::process([[maybe_unused]] FeatureHandler::Input input, [[maybe_unused]] Context const& ctx)
{
// For now this handler only fires when "vetoed" is set in the request.
// This always leads to a `notSupported` error as we don't want anyone to be able to
ASSERT(false, "FeatureHandler::process is not implemented.");
return Output{};
}
RpcSpecConstRef
FeatureHandler::spec([[maybe_unused]] uint32_t apiVersion)
{
static RpcSpec const rpcSpec = {
{JS(feature), validation::Type<std::string>{}},
{JS(vetoed),
meta::WithCustomError{
validation::NotSupported{},
Status(RippledError::rpcNO_PERMISSION, "The admin portion of feature API is not available through Clio.")
}},
};
return rpcSpec;
}
void
tag_invoke(boost::json::value_from_tag, boost::json::value& jv, FeatureHandler::Output const& output)
{
using boost::json::value_from;
jv = {
{JS(validated), output.validated},
};
}
FeatureHandler::Input
tag_invoke(boost::json::value_to_tag<FeatureHandler::Input>, boost::json::value const& jv)
{
auto input = FeatureHandler::Input{};
auto const jsonObject = jv.as_object();
if (jsonObject.contains(JS(feature)))
input.feature = jv.at(JS(feature)).as_string();
return input;
}
} // namespace rpc

View File

@@ -0,0 +1,95 @@
//------------------------------------------------------------------------------
/*
This file is part of clio: https://github.com/XRPLF/clio
Copyright (c) 2024, the clio developers.
Permission to use, copy, modify, and distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
//==============================================================================
#pragma once
#include "rpc/common/Specs.hpp"
#include "rpc/common/Types.hpp"
#include <boost/json/conversion.hpp>
#include <boost/json/value.hpp>
#include <ripple/protocol/jss.h>
#include <cstdint>
#include <string>
namespace rpc {
/**
* @brief Contains common functionality for handling the `server_info` command
*/
class FeatureHandler {
public:
/**
* @brief A struct to hold the input data for the command
*/
struct Input {
std::string feature;
};
/**
* @brief A struct to hold the output data of the command
*/
struct Output {
// validated should be sent via framework
bool validated = true;
};
using Result = HandlerReturnType<Output>;
/**
* @brief Returns the API specification for the command
*
* @param apiVersion The api version to return the spec for
* @return The spec for the given apiVersion
*/
static RpcSpecConstRef
spec([[maybe_unused]] uint32_t apiVersion);
/**
* @brief Process the Feature command
*
* @param input The input data for the command
* @param ctx The context of the request
* @return The result of the operation
*/
static Result
process(Input input, Context const& ctx); // NOLINT(readability-convert-member-functions-to-static)
private:
/**
* @brief Convert the Output to a JSON object
*
* @param [out] jv The JSON object to convert to
* @param output The output to convert
*/
friend void
tag_invoke(boost::json::value_from_tag, boost::json::value& jv, Output const& output);
/**
* @brief Convert a JSON object to Input type
*
* @param jv The JSON object to convert
* @return Input parsed from the JSON object
*/
friend Input
tag_invoke(boost::json::value_to_tag<Input>, boost::json::value const& jv);
};
} // namespace rpc

View File

@@ -59,7 +59,7 @@ GatewayBalancesHandler::process(GatewayBalancesHandler::Input input, Context con
{
// check ledger
auto const range = sharedPtrBackend_->fetchLedgerRange();
auto const lgrInfoOrStatus = getLedgerInfoFromHashOrSeq(
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, ctx.yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);

View File

@@ -25,6 +25,7 @@
#include "rpc/common/Specs.hpp"
#include "rpc/common/Types.hpp"
#include "rpc/common/Validators.hpp"
#include "util/AccountUtils.hpp"
#include <boost/json/array.hpp>
#include <boost/json/conversion.hpp>
@@ -116,14 +117,14 @@ public:
auto const wallets = value.is_array() ? value.as_array() : boost::json::array{value};
auto const getAccountID = [](auto const& j) -> std::optional<ripple::AccountID> {
if (j.is_string()) {
auto const pk = ripple::parseBase58<ripple::PublicKey>(
auto const pk = util::parseBase58Wrapper<ripple::PublicKey>(
ripple::TokenType::AccountPublic, boost::json::value_to<std::string>(j)
);
if (pk)
return ripple::calcAccountID(*pk);
return ripple::parseBase58<ripple::AccountID>(boost::json::value_to<std::string>(j));
return util::parseBase58Wrapper<ripple::AccountID>(boost::json::value_to<std::string>(j));
}
return {};

View File

@@ -23,6 +23,7 @@
#include "rpc/JS.hpp"
#include "rpc/RPCHelpers.hpp"
#include "rpc/common/Types.hpp"
#include "util/AccountUtils.hpp"
#include <boost/asio/spawn.hpp>
#include <boost/bimap/bimap.hpp>
@@ -61,7 +62,7 @@ GetAggregatePriceHandler::Result
GetAggregatePriceHandler::process(GetAggregatePriceHandler::Input input, Context const& ctx) const
{
auto const range = sharedPtrBackend_->fetchLedgerRange();
auto const lgrInfoOrStatus = getLedgerInfoFromHashOrSeq(
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, ctx.yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);
@@ -263,7 +264,7 @@ tag_invoke(boost::json::value_to_tag<GetAggregatePriceHandler::Input>, boost::js
for (auto const& oracle : jsonObject.at(JS(oracles)).as_array()) {
input.oracles.push_back(GetAggregatePriceHandler::Oracle{
.documentId = boost::json::value_to<std::uint64_t>(oracle.as_object().at(JS(oracle_document_id))),
.account = *ripple::parseBase58<ripple::AccountID>(
.account = *util::parseBase58Wrapper<ripple::AccountID>(
boost::json::value_to<std::string>(oracle.as_object().at(JS(account)))
)
});

View File

@@ -22,6 +22,8 @@
#include "data/BackendInterface.hpp"
#include "rpc/Errors.hpp"
#include "rpc/JS.hpp"
#include "rpc/common/MetaProcessors.hpp"
#include "rpc/common/Modifiers.hpp"
#include "rpc/common/Specs.hpp"
#include "rpc/common/Types.hpp"
#include "rpc/common/Validators.hpp"
@@ -31,7 +33,6 @@
#include <boost/json/conversion.hpp>
#include <ripple/basics/Number.h>
#include <ripple/protocol/AccountID.h>
#include <ripple/protocol/ErrorCodes.h>
#include <ripple/protocol/STAmount.h>
#include <ripple/protocol/STObject.h>
#include <ripple/protocol/jss.h>
@@ -44,6 +45,7 @@
#include <string_view>
#include <utility>
#include <vector>
namespace rpc {
/**
@@ -131,23 +133,26 @@ public:
static auto constexpr ORACLES_MAX = 200;
static auto const oraclesValidator =
validation::CustomValidator{[](boost::json::value const& value, std::string_view) -> MaybeError {
modifiers::CustomModifier{[](boost::json::value& value, std::string_view) -> MaybeError {
if (!value.is_array() or value.as_array().empty() or value.as_array().size() > ORACLES_MAX)
return Error{Status{RippledError::rpcORACLE_MALFORMED}};
for (auto oracle : value.as_array()) {
for (auto& oracle : value.as_array()) {
if (!oracle.is_object() or !oracle.as_object().contains(JS(oracle_document_id)) or
!oracle.as_object().contains(JS(account)))
return Error{Status{RippledError::rpcORACLE_MALFORMED}};
auto maybeError =
validation::Type<std::uint32_t>{}.verify(oracle.as_object(), JS(oracle_document_id));
auto maybeError = validation::Type<std::uint32_t, std::string>{}.verify(
oracle.as_object(), JS(oracle_document_id)
);
if (!maybeError)
return maybeError;
maybeError = modifiers::ToNumber::modify(oracle, JS(oracle_document_id));
if (!maybeError)
return maybeError;
maybeError = validation::AccountBase58Validator.verify(oracle.as_object(), JS(account));
if (!maybeError)
return Error{Status{RippledError::rpcINVALID_PARAMS}};
};
@@ -158,11 +163,15 @@ public:
static auto const rpcSpec = RpcSpec{
{JS(ledger_hash), validation::Uint256HexStringValidator},
{JS(ledger_index), validation::LedgerIndexValidator},
// note: Rippled's base_asset and quote_asset can be non-string. It will eventually return
// "rpcOBJECT_NOT_FOUND". Clio will return "rpcINVALID_PARAMS" if the base_asset or quote_asset is not a
// string. User can clearly know there is a mistake in the input.
{JS(base_asset), validation::Required{}, validation::Type<std::string>{}},
{JS(quote_asset), validation::Required{}, validation::Type<std::string>{}},
// validate quoteAsset and base_asset in accordance to the currency code found in XRPL doc:
// https://xrpl.org/docs/references/protocol/data-types/currency-formats#currency-codes
// usually Clio returns rpcMALFORMED_CURRENCY , return InvalidParam here just to mimic rippled
{JS(base_asset),
validation::Required{},
meta::WithCustomError{validation::CurrencyValidator, Status(RippledError::rpcINVALID_PARAMS)}},
{JS(quote_asset),
validation::Required{},
meta::WithCustomError{validation::CurrencyValidator, Status(RippledError::rpcINVALID_PARAMS)}},
{JS(oracles), validation::Required{}, oraclesValidator},
// note: Unlike `rippled`, Clio only supports UInt as input, no string, no `null`, etc.
{JS(time_threshold), validation::Type<std::uint32_t>{}},

View File

@@ -52,7 +52,7 @@ LedgerHandler::Result
LedgerHandler::process(LedgerHandler::Input input, Context const& ctx) const
{
auto const range = sharedPtrBackend_->fetchLedgerRange();
auto const lgrInfoOrStatus = getLedgerInfoFromHashOrSeq(
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, ctx.yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);

View File

@@ -70,9 +70,8 @@ public:
* - ledger
* - type
*
* Clio will throw an error when `queue` is set to `true`
* or if `full` or `accounts` are used.
* @see https://github.com/XRPLF/clio/issues/603
* Clio will throw an error when `queue`, `full` or `accounts` is set to `true`.
* @see https://github.com/XRPLF/clio/issues/603 and https://github.com/XRPLF/clio/issues/1537
*/
struct Input {
std::optional<std::string> ledgerHash;
@@ -105,9 +104,9 @@ public:
spec([[maybe_unused]] uint32_t apiVersion)
{
static auto const rpcSpec = RpcSpec{
{JS(full), validation::NotSupported{}},
{JS(full), validation::Type<bool>{}, validation::NotSupported{true}},
{JS(full), check::Deprecated{}},
{JS(accounts), validation::NotSupported{}},
{JS(accounts), validation::Type<bool>{}, validation::NotSupported{true}},
{JS(accounts), check::Deprecated{}},
{JS(owner_funds), validation::Type<bool>{}},
{JS(queue), validation::Type<bool>{}, validation::NotSupported{true}},

View File

@@ -61,7 +61,7 @@ LedgerDataHandler::process(Input input, Context const& ctx) const
return Error{Status{RippledError::rpcINVALID_PARAMS, "markerNotString"}};
auto const range = sharedPtrBackend_->fetchLedgerRange();
auto const lgrInfoOrStatus = getLedgerInfoFromHashOrSeq(
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, ctx.yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);

View File

@@ -23,6 +23,7 @@
#include "rpc/JS.hpp"
#include "rpc/RPCHelpers.hpp"
#include "rpc/common/Types.hpp"
#include "util/AccountUtils.hpp"
#include <boost/json/conversion.hpp>
#include <boost/json/object.hpp>
@@ -62,9 +63,9 @@ LedgerEntryHandler::process(LedgerEntryHandler::Input input, Context const& ctx)
if (input.index) {
key = ripple::uint256{std::string_view(*(input.index))};
} else if (input.accountRoot) {
key = ripple::keylet::account(*ripple::parseBase58<ripple::AccountID>(*(input.accountRoot))).key;
key = ripple::keylet::account(*util::parseBase58Wrapper<ripple::AccountID>(*(input.accountRoot))).key;
} else if (input.did) {
key = ripple::keylet::did(*ripple::parseBase58<ripple::AccountID>(*(input.did))).key;
key = ripple::keylet::did(*util::parseBase58Wrapper<ripple::AccountID>(*(input.did))).key;
} else if (input.directory) {
auto const keyOrStatus = composeKeyFromDirectory(*input.directory);
if (auto const status = std::get_if<Status>(&keyOrStatus))
@@ -73,13 +74,14 @@ LedgerEntryHandler::process(LedgerEntryHandler::Input input, Context const& ctx)
key = std::get<ripple::uint256>(keyOrStatus);
} else if (input.offer) {
auto const id =
ripple::parseBase58<ripple::AccountID>(boost::json::value_to<std::string>(input.offer->at(JS(account))));
util::parseBase58Wrapper<ripple::AccountID>(boost::json::value_to<std::string>(input.offer->at(JS(account)))
);
key = ripple::keylet::offer(*id, boost::json::value_to<std::uint32_t>(input.offer->at(JS(seq)))).key;
} else if (input.rippleStateAccount) {
auto const id1 = ripple::parseBase58<ripple::AccountID>(
auto const id1 = util::parseBase58Wrapper<ripple::AccountID>(
boost::json::value_to<std::string>(input.rippleStateAccount->at(JS(accounts)).as_array().at(0))
);
auto const id2 = ripple::parseBase58<ripple::AccountID>(
auto const id2 = util::parseBase58Wrapper<ripple::AccountID>(
boost::json::value_to<std::string>(input.rippleStateAccount->at(JS(accounts)).as_array().at(1))
);
auto const currency =
@@ -88,20 +90,22 @@ LedgerEntryHandler::process(LedgerEntryHandler::Input input, Context const& ctx)
key = ripple::keylet::line(*id1, *id2, currency).key;
} else if (input.escrow) {
auto const id =
ripple::parseBase58<ripple::AccountID>(boost::json::value_to<std::string>(input.escrow->at(JS(owner))));
util::parseBase58Wrapper<ripple::AccountID>(boost::json::value_to<std::string>(input.escrow->at(JS(owner)))
);
key = ripple::keylet::escrow(*id, input.escrow->at(JS(seq)).as_int64()).key;
} else if (input.depositPreauth) {
auto const owner = ripple::parseBase58<ripple::AccountID>(
auto const owner = util::parseBase58Wrapper<ripple::AccountID>(
boost::json::value_to<std::string>(input.depositPreauth->at(JS(owner)))
);
auto const authorized = ripple::parseBase58<ripple::AccountID>(
auto const authorized = util::parseBase58Wrapper<ripple::AccountID>(
boost::json::value_to<std::string>(input.depositPreauth->at(JS(authorized)))
);
key = ripple::keylet::depositPreauth(*owner, *authorized).key;
} else if (input.ticket) {
auto const id =
ripple::parseBase58<ripple::AccountID>(boost::json::value_to<std::string>(input.ticket->at(JS(account))));
util::parseBase58Wrapper<ripple::AccountID>(boost::json::value_to<std::string>(input.ticket->at(JS(account))
));
key = ripple::getTicketIndex(*id, input.ticket->at(JS(ticket_seq)).as_int64());
} else if (input.amm) {
@@ -112,7 +116,8 @@ LedgerEntryHandler::process(LedgerEntryHandler::Input input, Context const& ctx)
return ripple::xrpIssue();
}
auto const issuer =
ripple::parseBase58<ripple::AccountID>(boost::json::value_to<std::string>(assetJson.at(JS(issuer))));
util::parseBase58Wrapper<ripple::AccountID>(boost::json::value_to<std::string>(assetJson.at(JS(issuer)))
);
return ripple::Issue{currency, *issuer};
};
@@ -125,7 +130,7 @@ LedgerEntryHandler::process(LedgerEntryHandler::Input input, Context const& ctx)
return Error{Status{ClioError::rpcMALFORMED_REQUEST}};
if (input.bridgeAccount) {
auto const bridgeAccount = ripple::parseBase58<ripple::AccountID>(*(input.bridgeAccount));
auto const bridgeAccount = util::parseBase58Wrapper<ripple::AccountID>(*(input.bridgeAccount));
auto const chainType = ripple::STXChainBridge::srcChain(bridgeAccount == input.bridge->lockingChainDoor());
if (bridgeAccount != input.bridge->door(chainType))
@@ -149,7 +154,7 @@ LedgerEntryHandler::process(LedgerEntryHandler::Input input, Context const& ctx)
// check ledger exists
auto const range = sharedPtrBackend_->fetchLedgerRange();
auto const lgrInfoOrStatus = getLedgerInfoFromHashOrSeq(
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, ctx.yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);
@@ -201,7 +206,7 @@ LedgerEntryHandler::composeKeyFromDirectory(boost::json::object const& directory
}
auto const ownerID =
ripple::parseBase58<ripple::AccountID>(boost::json::value_to<std::string>(directory.at(JS(owner))));
util::parseBase58Wrapper<ripple::AccountID>(boost::json::value_to<std::string>(directory.at(JS(owner))));
return ripple::keylet::page(ripple::keylet::ownerDir(*ownerID), subIndex).key;
}
@@ -262,10 +267,10 @@ tag_invoke(boost::json::value_to_tag<LedgerEntryHandler::Input>, boost::json::va
};
auto const parseBridgeFromJson = [](boost::json::value const& bridgeJson) {
auto const lockingDoor = *ripple::parseBase58<ripple::AccountID>(
auto const lockingDoor = *util::parseBase58Wrapper<ripple::AccountID>(
boost::json::value_to<std::string>(bridgeJson.at(ripple::sfLockingChainDoor.getJsonName().c_str()))
);
auto const issuingDoor = *ripple::parseBase58<ripple::AccountID>(
auto const issuingDoor = *util::parseBase58Wrapper<ripple::AccountID>(
boost::json::value_to<std::string>(bridgeJson.at(ripple::sfIssuingChainDoor.getJsonName().c_str()))
);
auto const lockingIssue =
@@ -278,7 +283,7 @@ tag_invoke(boost::json::value_to_tag<LedgerEntryHandler::Input>, boost::json::va
auto const parseOracleFromJson = [](boost::json::value const& json) {
auto const account =
ripple::parseBase58<ripple::AccountID>(boost::json::value_to<std::string>(json.at(JS(account))));
util::parseBase58Wrapper<ripple::AccountID>(boost::json::value_to<std::string>(json.at(JS(account))));
auto const documentId = boost::json::value_to<uint32_t>(json.at(JS(oracle_document_id)));
return ripple::keylet::oracle(*account, documentId).key;

View File

@@ -24,9 +24,11 @@
#include "rpc/JS.hpp"
#include "rpc/common/Checkers.hpp"
#include "rpc/common/MetaProcessors.hpp"
#include "rpc/common/Modifiers.hpp"
#include "rpc/common/Specs.hpp"
#include "rpc/common/Types.hpp"
#include "rpc/common/Validators.hpp"
#include "util/AccountUtils.hpp"
#include <boost/json/conversion.hpp>
#include <boost/json/object.hpp>
@@ -135,9 +137,11 @@ public:
}
auto const id1 =
ripple::parseBase58<ripple::AccountID>(boost::json::value_to<std::string>(value.as_array()[0]));
util::parseBase58Wrapper<ripple::AccountID>(boost::json::value_to<std::string>(value.as_array()[0])
);
auto const id2 =
ripple::parseBase58<ripple::AccountID>(boost::json::value_to<std::string>(value.as_array()[1]));
util::parseBase58Wrapper<ripple::AccountID>(boost::json::value_to<std::string>(value.as_array()[1])
);
if (!id1 || !id2)
return Error{Status{ClioError::rpcMALFORMED_ADDRESS, "malformedAddresses"}};
@@ -296,8 +300,9 @@ public:
{JS(oracle_document_id),
meta::WithCustomError{validation::Required{}, Status(ClioError::rpcMALFORMED_REQUEST)},
meta::WithCustomError{
validation::Type<uint32_t>{}, Status(ClioError::rpcMALFORMED_ORACLE_DOCUMENT_ID)
}},
validation::Type<uint32_t, std::string>{}, Status(ClioError::rpcMALFORMED_ORACLE_DOCUMENT_ID)
},
meta::WithCustomError{modifiers::ToNumber{}, Status(ClioError::rpcMALFORMED_ORACLE_DOCUMENT_ID)}},
}}},
{JS(ledger), check::Deprecated{}},
};

View File

@@ -77,7 +77,7 @@ NFTHistoryHandler::process(NFTHistoryHandler::Input input, Context const& ctx) c
if (input.ledgerIndexMax || input.ledgerIndexMin)
return Error{Status{RippledError::rpcINVALID_PARAMS, "containsLedgerSpecifierAndRange"}};
auto const lgrInfoOrStatus = getLedgerInfoFromHashOrSeq(
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, ctx.yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);

View File

@@ -48,7 +48,7 @@ NFTInfoHandler::process(NFTInfoHandler::Input input, Context const& ctx) const
{
auto const tokenID = ripple::uint256{input.nftID.c_str()};
auto const range = sharedPtrBackend_->fetchLedgerRange();
auto const lgrInfoOrStatus = getLedgerInfoFromHashOrSeq(
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, ctx.yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);

View File

@@ -90,8 +90,9 @@ NFTOffersHandlerBase::iterateOfferDirectory(
) const
{
auto const range = sharedPtrBackend_->fetchLedgerRange();
auto const lgrInfoOrStatus =
getLedgerInfoFromHashOrSeq(*sharedPtrBackend_, yield, input.ledgerHash, input.ledgerIndex, range->maxSequence);
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);
if (auto const status = std::get_if<Status>(&lgrInfoOrStatus))
return Error{*status};

View File

@@ -48,13 +48,13 @@ NFTsByIssuerHandler::Result
NFTsByIssuerHandler::process(NFTsByIssuerHandler::Input input, Context const& ctx) const
{
auto const range = sharedPtrBackend_->fetchLedgerRange();
auto const lgrInfoOrStatus = getLedgerInfoFromHashOrSeq(
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, ctx.yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);
if (auto const status = std::get_if<Status>(&lgrInfoOrStatus))
return Error{*status};
auto const lgrInfo = std::get<LedgerInfo>(lgrInfoOrStatus);
auto const lgrInfo = std::get<LedgerHeader>(lgrInfoOrStatus);
auto const limit = input.limit.value_or(NFTsByIssuerHandler::LIMIT_DEFAULT);

View File

@@ -58,7 +58,7 @@ NoRippleCheckHandler::Result
NoRippleCheckHandler::process(NoRippleCheckHandler::Input input, Context const& ctx) const
{
auto const range = sharedPtrBackend_->fetchLedgerRange();
auto const lgrInfoOrStatus = getLedgerInfoFromHashOrSeq(
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, ctx.yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);

View File

@@ -222,7 +222,7 @@ public:
}
auto const serverInfoRippled =
balancer_->forwardToRippled({{"command", "server_info"}}, ctx.clientIp, ctx.yield);
balancer_->forwardToRippled({{"command", "server_info"}}, ctx.clientIp, ctx.isAdmin, ctx.yield);
if (serverInfoRippled && !serverInfoRippled->contains(JS(error))) {
if (serverInfoRippled->contains(JS(result)) &&

View File

@@ -43,7 +43,7 @@ TransactionEntryHandler::Result
TransactionEntryHandler::process(TransactionEntryHandler::Input input, Context const& ctx) const
{
auto const range = sharedPtrBackend_->fetchLedgerRange();
auto const lgrInfoOrStatus = getLedgerInfoFromHashOrSeq(
auto const lgrInfoOrStatus = getLedgerHeaderFromHashOrSeq(
*sharedPtrBackend_, ctx.yield, input.ledgerHash, input.ledgerIndex, range->maxSequence
);

67
src/util/AccountUtils.hpp Normal file
View File

@@ -0,0 +1,67 @@
//------------------------------------------------------------------------------
/*
This file is part of clio: https://github.com/XRPLF/clio
Copyright (c) 2024, the clio developers.
Permission to use, copy, modify, and distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
//==============================================================================
#pragma once
#include <ripple/protocol/tokens.h>
#include <cctype>
#include <optional>
#include <string>
namespace util {
/**
* @brief A wrapper of parseBase58 function. It adds the check if all characters in the input string are alphanumeric.
* If not, it returns an empty optional, instead of calling the parseBase58 function.
*
* @tparam T The type of the value to parse to.
* @param str The string to parse.
* @return An optional with the parsed value, or an empty optional if the parse fails.
*/
template <class T>
[[nodiscard]] std::optional<T>
parseBase58Wrapper(std::string const& str)
{
if (!std::all_of(std::begin(str), std::end(str), [](unsigned char c) { return std::isalnum(c); }))
return std::nullopt;
return ripple::parseBase58<T>(str);
}
/**
* @brief A wrapper of parseBase58 function. It add the check if all characters in the input string are alphanumeric. If
* not, it returns an empty optional, instead of calling the parseBase58 function.
*
* @tparam T The type of the value to parse to.
* @param type The type of the token to parse.
* @param str The string to parse.
* @return An optional with the parsed value, or an empty optional if the parse fails.
*/
template <class T>
[[nodiscard]] std::optional<T>
parseBase58Wrapper(ripple::TokenType type, std::string const& str)
{
if (!std::all_of(std::begin(str), std::end(str), [](unsigned char c) { return std::isalnum(c); }))
return std::nullopt;
return ripple::parseBase58<T>(type, str);
}
} // namespace util

View File

@@ -20,12 +20,10 @@
#include "util/SignalsHandler.hpp"
#include "util/Assert.hpp"
#include "util/Constants.hpp"
#include "util/config/Config.hpp"
#include "util/log/Logger.hpp"
#include <chrono>
#include <cmath>
#include <csignal>
#include <cstddef>
#include <functional>
@@ -101,10 +99,8 @@ SignalsHandler::SignalsHandler(Config const& config, std::function<void()> force
{
impl::SignalsHandlerStatic::registerHandler(*this);
auto const gracefulPeriod =
std::round(config.valueOr("graceful_period", 10.f) * static_cast<float>(util::MILLISECONDS_PER_SECOND));
ASSERT(gracefulPeriod >= 0.f, "Graceful period must be non-negative");
gracefulPeriod_ = std::chrono::milliseconds{static_cast<size_t>(gracefulPeriod)};
gracefulPeriod_ = Config::toMilliseconds(config.valueOr("graceful_period", 10.f));
ASSERT(gracefulPeriod_.count() >= 0, "Graceful period must be non-negative");
setHandler(impl::SignalsHandlerStatic::handleSignal);
}

View File

@@ -19,6 +19,8 @@
#include "util/config/Config.hpp"
#include "util/Assert.hpp"
#include "util/Constants.hpp"
#include "util/config/impl/Helpers.hpp"
#include "util/log/Logger.hpp"
@@ -28,6 +30,8 @@
#include <boost/json/value.hpp>
#include <algorithm>
#include <chrono>
#include <cmath>
#include <exception>
#include <filesystem>
#include <fstream>
@@ -178,6 +182,13 @@ Config::array() const
return out;
}
std::chrono::milliseconds
Config::toMilliseconds(float value)
{
ASSERT(value >= 0.0f, "Floating point value of seconds must be non-negative, got: {}", value);
return std::chrono::milliseconds{std::lroundf(value * static_cast<float>(util::MILLISECONDS_PER_SECOND))};
}
Config
ConfigReader::open(std::filesystem::path path)
{

View File

@@ -26,6 +26,7 @@
#include <boost/json/object.hpp>
#include <boost/json/value.hpp>
#include <chrono>
#include <cstdint>
#include <exception>
#include <filesystem>
@@ -362,6 +363,15 @@ public:
[[nodiscard]] ArrayType
array() const;
/**
* @brief Method to convert a float seconds value to milliseconds.
*
* @param value The value to convert
* @return The value in milliseconds
*/
static std::chrono::milliseconds
toMilliseconds(float value);
private:
template <typename Return>
[[nodiscard]] Return

View File

@@ -53,20 +53,29 @@ public:
* @brief Read a message from the WebSocket
*
* @param yield yield context
* @param timeout timeout for the operation
* @return Message or error
*/
virtual std::expected<std::string, RequestError>
read(boost::asio::yield_context yield) = 0;
read(
boost::asio::yield_context yield,
std::optional<std::chrono::steady_clock::duration> timeout = std::nullopt
) = 0;
/**
* @brief Write a message to the WebSocket
*
* @param message message to write
* @param yield yield context
* @param timeout timeout for the operation
* @return Error if any
*/
virtual std::optional<RequestError>
write(std::string const& message, boost::asio::yield_context yield) = 0;
write(
std::string const& message,
boost::asio::yield_context yield,
std::optional<std::chrono::steady_clock::duration> timeout = std::nullopt
) = 0;
/**
* @brief Close the WebSocket

View File

@@ -22,8 +22,13 @@
#include "util/requests/Types.hpp"
#include "util/requests/WsConnection.hpp"
#include <boost/asio/associated_executor.hpp>
#include <boost/asio/bind_cancellation_slot.hpp>
#include <boost/asio/buffer.hpp>
#include <boost/asio/cancellation_signal.hpp>
#include <boost/asio/cancellation_type.hpp>
#include <boost/asio/spawn.hpp>
#include <boost/asio/steady_timer.hpp>
#include <boost/beast/core/buffers_to_string.hpp>
#include <boost/beast/core/error.hpp>
#include <boost/beast/core/flat_buffer.hpp>
@@ -32,9 +37,12 @@
#include <boost/beast/websocket/rfc6455.hpp>
#include <boost/beast/websocket/stream.hpp>
#include <boost/beast/websocket/stream_base.hpp>
#include <boost/system/errc.hpp>
#include <atomic>
#include <chrono>
#include <expected>
#include <memory>
#include <optional>
#include <string>
#include <utility>
@@ -51,27 +59,46 @@ public:
}
std::expected<std::string, RequestError>
read(boost::asio::yield_context yield) override
read(boost::asio::yield_context yield, std::optional<std::chrono::steady_clock::duration> timeout = std::nullopt)
override
{
boost::beast::error_code errorCode;
boost::beast::flat_buffer buffer;
ws_.async_read(buffer, yield[errorCode]);
auto operation = [&](auto&& token) { ws_.async_read(buffer, token); };
if (timeout) {
withTimeout(operation, yield[errorCode], *timeout);
} else {
operation(yield[errorCode]);
}
if (errorCode)
if (errorCode) {
errorCode = mapError(errorCode);
return std::unexpected{RequestError{"Read error", errorCode}};
}
return boost::beast::buffers_to_string(std::move(buffer).data());
}
std::optional<RequestError>
write(std::string const& message, boost::asio::yield_context yield) override
write(
std::string const& message,
boost::asio::yield_context yield,
std::optional<std::chrono::steady_clock::duration> timeout = std::nullopt
) override
{
boost::beast::error_code errorCode;
ws_.async_write(boost::asio::buffer(message), yield[errorCode]);
auto operation = [&](auto&& token) { ws_.async_write(boost::asio::buffer(message), token); };
if (timeout) {
withTimeout(operation, yield[errorCode], *timeout);
} else {
operation(yield[errorCode]);
}
if (errorCode)
if (errorCode) {
errorCode = mapError(errorCode);
return RequestError{"Write error", errorCode};
}
return std::nullopt;
}
@@ -92,6 +119,36 @@ public:
return RequestError{"Close error", errorCode};
return std::nullopt;
}
private:
template <typename Operation>
static void
withTimeout(Operation&& operation, boost::asio::yield_context yield, std::chrono::steady_clock::duration timeout)
{
auto isCompleted = std::make_shared<bool>(false);
boost::asio::cancellation_signal cancellationSignal;
auto cyield = boost::asio::bind_cancellation_slot(cancellationSignal.slot(), yield);
boost::asio::steady_timer timer{boost::asio::get_associated_executor(cyield), timeout};
// The timer below can be called with no error code even if the operation is completed before the timeout, so we
// need an additional flag here
timer.async_wait([&cancellationSignal, isCompleted](boost::system::error_code errorCode) {
if (!errorCode and not*isCompleted)
cancellationSignal.emit(boost::asio::cancellation_type::terminal);
});
operation(cyield);
*isCompleted = true;
}
static boost::system::error_code
mapError(boost::system::error_code const ec)
{
if (ec == boost::system::errc::operation_canceled) {
return boost::system::errc::make_error_code(boost::system::errc::timed_out);
}
return ec;
}
};
using PlainWsConnection = WsConnectionImpl<boost::beast::websocket::stream<boost::beast::tcp_stream>>;

View File

@@ -69,10 +69,8 @@ namespace web {
* @tparam HandlerType The executor to handle the requests
*/
template <
template <typename>
class PlainSessionType,
template <typename>
class SslSessionType,
template <typename> class PlainSessionType,
template <typename> class SslSessionType,
SomeServerHandler HandlerType>
class Detector : public std::enable_shared_from_this<Detector<PlainSessionType, SslSessionType, HandlerType>> {
using std::enable_shared_from_this<Detector<PlainSessionType, SslSessionType, HandlerType>>::shared_from_this;
@@ -191,10 +189,8 @@ public:
* @tparam HandlerType The handler to process the request and return response.
*/
template <
template <typename>
class PlainSessionType,
template <typename>
class SslSessionType,
template <typename> class PlainSessionType,
template <typename> class SslSessionType,
SomeServerHandler HandlerType>
class Server : public std::enable_shared_from_this<Server<PlainSessionType, SslSessionType, HandlerType>> {
using std::enable_shared_from_this<Server<PlainSessionType, SslSessionType, HandlerType>>::shared_from_this;

View File

@@ -23,6 +23,9 @@
#include "rpc/common/Types.hpp"
#include "util/Taggable.hpp"
#include "util/log/Logger.hpp"
#include "util/prometheus/Gauge.hpp"
#include "util/prometheus/Label.hpp"
#include "util/prometheus/Prometheus.hpp"
#include "web/DOSGuard.hpp"
#include "web/interface/Concepts.hpp"
#include "web/interface/ConnectionBase.hpp"
@@ -71,6 +74,8 @@ template <template <typename> typename Derived, SomeServerHandler HandlerType>
class WsBase : public ConnectionBase, public std::enable_shared_from_this<WsBase<Derived, HandlerType>> {
using std::enable_shared_from_this<WsBase<Derived, HandlerType>>::shared_from_this;
std::reference_wrapper<util::prometheus::GaugeInt> messagesLength_;
boost::beast::flat_buffer buffer_;
std::reference_wrapper<web::DOSGuard> dosGuard_;
bool sending_ = false;
@@ -103,15 +108,26 @@ public:
std::shared_ptr<HandlerType> const& handler,
boost::beast::flat_buffer&& buffer
)
: ConnectionBase(tagFactory, ip), buffer_(std::move(buffer)), dosGuard_(dosGuard), handler_(handler)
: ConnectionBase(tagFactory, ip)
, messagesLength_(PrometheusService::gaugeInt(
"ws_messages_length",
util::prometheus::Labels(),
"The total length of messages in the queue"
))
, buffer_(std::move(buffer))
, dosGuard_(dosGuard)
, handler_(handler)
{
upgraded = true; // NOLINT (cppcoreguidelines-pro-type-member-init)
LOG(perfLog_.debug()) << tag() << "session created";
}
~WsBase() override
{
LOG(perfLog_.debug()) << tag() << "session closed";
if (!messages_.empty())
messagesLength_.get() -= messages_.size();
dosGuard_.get().decrement(clientIp);
}
@@ -135,6 +151,7 @@ public:
onWrite(boost::system::error_code ec, std::size_t)
{
messages_.pop();
--messagesLength_.get();
sending_ = false;
if (ec) {
wsFail(ec, "Failed to write");
@@ -165,6 +182,7 @@ public:
derived().ws().get_executor(),
[this, self = derived().shared_from_this(), msg = std::move(msg)]() {
messages_.push(msg);
++messagesLength_.get();
maybeSendNext();
}
);

View File

@@ -2,6 +2,7 @@ add_library(clio_testing_common)
target_sources(
clio_testing_common PRIVATE util/StringUtils.cpp util/TestHttpServer.cpp util/TestWsServer.cpp util/TestObject.cpp
util/AssignRandomPort.cpp
)
include(deps/gtest)

View File

@@ -0,0 +1,45 @@
//------------------------------------------------------------------------------
/*
This file is part of clio: https://github.com/XRPLF/clio
Copyright (c) 2024, the clio developers.
Permission to use, copy, modify, and distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
//==============================================================================
#include "util/AssignRandomPort.hpp"
#include <boost/asio/io_context.hpp>
#include <boost/asio/ip/tcp.hpp>
#include <cstdint>
using tcp = boost::asio::ip::tcp;
namespace tests::util {
uint32_t
generateFreePort()
{
boost::asio::io_context io_context;
tcp::acceptor acceptor(io_context);
tcp::endpoint const endpoint(tcp::v4(), 0);
acceptor.open(endpoint.protocol());
acceptor.set_option(tcp::acceptor::reuse_address(true));
acceptor.bind(endpoint);
return acceptor.local_endpoint().port();
}
} // namespace tests::util

View File

@@ -0,0 +1,29 @@
//------------------------------------------------------------------------------
/*
This file is part of clio: https://github.com/XRPLF/clio
Copyright (c) 2024, the clio developers.
Permission to use, copy, modify, and distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
//==============================================================================
#pragma once
#include <cstdint>
namespace tests::util {
uint32_t
generateFreePort();
} // namespace tests::util

View File

@@ -44,14 +44,14 @@ struct MockBackend : public BackendInterface {
}
MOCK_METHOD(
std::optional<ripple::LedgerInfo>,
std::optional<ripple::LedgerHeader>,
fetchLedgerBySequence,
(std::uint32_t const, boost::asio::yield_context),
(const, override)
);
MOCK_METHOD(
std::optional<ripple::LedgerInfo>,
std::optional<ripple::LedgerHeader>,
fetchLedgerByHash,
(ripple::uint256 const&, boost::asio::yield_context),
(const, override)
@@ -170,7 +170,7 @@ struct MockBackend : public BackendInterface {
MOCK_METHOD(std::optional<LedgerRange>, hardFetchLedgerRange, (boost::asio::yield_context), (const, override));
MOCK_METHOD(void, writeLedger, (ripple::LedgerInfo const&, std::string&&), (override));
MOCK_METHOD(void, writeLedger, (ripple::LedgerHeader const&, std::string&&), (override));
MOCK_METHOD(void, writeLedgerObject, (std::string&&, std::uint32_t const, std::string&&), (override));

View File

@@ -35,8 +35,8 @@ struct MockLedgerLoader {
MOCK_METHOD(
FormattedTransactionsData,
insertTransactions,
(ripple::LedgerInfo const&, GetLedgerResponseType& data),
(ripple::LedgerHeader const&, GetLedgerResponseType& data),
()
);
MOCK_METHOD(std::optional<ripple::LedgerInfo>, loadInitialLedger, (uint32_t sequence), ());
MOCK_METHOD(std::optional<ripple::LedgerHeader>, loadInitialLedger, (uint32_t sequence), ());
};

View File

@@ -28,7 +28,7 @@
struct MockLedgerPublisher {
MOCK_METHOD(bool, publish, (uint32_t, std::optional<uint32_t>), ());
MOCK_METHOD(void, publish, (ripple::LedgerInfo const&), ());
MOCK_METHOD(void, publish, (ripple::LedgerHeader const&), ());
MOCK_METHOD(std::uint32_t, lastPublishAgeSeconds, (), (const));
MOCK_METHOD(std::chrono::time_point<std::chrono::system_clock>, getLastPublish, (), (const));
MOCK_METHOD(std::uint32_t, lastCloseAgeSeconds, (), (const));

View File

@@ -40,7 +40,7 @@ struct MockLoadBalancer {
MOCK_METHOD(
std::optional<boost::json::object>,
forwardToRippled,
(boost::json::object const&, std::optional<std::string> const&, boost::asio::yield_context),
(boost::json::object const&, std::optional<std::string> const&, bool, boost::asio::yield_context),
(const)
);
};

View File

@@ -20,6 +20,7 @@
#include "data/BackendInterface.hpp"
#include "etl/ETLHelpers.hpp"
#include "etl/NetworkValidatedLedgersInterface.hpp"
#include "etl/Source.hpp"
#include "feed/SubscriptionManagerInterface.hpp"
#include "util/config/Config.hpp"
@@ -34,12 +35,13 @@
#include <org/xrpl/rpc/v1/get_ledger.pb.h>
#include <algorithm>
#include <chrono>
#include <cstddef>
#include <cstdint>
#include <memory>
#include <optional>
#include <string>
#include <unordered_map>
#include <string_view>
#include <utility>
#include <vector>
@@ -60,7 +62,7 @@ struct MockSource : etl::SourceBase {
MOCK_METHOD(
std::optional<boost::json::object>,
forwardToRippled,
(boost::json::object const&, std::optional<std::string> const&, boost::asio::yield_context),
(boost::json::object const&, std::optional<std::string> const&, std::string_view, boost::asio::yield_context),
(const, override)
);
};
@@ -129,10 +131,11 @@ public:
forwardToRippled(
boost::json::object const& request,
std::optional<std::string> const& forwardToRippledClientIp,
std::string_view xUserValue,
boost::asio::yield_context yield
) const override
{
return mock_->forwardToRippled(request, forwardToRippledClientIp, yield);
return mock_->forwardToRippled(request, forwardToRippledClientIp, xUserValue, yield);
}
};
@@ -155,29 +158,58 @@ class MockSourceFactoryImpl {
public:
MockSourceFactoryImpl(size_t numSources)
{
setSourcesNumber(numSources);
ON_CALL(*this, makeSource)
.WillByDefault([this](
util::Config const&,
boost::asio::io_context&,
std::shared_ptr<BackendInterface>,
std::shared_ptr<feed::SubscriptionManagerInterface>,
std::shared_ptr<etl::NetworkValidatedLedgersInterface>,
std::chrono::steady_clock::duration,
etl::SourceBase::OnConnectHook onConnect,
etl::SourceBase::OnDisconnectHook onDisconnect,
etl::SourceBase::OnLedgerClosedHook onLedgerClosed
) {
auto it = std::ranges::find_if(mockData_, [](auto const& d) { return not d.callbacks.has_value(); });
[&]() { ASSERT_NE(it, mockData_.end()) << "Make source called more than expected"; }();
it->callbacks =
MockSourceCallbacks{std::move(onDisconnect), std::move(onConnect), std::move(onLedgerClosed)};
return std::make_unique<MockSourceWrapper<MockType>>(it->source);
});
}
void
setSourcesNumber(size_t numSources)
{
mockData_.clear();
mockData_.reserve(numSources);
std::ranges::generate_n(std::back_inserter(mockData_), numSources, [] { return MockSourceData<MockType>{}; });
}
template <typename... Args>
etl::SourcePtr
makeSourceMock(
util::Config const&,
boost::asio::io_context&,
std::shared_ptr<BackendInterface>,
std::shared_ptr<feed::SubscriptionManagerInterface>,
std::shared_ptr<etl::NetworkValidatedLedgersInterface>,
etl::SourceBase::OnConnectHook onConnect,
etl::SourceBase::OnDisconnectHook onDisconnect,
etl::SourceBase::OnLedgerClosedHook onLedgerClosed
)
operator()(Args&&... args)
{
auto it = std::ranges::find_if(mockData_, [](auto const& d) { return not d.callbacks.has_value(); });
[&]() { ASSERT_NE(it, mockData_.end()) << "Make source called more than expected"; }();
it->callbacks = MockSourceCallbacks{std::move(onDisconnect), std::move(onConnect), std::move(onLedgerClosed)};
return std::make_unique<MockSourceWrapper<MockType>>(it->source);
return makeSource(std::forward<Args>(args)...);
}
MOCK_METHOD(
etl::SourcePtr,
makeSource,
(util::Config const&,
boost::asio::io_context&,
std::shared_ptr<BackendInterface>,
std::shared_ptr<feed::SubscriptionManagerInterface>,
std::shared_ptr<etl::NetworkValidatedLedgersInterface>,
std::chrono::steady_clock::duration,
etl::SourceBase::OnConnectHook,
etl::SourceBase::OnDisconnectHook,
etl::SourceBase::OnLedgerClosedHook)
);
MockType<MockSource>&
sourceAt(size_t index)
{
@@ -193,5 +225,5 @@ public:
}
};
using MockSourceFactory = MockSourceFactoryImpl<>;
using StrictMockSourceFactory = MockSourceFactoryImpl<testing::StrictMock>;
using MockSourceFactory = testing::NiceMock<MockSourceFactoryImpl<>>;
using StrictMockSourceFactory = testing::StrictMock<MockSourceFactoryImpl<testing::StrictMock>>;

View File

@@ -55,7 +55,7 @@ struct MockSubscriptionManager : feed::SubscriptionManagerInterface {
MOCK_METHOD(
void,
pubBookChanges,
(ripple::LedgerInfo const&, std::vector<data::TransactionAndMetadata> const&),
(ripple::LedgerHeader const&, std::vector<data::TransactionAndMetadata> const&),
(const, override)
);
@@ -65,7 +65,7 @@ struct MockSubscriptionManager : feed::SubscriptionManagerInterface {
MOCK_METHOD(void, unsubTransactions, (feed::SubscriberSharedPtr const&), (override));
MOCK_METHOD(void, pubTransaction, (data::TransactionAndMetadata const&, ripple::LedgerInfo const&), (override));
MOCK_METHOD(void, pubTransaction, (data::TransactionAndMetadata const&, ripple::LedgerHeader const&), (override));
MOCK_METHOD(void, subAccount, (ripple::AccountID const&, feed::SubscriberSharedPtr const&), (override));

View File

@@ -82,7 +82,7 @@ struct WithMockXrpLedgerAPIService : virtual ::testing::Test {
WithMockXrpLedgerAPIService(std::string serverAddress)
{
grpc::ServerBuilder builder;
builder.AddListeningPort(serverAddress, grpc::InsecureServerCredentials());
builder.AddListeningPort(serverAddress, grpc::InsecureServerCredentials(), &port_);
builder.RegisterService(&mockXrpLedgerAPIService);
server_ = builder.BuildAndStart();
serverThread_ = std::thread([this] { server_->Wait(); });
@@ -94,11 +94,17 @@ struct WithMockXrpLedgerAPIService : virtual ::testing::Test {
serverThread_.join();
}
int
getXRPLMockPort() const
{
return port_;
}
MockXrpLedgerAPIService mockXrpLedgerAPIService;
private:
std::unique_ptr<grpc::Server> server_;
std::thread serverThread_;
int port_{};
};
} // namespace tests::util

View File

@@ -0,0 +1,28 @@
//------------------------------------------------------------------------------
/*
This file is part of clio: https://github.com/XRPLF/clio
Copyright (c) 2024, the clio developers.
Permission to use, copy, modify, and distribute this software for any
purpose with or without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
*/
//==============================================================================
#pragma once
#include <string>
namespace tests::util {
static auto const NameGenerator = [](auto const& info) { return info.param.testName; };
} // namespace tests::util

View File

@@ -46,9 +46,9 @@ binaryStringToUint256(std::string const& bin)
}
std::string
ledgerInfoToBinaryString(ripple::LedgerInfo const& info)
ledgerHeaderToBinaryString(ripple::LedgerHeader const& info)
{
auto const blob = rpc::ledgerInfoToBlob(info, true);
auto const blob = rpc::ledgerHeaderToBlob(info, true);
std::string strBlob;
for (auto c : blob)
strBlob += c;

View File

@@ -32,4 +32,4 @@ ripple::uint256
binaryStringToUint256(std::string const& bin);
std::string
ledgerInfoToBinaryString(ripple::LedgerInfo const& info);
ledgerHeaderToBinaryString(ripple::LedgerHeader const& info);

View File

@@ -105,9 +105,9 @@ doSession(
} // namespace
TestHttpServer::TestHttpServer(boost::asio::io_context& context, std::string host, int const port) : acceptor_(context)
TestHttpServer::TestHttpServer(boost::asio::io_context& context, std::string host) : acceptor_(context)
{
boost::asio::ip::tcp::endpoint const endpoint(boost::asio::ip::make_address(host), port);
boost::asio::ip::tcp::endpoint const endpoint(boost::asio::ip::make_address(host), 0);
acceptor_.open(endpoint.protocol());
acceptor_.set_option(asio::socket_base::reuse_address(true));
acceptor_.bind(endpoint);
@@ -134,3 +134,9 @@ TestHttpServer::handleRequest(TestHttpServer::RequestHandler handler, bool const
boost::asio::detached
);
}
std::string
TestHttpServer::port() const
{
return std::to_string(acceptor_.local_endpoint().port());
}

View File

@@ -41,9 +41,8 @@ public:
*
* @param context boost::asio::io_context to use for networking
* @param host host to bind to
* @param port port to bind to
*/
TestHttpServer(boost::asio::io_context& context, std::string host, int port);
TestHttpServer(boost::asio::io_context& context, std::string host);
/**
* @brief Start the server
@@ -56,6 +55,14 @@ public:
void
handleRequest(RequestHandler handler, bool allowToFail = false);
/**
* @brief Return the port HTTP server is connected to
*
* @return string port number
*/
std::string
port() const;
private:
boost::asio::ip::tcp::acceptor acceptor_;
};

View File

@@ -21,6 +21,7 @@
#include "data/DBHelpers.hpp"
#include "data/Types.hpp"
#include "util/AccountUtils.hpp"
#include "util/Assert.hpp"
#include <ripple/basics/Blob.h>
@@ -60,7 +61,7 @@ constexpr static auto INDEX1 = "1B8590C01B0006EDFA9ED60296DD052DC5E90F99659B2501
ripple::AccountID
GetAccountIDWithString(std::string_view id)
{
return ripple::parseBase58<ripple::AccountID>(std::string(id)).value();
return util::parseBase58Wrapper<ripple::AccountID>(std::string(id)).value();
}
ripple::uint256
@@ -75,22 +76,22 @@ GetAccountKey(ripple::AccountID const& acc)
return ripple::keylet::account(acc).key;
}
ripple::LedgerInfo
CreateLedgerInfo(std::string_view ledgerHash, ripple::LedgerIndex seq, std::optional<uint32_t> age)
ripple::LedgerHeader
CreateLedgerHeader(std::string_view ledgerHash, ripple::LedgerIndex seq, std::optional<uint32_t> age)
{
using namespace std::chrono;
auto ledgerinfo = ripple::LedgerInfo();
ledgerinfo.hash = ripple::uint256{ledgerHash};
ledgerinfo.seq = seq;
auto ledgerHeader = ripple::LedgerHeader();
ledgerHeader.hash = ripple::uint256{ledgerHash};
ledgerHeader.seq = seq;
if (age) {
auto const now = duration_cast<seconds>(system_clock::now().time_since_epoch());
auto const closeTime = (now - seconds{age.value()}).count() - rippleEpochStart;
ledgerinfo.closeTime = ripple::NetClock::time_point{seconds{closeTime}};
ledgerHeader.closeTime = ripple::NetClock::time_point{seconds{closeTime}};
}
return ledgerinfo;
return ledgerHeader;
}
ripple::STObject
@@ -154,11 +155,11 @@ CreatePaymentTransactionObject(
{
ripple::STObject obj(ripple::sfTransaction);
obj.setFieldU16(ripple::sfTransactionType, ripple::ttPAYMENT);
auto account = ripple::parseBase58<ripple::AccountID>(std::string(accountId1));
auto account = util::parseBase58Wrapper<ripple::AccountID>(std::string(accountId1));
obj.setAccountID(ripple::sfAccount, account.value());
obj.setFieldAmount(ripple::sfAmount, ripple::STAmount(amount, false));
obj.setFieldAmount(ripple::sfFee, ripple::STAmount(fee, false));
auto account2 = ripple::parseBase58<ripple::AccountID>(std::string(accountId2));
auto account2 = util::parseBase58Wrapper<ripple::AccountID>(std::string(accountId2));
obj.setAccountID(ripple::sfDestination, account2.value());
obj.setFieldU32(ripple::sfSequence, seq);
char const* key = "test";
@@ -258,14 +259,14 @@ CreateCreateOfferTransactionObject(
{
ripple::STObject obj(ripple::sfTransaction);
obj.setFieldU16(ripple::sfTransactionType, ripple::ttOFFER_CREATE);
auto account = ripple::parseBase58<ripple::AccountID>(std::string(accountId));
auto account = util::parseBase58Wrapper<ripple::AccountID>(std::string(accountId));
obj.setAccountID(ripple::sfAccount, account.value());
auto amount = ripple::STAmount(fee, false);
obj.setFieldAmount(ripple::sfFee, amount);
obj.setFieldU32(ripple::sfSequence, seq);
// add amount
ripple::Issue const issue1(
ripple::Currency{currency}, ripple::parseBase58<ripple::AccountID>(std::string(issuer)).value()
ripple::Currency{currency}, util::parseBase58Wrapper<ripple::AccountID>(std::string(issuer)).value()
);
if (reverse) {
obj.setFieldAmount(ripple::sfTakerPays, ripple::STAmount(issue1, takerGets));
@@ -288,11 +289,11 @@ GetIssue(std::string_view currency, std::string_view issuerId)
if (currency.size() == 3) {
return ripple::Issue(
ripple::to_currency(std::string(currency)),
ripple::parseBase58<ripple::AccountID>(std::string(issuerId)).value()
util::parseBase58Wrapper<ripple::AccountID>(std::string(issuerId)).value()
);
}
return ripple::Issue(
ripple::Currency{currency}, ripple::parseBase58<ripple::AccountID>(std::string(issuerId)).value()
ripple::Currency{currency}, util::parseBase58Wrapper<ripple::AccountID>(std::string(issuerId)).value()
);
}
@@ -636,7 +637,7 @@ CreateMintNFTTxWithMetadata(
// tx
ripple::STObject tx(ripple::sfTransaction);
tx.setFieldU16(ripple::sfTransactionType, ripple::ttNFTOKEN_MINT);
auto account = ripple::parseBase58<ripple::AccountID>(std::string(accountId));
auto account = util::parseBase58Wrapper<ripple::AccountID>(std::string(accountId));
tx.setAccountID(ripple::sfAccount, account.value());
auto amount = ripple::STAmount(fee, false);
tx.setFieldAmount(ripple::sfFee, amount);
@@ -693,7 +694,7 @@ CreateAcceptNFTOfferTxWithMetadata(std::string_view accountId, uint32_t seq, uin
// tx
ripple::STObject tx(ripple::sfTransaction);
tx.setFieldU16(ripple::sfTransactionType, ripple::ttNFTOKEN_ACCEPT_OFFER);
auto account = ripple::parseBase58<ripple::AccountID>(std::string(accountId));
auto account = util::parseBase58Wrapper<ripple::AccountID>(std::string(accountId));
tx.setAccountID(ripple::sfAccount, account.value());
auto amount = ripple::STAmount(fee, false);
tx.setFieldAmount(ripple::sfFee, amount);
@@ -737,7 +738,7 @@ CreateCancelNFTOffersTxWithMetadata(
// tx
ripple::STObject tx(ripple::sfTransaction);
tx.setFieldU16(ripple::sfTransactionType, ripple::ttNFTOKEN_CANCEL_OFFER);
auto account = ripple::parseBase58<ripple::AccountID>(std::string(accountId));
auto account = util::parseBase58Wrapper<ripple::AccountID>(std::string(accountId));
tx.setAccountID(ripple::sfAccount, account.value());
auto amount = ripple::STAmount(fee, false);
tx.setFieldAmount(ripple::sfFee, amount);
@@ -791,7 +792,7 @@ CreateCreateNFTOfferTxWithMetadata(
// tx
ripple::STObject tx(ripple::sfTransaction);
tx.setFieldU16(ripple::sfTransactionType, ripple::ttNFTOKEN_CREATE_OFFER);
auto account = ripple::parseBase58<ripple::AccountID>(std::string(accountId));
auto account = util::parseBase58Wrapper<ripple::AccountID>(std::string(accountId));
tx.setAccountID(ripple::sfAccount, account.value());
auto amount = ripple::STAmount(fee, false);
tx.setFieldAmount(ripple::sfFee, amount);
@@ -839,7 +840,7 @@ CreateOracleSetTxWithMetadata(
// tx
ripple::STObject tx(ripple::sfTransaction);
tx.setFieldU16(ripple::sfTransactionType, ripple::ttORACLE_SET);
auto account = ripple::parseBase58<ripple::AccountID>(std::string(accountId));
auto account = util::parseBase58Wrapper<ripple::AccountID>(std::string(accountId));
tx.setAccountID(ripple::sfAccount, account.value());
auto amount = ripple::STAmount(fee, false);
tx.setFieldAmount(ripple::sfFee, amount);
@@ -909,7 +910,7 @@ CreateAMMObject(
amm.setFieldIssue(ripple::sfAsset2, ripple::STIssue{ripple::sfAsset2, GetIssue(asset2Currency, asset2Issuer)});
ripple::Issue const issue1(
ripple::Currency{lpTokenBalanceIssueCurrency},
ripple::parseBase58<ripple::AccountID>(std::string(accountId)).value()
util::parseBase58Wrapper<ripple::AccountID>(std::string(accountId)).value()
);
amm.setFieldAmount(ripple::sfLPTokenBalance, ripple::STAmount(issue1, lpTokenBalanceIssueAmount));
amm.setFieldU32(ripple::sfFlags, 0);

View File

@@ -60,10 +60,10 @@ GetAccountKey(std::string_view id);
GetAccountKey(ripple::AccountID const& acc);
/*
* Create a simple ledgerInfo object with only hash and seq
* Create a simple ledgerHeader object with only hash and seq
*/
[[nodiscard]] ripple::LedgerInfo
CreateLedgerInfo(std::string_view ledgerHash, ripple::LedgerIndex seq, std::optional<uint32_t> age = std::nullopt);
[[nodiscard]] ripple::LedgerHeader
CreateLedgerHeader(std::string_view ledgerHash, ripple::LedgerIndex seq, std::optional<uint32_t> age = std::nullopt);
/*
* Create a Legacy (pre XRPFees amendment) FeeSetting ledger object

Some files were not shown because too many files have changed in this diff Show More