Bumps
[actions/download-artifact](https://github.com/actions/download-artifact)
from 4 to 5.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/actions/download-artifact/releases">actions/download-artifact's
releases</a>.</em></p>
<blockquote>
<h2>v5.0.0</h2>
<h2>What's Changed</h2>
<ul>
<li>Update README.md by <a
href="https://github.com/nebuk89"><code>@nebuk89</code></a> in <a
href="https://redirect.github.com/actions/download-artifact/pull/407">actions/download-artifact#407</a></li>
<li>BREAKING fix: inconsistent path behavior for single artifact
downloads by ID by <a
href="https://github.com/GrantBirki"><code>@GrantBirki</code></a> in <a
href="https://redirect.github.com/actions/download-artifact/pull/416">actions/download-artifact#416</a></li>
</ul>
<h2>v5.0.0</h2>
<h3>🚨 Breaking Change</h3>
<p>This release fixes an inconsistency in path behavior for single
artifact downloads by ID. <strong>If you're downloading single artifacts
by ID, the output path may change.</strong></p>
<h4>What Changed</h4>
<p>Previously, <strong>single artifact downloads</strong> behaved
differently depending on how you specified the artifact:</p>
<ul>
<li><strong>By name</strong>: <code>name: my-artifact</code> → extracted
to <code>path/</code> (direct)</li>
<li><strong>By ID</strong>: <code>artifact-ids: 12345</code> → extracted
to <code>path/my-artifact/</code> (nested)</li>
</ul>
<p>Now both methods are consistent:</p>
<ul>
<li><strong>By name</strong>: <code>name: my-artifact</code> → extracted
to <code>path/</code> (unchanged)</li>
<li><strong>By ID</strong>: <code>artifact-ids: 12345</code> → extracted
to <code>path/</code> (fixed - now direct)</li>
</ul>
<h4>Migration Guide</h4>
<h5>✅ No Action Needed If:</h5>
<ul>
<li>You download artifacts by <strong>name</strong></li>
<li>You download <strong>multiple</strong> artifacts by ID</li>
<li>You already use <code>merge-multiple: true</code> as a
workaround</li>
</ul>
<h5>⚠️ Action Required If:</h5>
<p>You download <strong>single artifacts by ID</strong> and your
workflows expect the nested directory structure.</p>
<p><strong>Before v5 (nested structure):</strong></p>
<pre lang="yaml"><code>- uses: actions/download-artifact@v4
with:
artifact-ids: 12345
path: dist
# Files were in: dist/my-artifact/
</code></pre>
<blockquote>
<p>Where <code>my-artifact</code> is the name of the artifact you
previously uploaded</p>
</blockquote>
<p><strong>To maintain old behavior (if needed):</strong></p>
<pre lang="yaml"><code></tr></table>
</code></pre>
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="634f93cb29"><code>634f93c</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/download-artifact/issues/416">#416</a>
from actions/single-artifact-id-download-path</li>
<li><a
href="b19ff43027"><code>b19ff43</code></a>
refactor: resolve download path correctly in artifact download tests
(mainly ...</li>
<li><a
href="e262cbee4a"><code>e262cbe</code></a>
bundle dist</li>
<li><a
href="bff23f9308"><code>bff23f9</code></a>
update docs</li>
<li><a
href="fff8c148a8"><code>fff8c14</code></a>
fix download path logic when downloading a single artifact by id</li>
<li><a
href="448e3f862a"><code>448e3f8</code></a>
Merge pull request <a
href="https://redirect.github.com/actions/download-artifact/issues/407">#407</a>
from actions/nebuk89-patch-1</li>
<li><a
href="47225c44b3"><code>47225c4</code></a>
Update README.md</li>
<li>See full diff in <a
href="https://github.com/actions/download-artifact/compare/v4...v5">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Ayaz Salikhov <mathbunnyru@users.noreply.github.com>
Updates the GitHub Actions workflow concurrency configuration to allow
parallel execution of workflows on the `develop` branch while
maintaining cancellation behavior for other branches.
## Problem
Previously, all workflow runs were subject to cancellation when new
commits were pushed to the same branch (`cancel-in-progress: true`).
This caused issues on the `develop` branch where important CI runs could
be prematurely cancelled, potentially missing critical feedback on the
main development branch.
## Solution
Modified the `concurrency` settings in `.github/workflows/build.yml` to
use separate concurrency groups and conditional cancellation:
```yaml
concurrency:
# Use separate concurrency groups for develop vs other branches to prevent cross-cancellation
group: ${{ github.workflow }}-${{ github.ref }}-${{ github.ref == 'refs/heads/develop' && github.run_number || 'branch' }}
cancel-in-progress: ${{ github.ref != 'refs/heads/develop' }}
```
## Behavior
- **`develop` branch**: Each run gets its own concurrency group (using
`run_number`), allowing parallel execution without cancellation or
queuing delays
- **All other branches**: Shared concurrency group with
`cancel-in-progress: true` - workflows are cancelled when new commits
are pushed
This ensures that the `develop` branch can run multiple workflows in
parallel without interference while still providing the efficiency
benefits of workflow cancellation on feature branches and pull requests.
<!-- START COPILOT CODING AGENT TIPS -->
---
💬 Share your feedback on Copilot coding agent for the chance to win a
$200 gift card! Click
[here](https://survey.alchemer.com/s3/8343779/Copilot-Coding-agent) to
start the survey.
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: kuznetsss <15742918+kuznetsss@users.noreply.github.com>
Co-authored-by: mathbunnyru <12270691+mathbunnyru@users.noreply.github.com>
Having such a long init list in almost every translation unit is
definitely not nice.
It makes it more difficult to work on custom logger (because I need to
adjust log_format), so let's move it to cpp file
…s to set version
Advantages:
- not modifying src dir in build time (`git clean` will also remove
less)
- simpler codecov (if we decide to move it to separate stage in the
future)
- The cpp file is perfectly valid, so it will be treated as a C++ files
by all the tooling
- No need to use CMAKE_PROJECT_INCLUDE_BEFORE, should work fine without
it
- Instead of generating + recompiling, we will just be doing
recompilation of 1 file
Tool works locally and doesn't require internet connection, if
`--no-verification` is passed.
I also tried to add `secret.cpp` with github PAT and the tool detected
it.
The issue is that we previously didn't care about `[conf]` section.
And for example, we uploaded `clang.ubsan` build with the same
package_id as a regular clang build.
This was fixed in https://github.com/XRPLF/clio/pull/2233 and
https://github.com/XRPLF/clio/pull/2234
Adding `global.conf` almost fixed the problem, but since our
non-sanitized builds don't set anything in `[conf]`, we use the same
package_id as before.
So, for the `clang` build we end up with previously uploaded
`clang.ubsan` build artifacts.
To fix this, we should force the upload.
Bumps
[docker/setup-buildx-action](https://github.com/docker/setup-buildx-action)
from 3.10.0 to 3.11.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/docker/setup-buildx-action/releases">docker/setup-buildx-action's
releases</a>.</em></p>
<blockquote>
<h2>v3.11.0</h2>
<ul>
<li>Keep BuildKit state support by <a
href="https://github.com/crazy-max"><code>@crazy-max</code></a> in <a
href="https://redirect.github.com/docker/setup-buildx-action/pull/427">docker/setup-buildx-action#427</a></li>
<li>Remove aliases created when installing by default by <a
href="https://github.com/hashhar"><code>@hashhar</code></a> in <a
href="https://redirect.github.com/docker/setup-buildx-action/pull/139">docker/setup-buildx-action#139</a></li>
<li>Bump <code>@docker/actions-toolkit</code> from 0.56.0 to 0.62.1 in
<a
href="https://redirect.github.com/docker/setup-buildx-action/pull/422">docker/setup-buildx-action#422</a>
<a
href="https://redirect.github.com/docker/setup-buildx-action/pull/425">docker/setup-buildx-action#425</a></li>
</ul>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/docker/setup-buildx-action/compare/v3.10.0...v3.11.0">https://github.com/docker/setup-buildx-action/compare/v3.10.0...v3.11.0</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="18ce135bb5"><code>18ce135</code></a>
Merge pull request <a
href="https://redirect.github.com/docker/setup-buildx-action/issues/425">#425</a>
from docker/dependabot/npm_and_yarn/docker/actions-to...</li>
<li><a
href="0e198e93af"><code>0e198e9</code></a>
chore: update generated content</li>
<li><a
href="05f3f3ac10"><code>05f3f3a</code></a>
build(deps): bump <code>@docker/actions-toolkit</code> from 0.61.0 to
0.62.1</li>
<li><a
href="622913496d"><code>6229134</code></a>
Merge pull request <a
href="https://redirect.github.com/docker/setup-buildx-action/issues/427">#427</a>
from crazy-max/keep-state</li>
<li><a
href="c6f6a07025"><code>c6f6a07</code></a>
chore: update generated content</li>
<li><a
href="6c5e29d848"><code>6c5e29d</code></a>
skip builder creation if one already exists with the same name</li>
<li><a
href="548b297749"><code>548b297</code></a>
ci: keep-state check</li>
<li><a
href="36590ad0c1"><code>36590ad</code></a>
check if driver compatible with keep-state</li>
<li><a
href="4143b5899b"><code>4143b58</code></a>
Support to retain cache</li>
<li><a
href="3f1544eb9e"><code>3f1544e</code></a>
Merge pull request <a
href="https://redirect.github.com/docker/setup-buildx-action/issues/139">#139</a>
from hashhar/hashhar/cleanup-aliases</li>
<li>Additional commits viewable in <a
href="b5ca514318...18ce135bb5">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
I built all conan packages locally and this flag is only required for
grpc, so let's only set it for grpc.
This is better - it's explicit, and we'll know that if we update grpc
recipe, we can remove this.
I also uploaded all rebuilt packages to the artifactory.
Add a wrap for `boost::asio::yield_context` to make async operations
cancellable by default. The API could be a bit adjusted when we start
switching our code to use it.
Bumps
[docker/build-push-action](https://github.com/docker/build-push-action)
from 6.17.0 to 6.18.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/docker/build-push-action/releases">docker/build-push-action's
releases</a>.</em></p>
<blockquote>
<h2>v6.18.0</h2>
<ul>
<li>Bump <code>@docker/actions-toolkit</code> from 0.61.0 to 0.62.1 in
<a
href="https://redirect.github.com/docker/build-push-action/pull/1381">docker/build-push-action#1381</a></li>
</ul>
<blockquote>
<p>[!NOTE]
<a
href="https://docs.docker.com/build/ci/github-actions/build-summary/">Build
summary</a> is now supported with <a
href="https://docs.docker.com/build-cloud/">Docker Build Cloud</a>.</p>
</blockquote>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/docker/build-push-action/compare/v6.17.0...v6.18.0">https://github.com/docker/build-push-action/compare/v6.17.0...v6.18.0</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="263435318d"><code>2634353</code></a>
Merge pull request <a
href="https://redirect.github.com/docker/build-push-action/issues/1381">#1381</a>
from docker/dependabot/npm_and_yarn/docker/actions-t...</li>
<li><a
href="c0432d2e01"><code>c0432d2</code></a>
chore: update generated content</li>
<li><a
href="0bb1f27d6b"><code>0bb1f27</code></a>
set builder driver and endpoint attributes for dbc summary support</li>
<li><a
href="5f9dbf956c"><code>5f9dbf9</code></a>
chore(deps): Bump <code>@docker/actions-toolkit</code> from 0.61.0 to
0.62.1</li>
<li><a
href="0788c444d8"><code>0788c44</code></a>
Merge pull request <a
href="https://redirect.github.com/docker/build-push-action/issues/1375">#1375</a>
from crazy-max/remove-gcr</li>
<li><a
href="aa179ca4f4"><code>aa179ca</code></a>
e2e: remove GCR</li>
<li>See full diff in <a
href="1dc7386353...263435318d">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps
[docker/build-push-action](https://github.com/docker/build-push-action)
from 6.16.0 to 6.17.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/docker/build-push-action/releases">docker/build-push-action's
releases</a>.</em></p>
<blockquote>
<h2>v6.17.0</h2>
<ul>
<li>Bump <code>@docker/actions-toolkit</code> from 0.59.0 to 0.61.0 by
<a href="https://github.com/crazy-max"><code>@crazy-max</code></a> in
<a
href="https://redirect.github.com/docker/build-push-action/pull/1364">docker/build-push-action#1364</a></li>
</ul>
<blockquote>
<p>[!NOTE]
Build record is now exported using the <a
href="https://docs.docker.com/reference/cli/docker/buildx/history/export/"><code>buildx
history export</code></a> command instead of the legacy export-build
tool.</p>
</blockquote>
<p><strong>Full Changelog</strong>: <a
href="https://github.com/docker/build-push-action/compare/v6.16.0...v6.17.0">https://github.com/docker/build-push-action/compare/v6.16.0...v6.17.0</a></p>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="1dc7386353"><code>1dc7386</code></a>
Merge pull request <a
href="https://redirect.github.com/docker/build-push-action/issues/1364">#1364</a>
from crazy-max/history-export-cmd</li>
<li><a
href="9c9803f364"><code>9c9803f</code></a>
chore: update generated content</li>
<li><a
href="db1f6c46e8"><code>db1f6c4</code></a>
DOCKER_BUILD_EXPORT_LEGACY env var to opt-in for legacy export</li>
<li><a
href="721e8c79de"><code>721e8c7</code></a>
Bump <code>@docker/actions-toolkit</code> from 0.59.0 to 0.61.0</li>
<li>See full diff in <a
href="14487ce63c...1dc7386353">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Let's see if this does the trick.
Alternatives are
- more complicated cleanup using sed or similar
- don't multiline the expression (e.g. "${{ multiline expression becomes
single line }}")
- don't wrap `SANITIZER_OPTION` in double quotes (like it was before)
- something else that i did not find - let me know if you know
…eases
Work on: https://github.com/XRPLF/clio/issues/1779
Features:
- works in PRs (everything but actually pushing a release)
- supports custom release notes header
- removes previous release for nightly releases
The idea is that creating releases should be automatic or almost
automatic and should work more or less the same for nightly and actual
releases.
I haven't yet implemented automatic release process for the actual
release (and don't plan to do it here), but this PR will make it much
easier to implement it in the future.
I suggest deferring discussion of how actual releases should work till
the next PR.
I don't think I can change where it's pushed and start using it in the
same PR, so will try to first update the push part, and then change
where the image is used. Only doing it for CI image to not break
anything.
~I will add pre-commit hook later if I find a good one~
Found a nice repo, but it is no longer maintained:
https://github.com/dnephin/pre-commit-golang
So, I implemented the check as a local hook.
I started with really simple pre-commit hooks and will add more on top.
Important files:
- `.pre-commit-config.yaml` - the config for pre-commit
- `.github/workflows/pre-commit.yml` - runs pre-commit hooks in branches
and `develop`
- `.github/workflows/pre-commit-autoupdate.yml` - autoupdates pre-commit
hooks once in a month
Sorry, it seems to have been the wrong change.
I pushed this to `develop` in my fork, enabled Dependabot there and PRs
were created in my fork.
Close: https://github.com/XRPLF/clio/issues/2022
Note: I used exactly the version which are already being used (didn't
update anything).
Updates will come when we merge https://github.com/XRPLF/clio/pull/2018
Fixes#1980.
An SSL handshake was missing and WsConnection should be build from
stream not socket because in case SSL connection stream already
completed SSL handshake.
Implement snapshot import cmd
`clio_snapshot --server --grpc_server 0.0.0.0:12345 --path
<snapshot_path>`
Implement snapshot range cmd
`./clio_snapshot --range --path <snapshot_path>`
Add
LedgerHouses: It is responsible for reading/writing snapshot data
Server: Start grpc server and ws server
There was a data race inside `CoroutineGroup` because internal timer was
used from multiple threads in the methods `asyncWait()` and
`onCoroutineComplete()`. Changing `registerForeign()` to spawn to the
same `yield_context` fixes the problem because now the timer is accessed
only from the same coroutine which has an internal strand.
During debugging I also added websocket support for `request_gun` tool.
Improving array parsing in config:
- Allow null values in arrays for optional fields
- Allow empty array even for required field
- Allow to not put an empty array in config even if array contains
required fields
In this PR:
1 We create a golang grpc client to request data from rippled
2 We store the data into the specific place
3 Add unittests
4 Create build script, the build can be initiated by set conan option
`snapshot` being true.
Please ignore the grpc server part. It will be implemented in importing
tool.
If cache is disabled or Clio starts with and empty DB, `loader_` inside
cache is not created. So calling `CacheLoader::stop()` or
`CacheLoader::wait()` was causing crash.
Add:
- Block server if migration is blocking
- Initialise the migration related table when server starts against
empty DB
Add MigrationInspectorInterface. server uses inspector to check the
migrators status.
Fixes#1825 by removing the check in the gateway_balances RPC handler
that returns the RpcInvalidHotWallet error code if one of the addresses
supplied in the request's `hotwallet` array does not have a trustline
with the `account` from the request.
As stated in the original ticket, this change fixes a discrepancy in
behavior between Clio and rippled, as rippled does not check for
trustline existence when handling gateway_balances RPCs
Co-authored-by: Sergey Kuznetsov <skuznetsov@ripple.com>
Async framework needed a way to do repeating operations (think simplest
cases like AmendmentBlockHandler).
This PR implements the **absolute minimum**, barebones repeating
operations that
- Can't return any values (void)
- Do not take any arguments in the user-provided function
- Can't be scheduled (i.e. a delay before starting repeating)
- Can't be stopped from inside the user block of code (i.e. does not
have stop token or anything like that)
- Can be stopped through the RepeatedOperation's `abort()` function (but
not from the user-provided repeating function)
Fixes#919.
Fixes bugs for new webserver:
- Unhandled exception when closing already closed websocket
- No pings for plain websocket connection
- Server drops websocket connection when client responds to pings but
doesn't send anything
Also changing API of ng connections. Now timeout is set by a separate
method instead of providing it for each call.
This PR implemented the migration framework, which contains the command
line interface to execute migration and helps to migrate data easily.
Please read README.md for more information about this framework.
There are a few files that cannot move the setRange into constructor of
the test because either the place that calls setRange matters or tests
checks range doesn't exist
Bumps
[wandalen/wretry.action](https://github.com/wandalen/wretry.action) from
3.7.0 to 3.7.2.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="8ceaefd717"><code>8ceaefd</code></a>
version 3.7.2</li>
<li><a
href="ce976ac9e7"><code>ce976ac</code></a>
version 3.7.1</li>
<li><a
href="7a8f8d4bf2"><code>7a8f8d4</code></a>
Merge pull request <a
href="https://redirect.github.com/wandalen/wretry.action/issues/174">#174</a>
from dmvict/master</li>
<li><a
href="2103bce855"><code>2103bce</code></a>
Fix action, add option <code>pre_retry_command</code> to call of
subaction</li>
<li>See full diff in <a
href="https://github.com/wandalen/wretry.action/compare/v3.7.0...v3.7.2">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Alex Kremer <akremer@ripple.com>
Bumps
[wandalen/wretry.action](https://github.com/wandalen/wretry.action) from
3.5.0 to 3.7.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="f8754f7974"><code>f8754f7</code></a>
version 3.7.0</li>
<li><a
href="03db9837ed"><code>03db983</code></a>
Merge pull request <a
href="https://redirect.github.com/wandalen/wretry.action/issues/171">#171</a>
from dmvict/docker_readme</li>
<li><a
href="d80901cd5c"><code>d80901c</code></a>
Sync readme for new feature</li>
<li><a
href="e00d406ade"><code>e00d406</code></a>
version 3.6.0</li>
<li><a
href="e00deaa9ba"><code>e00deaa</code></a>
Merge pull request <a
href="https://redirect.github.com/wandalen/wretry.action/issues/170">#170</a>
from dmvict/pre_retry_action</li>
<li><a
href="8b50f3152e"><code>8b50f31</code></a>
Update action, add option <code>pre_retry_command</code> to run command
between retries</li>
<li><a
href="990f16983d"><code>990f169</code></a>
Merge pull request <a
href="https://redirect.github.com/wandalen/wretry.action/issues/167">#167</a>
from Vampire/add-typing</li>
<li><a
href="aeb34f4d13"><code>aeb34f4</code></a>
Add action typing</li>
<li>See full diff in <a
href="https://github.com/wandalen/wretry.action/compare/v3.5.0...v3.7.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Alex Kremer <akremer@ripple.com>
Add constraint + parse json into Config
Second part of refactoring Clio Config; First PR found
[here](https://github.com/XRPLF/clio/pull/1544)
Steps that are left to implement:
- Replacing all the places where we fetch config values (by using
config.valueOr/MaybeValue) to instead get it from Config Definition
- Generate markdown file using Clio Config Description
Fixes#1620.
Cherry pick of #1626 into develop.
- Add timeouts for websocket operations for connections to rippled.
Without these timeouts if connection hangs for some reason, clio
wouldn't know the connection is hanging.
- Fix potential data race in choosing new subscription source which will
forward messages to users.
- Optimise switching between subscription sources.
Implementation of new config definition + methods + UT
Steps that still need to be implemented:
- Make ClioConfigDefinition and it's method to be as constexpr as
possible
- Getting User Config file and populating the values in ConfigDefinition
while checking for constraints on user values
- Replacing all the places where we fetch config values (by using
config.valueOr/MaybeValue) to instead get it from Config Definition
- Generate markdown file using Clio Config Description
Bumps
[wandalen/wretry.action](https://github.com/wandalen/wretry.action) from
1.4.10 to 3.5.0.
<details>
<summary>Commits</summary>
<ul>
<li><a
href="6feedb7ded"><code>6feedb7</code></a>
version 3.5.0</li>
<li><a
href="b5ca17cbcf"><code>b5ca17c</code></a>
Merge pull request <a
href="https://redirect.github.com/wandalen/wretry.action/issues/163">#163</a>
from dmvict/master</li>
<li><a
href="82f08e4ccd"><code>82f08e4</code></a>
Add dependency to build script</li>
<li><a
href="c14e35fac1"><code>c14e35f</code></a>
Merge pull request <a
href="https://redirect.github.com/wandalen/wretry.action/issues/162">#162</a>
from dmvict/master</li>
<li><a
href="979d4e538d"><code>979d4e5</code></a>
Synchronize info with <code>Readme</code> of <code>js_action</code>
branch</li>
<li><a
href="0dd1d5d77d"><code>0dd1d5d</code></a>
version 3.4.0</li>
<li><a
href="8e449ec3ce"><code>8e449ec</code></a>
Merge pull request <a
href="https://redirect.github.com/wandalen/wretry.action/issues/156">#156</a>
from dmvict/master</li>
<li><a
href="050710def1"><code>050710d</code></a>
Update action files, add option <code>steps_context</code>, update file
<code>Readme</code>, add op...</li>
<li><a
href="dfc978b430"><code>dfc978b</code></a>
version 3.3.0</li>
<li><a
href="b7882d347e"><code>b7882d3</code></a>
Merge pull request <a
href="https://redirect.github.com/wandalen/wretry.action/issues/154">#154</a>
from dmvict/master</li>
<li>Additional commits viewable in <a
href="https://github.com/wandalen/wretry.action/compare/v1.4.10...v3.5.0">compare
view</a></li>
</ul>
</details>
<br />
[](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)
Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.
[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)
---
<details>
<summary>Dependabot commands and options</summary>
<br />
You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
</details>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
I'm constantly harassed to follow the rules when I'm not doing anything
relevant.
This is an attempt to ameliorate that rather than `--no-verify` or just
disabling hooks altogether. 🍄☁️
Somebody might want to confirm `basename` is included with macOS.
Fixes#1436
This is a temporary implementation of the `feature` RPC that will always
return `noPermission` iff `vetoed` is set.
If `vetoed` isn't specified, Clio will always forward the request to
`rippled` instead.
In the future, #1131 will implement a Clio-native `feature` RPC. This
requires specific support from `libxrpl` side and that is not going to
be available till at least 2.2.1, hence the temporary forwarding.
It would be great to review the error message and code so that we pick
the right one from the start.
Co-authored-by: Sergey Kuznetsov <skuznetsov@ripple.com>
A release's version string should be a signed annotated tag which Clio
has generally been following.
Uses the date of the commit since that seems like a more useful item to
track to not have identical source builds have different version
strings.
Partially fixes#884.
Adds:
- Docker image for CI on Linux
- Nightly builds without cache and releases
- Nightly clang-tidy checks
- Fix typos in .clang-tidy
Fixes#932
Also fixes#966
Decided not to add Summary type because it has the same functionality as Histogram but makes more calculations on client side (Clio side). See https://prometheus.io/docs/practices/histograms for detailed comparison.
${{ contains(github.ref_name, '-') && '> **Note:** Please remember that this is a release candidate and it is not recommended for production use.' || '' }}
Thank you for your interest in contributing to the `clio` project 🙏
## Workflow
To contribute, please:
1. Fork the repository under your own user.
2. Create a new branch on which to commit/push your changes.
3. Write and test your code.
4. Ensure that your code compiles with the provided build engine and update the provided build engine as part of your PR where needed and where appropriate.
5. Where applicable, write test cases for your code and include those in `unittests`.
6. Ensure your code passes automated checks (e.g. clang-format)
5. Where applicable, write test cases for your code and include those in the relevant subfolder under `tests`.
6. Ensure your code passes [automated checks](#pre-commit-hooks)
7. Squash your commits (i.e. rebase) into as few commits as is reasonable to describe your changes at a high level (typically a single commit for a small change). See below for more details.
8. Open a PR to the main repository onto the _develop_ branch, and follow the provided template.
> **Note:** Please read the [Style guide](#style-guide).
## Install git hooks
Please run the following command in order to use git hooks that are helpful for `clio` development.
### `git lfs` hooks
``` bash
git config --local core.hooksPath .githooks
Install `git lfs` hooks using the following command:
```bash
git lfs install
```
## Git commands
This sections offers a detailed look at the git commands you will need to use to get your PR submitted.
> **Note:** You need to install Git LFS hooks before installing `pre-commit` hooks.
### `pre-commit` hooks
To ensure code quality and style, we use [`pre-commit`](https://pre-commit.com/).
Run the following command to enable `pre-commit` hooks that help with Clio development:
`pre-commit` takes care of running each tool in [`.pre-commit-config.yaml`](https://github.com/XRPLF/clio/blob/develop/.pre-commit-config.yaml) in a separate environment.
`pre-commit` also attempts to automatically use Doxygen to verify that everything public in the codebase has doc comments.
If Doxygen is not installed, the hook issues a warning and recommends installing Doxygen for future commits.
### Git commands
This sections offers a detailed look at the git commands you will need to use to get your PR submitted.
Please note that there are more than one way to do this and these commands are provided for your convenience.
At this point it's assumed that you have already finished working on your feature/bug.
> **Important:** Before you issue any of the commands below, please hit the `Sync fork` button and make sure your fork's `develop` branch is up-to-date with the main `clio` repository.
``` bash
```bash
# Create a backup of your branch
git branch <your feature branch>_bk
@@ -37,18 +61,20 @@ git pull origin develop
git checkout <your feature branch>
git rebase -i develop
```
For each commit in the list other than the first one, enter `s` to squash.
After this is done, you will have the opportunity to write a message for the squashed commit.
> **Hint:** Please use **imperative mood** in the commit message, and capitalize the first word.
``` bash
```bash
# You should now have a single commit on top of a commit in `develop`
git log
```
> **Note:** If there are merge conflicts, please resolve them now.
``` bash
```bash
# Use the same commit message as you did above
git commit -m 'Your message'
git rebase --continue
@@ -56,19 +82,30 @@ git rebase --continue
> **Important:** If you have no GPG keys set up, please follow [this tutorial](https://docs.github.com/en/authentication/managing-commit-signature-verification/adding-a-gpg-key-to-your-github-account)
``` bash
```bash
# Sign the commit with your GPG key, and push your changes
git commit --amend -S
git push --force
```
## Fixing issues found during code review
### Use ccache (optional)
Clio uses `ccache` to speed up compilation. If you want to use it, please make sure it is installed on your machine.
CMake will automatically detect it and use it if it is available.
### Opening a pull request
When a pull request is open CI will perform checks on the new code.
Title of the pull request and squashed commit should follow [conventional commits specification](https://www.conventionalcommits.org/en/v1.0.0/).
### Fixing issues found during code review
While your code is in review, it's possible that some changes will be requested by reviewer(s).
This section describes the process of adding your fixes.
We assume that you already made the required changes on your feature branch.
When your PR is approved and ready to merge, use `Squash and merge`.
The button for that is near the bottom of the PR's page on GitHub.
> **Important:** Please leave the automatically-generated mention/link to the PR in the subject line **and** in the description field add `"Fix #ISSUE_ID"` (replacing `ISSUE_ID` with yours) if the PR fixes an issue.
> **Note:** See [issues](https://github.com/XRPLF/clio/issues) to find the `ISSUE_ID` for the feature/bug you were working on.
# Style guide
## Style guide
This is a non-exhaustive list of recommended style guidelines. These are not always strictly enforced and serve as a way to keep the codebase coherent.
## Formatting
Code must conform to `clang-format` version 16, unless the result would be unreasonably difficult to read or maintain.
To change your code to conform use `clang-format -i <yourchangedfiles>`.
### Formatting
## Avoid
* Proliferation of nearly identical code.
* Proliferation of new files and classes unless it improves readability or/and compilation time.
* Unmanaged memory allocation and raw pointers.
* Macros (unless they add significant value.)
* Lambda patterns (unless these add significant value.)
* CPU or architecture-specific code unless there is a good reason to include it, and where it is used guard it with macros and provide explanatory comments.
* Importing new libraries unless there is a very good reason to do so.
Code must conform to `clang-format`, unless the result is unreasonably difficult to read or maintain.
In most cases the `pre-commit` hook takes care of formatting and fixes any issues automatically.
To manually format your code, run `pre-commit run clang-format --files <your changed files>` for C++ files, and `pre-commit run cmake-format --files <your changed files>` for CMake files.
## Seek to
* Extend functionality of existing code rather than creating new code.
* Prefer readability over terseness where important logic is concerned.
* Inline functions that are not used or are not likely to be used elsewhere in the codebase.
* Use clear and self-explanatory names for functions, variables, structs and classes.
* Use TitleCase for classes, structs and filenames, camelCase for function and variable names, lower case for namespaces and folders.
* Provide as many comments as you feel that a competent programmer would need to understand what your code does.
### Documentation
All public namespaces, classes and functions must be covered by doc (`doxygen`) comments. Everything that is not within a nested `impl` namespace is considered public.
> **Note:** Keep in mind that this is enforced by Clio's CI and your build will fail if newly added public code lacks documentation.
### Avoid
- Proliferation of nearly identical code.
- Proliferation of new files and classes unless it improves readability or/and compilation time.
- Unmanaged memory allocation and raw pointers.
- Macros (unless they add significant value.)
- Lambda patterns (unless these add significant value.)
- CPU or architecture-specific code unless there is a good reason to include it, and where it is used guard it with macros and provide explanatory comments.
- Importing new libraries unless there is a very good reason to do so.
### Seek to
- Extend functionality of existing code rather than creating new code.
- Prefer readability over terseness where important logic is concerned.
- Inline functions that are not used or are not likely to be used elsewhere in the codebase.
- Use clear and self-explanatory names for functions, variables, structs and classes.
- Use TitleCase for classes, structs and filenames, camelCase for function and variable names, lower case for namespaces and folders.
- Provide as many comments as you feel that a competent programmer would need to understand what your code does.
## Maintainers
# Maintainers
Maintainers are ecosystem participants with elevated access to the repository. They are able to push new code, make decisions on when a release should be made, etc.
## Code Review
### Code Review
A PR must be reviewed and approved by at least one of the maintainers before it can be merged.
## Adding and Removing
### Adding and Removing
New maintainers can be proposed by two existing maintainers, subject to a vote by a quorum of the existing maintainers. A minimum of 50% support and a 50% participation is required. In the event of a tie vote, the addition of the new maintainer will be rejected.
Existing maintainers can resign, or be subject to a vote for removal at the behest of two existing maintainers. A minimum of 60% agreement and 50% participation are required. The XRP Ledger Foundation will have the ability, for cause, to remove an existing maintainer without a vote.
Permission to use, copy, modify, and distribute this software for any purpose with or without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
Clio offers the full rippled API, with the caveat that Clio by default only returns validated data.
This means that `ledger_index` defaults to `validated` instead of `current` for all requests.
Other non-validated data is also not returned, such as information about queued transactions.
For requests that require access to the p2p network, such as `fee` or `submit`, Clio automatically forwards the request to a rippled node and propagates the response back to the client.
To access non-validated data for *any* request, simply add `ledger_index: "current"` to the request, and Clio will forward the request to rippled.
Clio is an XRP Ledger API server optimized for RPC calls over WebSocket or JSON-RPC.
It stores validated historical ledger and transaction data in a more space efficient format, and uses up to 4 times less space than [rippled](https://github.com/XRPLF/rippled).
Clio does not connect to the peer-to-peer network. Instead, Clio extracts data from a group of specified rippled nodes. Running Clio requires access to at least one rippled node
from which data can be extracted. The rippled node does not need to be running on the same machine as Clio.
Clio can be configured to store data in [Apache Cassandra](https://cassandra.apache.org/_/index.html) or [ScyllaDB](https://www.scylladb.com/), enabling scalable read throughput.
Multiple Clio nodes can share access to the same dataset, which allows for a highly available cluster of Clio nodes without the need for redundant data storage or computation.
## 📡 Clio and `rippled`
Clio offers the full `rippled` API, with the caveat that Clio by default only returns validated data. This means that `ledger_index` defaults to `validated` instead of `current` for all requests. Other non-validated data, such as information about queued transactions, is also not returned.
Clio retrieves data from a designated group of `rippled` nodes instead of connecting to the peer-to-peer network.
For requests that require access to the peer-to-peer network, such as `fee` or `submit`, Clio automatically forwards the request to a `rippled` node and propagates the response back to the client. To access non-validated data for _any_ request, simply add `ledger_index: "current"` to the request, and Clio will forward the request to `rippled`.
> [!NOTE]
> Clio requires access to at least one `rippled` node, which can run on the same machine as Clio or separately.
## 📚 Learn more about Clio
Below are some useful docs to learn more about Clio.
**For Developers**:
- [How to build Clio](./docs/build-clio.md)
- [Coverage report](./docs/coverage-report.md)
**For Operators**:
- [How to configure Clio and rippled](./docs/configure-clio.md)
Feel free to open an [issue](https://github.com/XRPLF/clio/issues) if you have a feature request or something doesn't work as expected.
If you have any questions about building, running, contributing, using clio or any other, you could always start a new [discussion](https://github.com/XRPLF/clio/discussions).
## Requirements
1. Access to a Cassandra cluster or ScyllaDB cluster. Can be local or remote.
2. Access to one or more rippled nodes. Can be local or remote.
## Building
Clio is built with CMake and uses Conan for managing dependencies.
It is written in C++20 and therefore requires a modern compiler.
## Prerequisites
### Minimum Requirements
- [Python 3.7](https://www.python.org/downloads/)
- [Conan 1.55](https://conan.io/downloads.html)
- [CMake 3.16](https://cmake.org/download/)
- [**Optional**] [GCovr](https://gcc.gnu.org/onlinedocs/gcc/Gcov.html) (needed for code coverage generation)
| Compiler | Version |
|-------------|---------|
| GCC | 11 |
| Clang | 14 |
| Apple Clang | 14.0.3 |
### Conan configuration
Clio does not require anything but default settings in your (`~/.conan/profiles/default`) Conan profile. It's best to have no extra flags specified.
cmake --build . --parallel 8 # or without the number if you feel extra adventurous
```
If all goes well, `conan install` will find required packages and `cmake` will do the rest. you should end up with `clio_server` and `clio_tests` in the `build` directory (the current directory).
> **Tip:** You can omit the `-o tests=True` in `conan install` command above if you don't want to build `clio_tests`.
> **Tip:** To generate a Code Coverage report, include `-o coverage=True` in the `conan install` command above, along with `-o tests=True` to enable tests. After running the `cmake` commands, execute `make clio_tests-ccov`. The coverage report will be found at `clio_tests-llvm-cov/index.html`.
## Running
```sh
./clio_server config.json
```
Clio needs access to a rippled server. The config files of rippled and Clio need
to match in a certain sense.
Clio needs to know:
- the IP of rippled
- the port on which rippled is accepting unencrypted WebSocket connections
- the port on which rippled is handling gRPC requests
rippled needs to open:
- a port to accept unencrypted websocket connections
- a port to handle gRPC requests, with the IP(s) of Clio specified in the `secure_gateway` entry
The example configs of rippled and Clio are setups such that minimal changes are
required. When running locally, the only change needed is to uncomment the `port_grpc`
section of the rippled config. When running Clio and rippled on separate machines,
in addition to uncommenting the `port_grpc` section, a few other steps must be taken:
1. change the `ip` of the first entry of `etl_sources` to the IP where your rippled
server is running
2. open a public, unencrypted WebSocket port on your rippled server
3. change the IP specified in `secure_gateway` of `port_grpc` section of the rippled config
to the IP of your Clio server. This entry can take the form of a comma-separated list if
you are running multiple Clio nodes.
In addition, the parameter `start_sequence` can be included and configured within the top level of the config file. This parameter specifies the sequence of first ledger to extract if the database is empty. Note that ETL extracts ledgers in order and that no backfilling functionality currently exists, meaning Clio will not retroactively learn ledgers older than the one you specify. Choosing to specify this or not will yield the following behavior:
- If this setting is absent and the database is empty, ETL will start with the next ledger validated by the network.
- If this setting is present and the database is not empty, an exception is thrown.
In addition, the optional parameter `finish_sequence` can be added to the json file as well, specifying where the ledger can stop.
To add `start_sequence` and/or `finish_sequence` to the config.json file appropriately, they will be on the same top level of precedence as other parameters (such as `database`, `etl_sources`, `read_only`, etc.) and be specified with an integer. Here is an example snippet from the config file:
```json
"start_sequence": 12345,
"finish_sequence": 54321
```
The parameters `ssl_cert_file` and `ssl_key_file` can also be added to the top level of precedence of our Clio config. `ssl_cert_file` specifies the filepath for your SSL cert while `ssl_key_file` specifies the filepath for your SSL key. It is up to you how to change ownership of these folders for your designated Clio user. Your options include:
- Copying the two files as root somewhere that's accessible by the Clio user, then running `sudo chown` to your user
- Changing the permissions directly so it's readable by your Clio user
- Running Clio as root (strongly discouraged)
An example of how to specify `ssl_cert_file` and `ssl_key_file` in the config:
```json
"server": {
"ip": "0.0.0.0",
"port": 51233
},
"ssl_cert_file": "/full/path/to/cert.file",
"ssl_key_file": "/full/path/to/key.file"
```
Once your config files are ready, start rippled and Clio. It doesn't matter which you
start first, and it's fine to stop one or the other and restart at any given time.
Clio will wait for rippled to sync before extracting any ledgers. If there is already
data in Clio's database, Clio will begin extraction with the ledger whose sequence
is one greater than the greatest sequence currently in the database. Clio will wait
for this ledger to be available. Be aware that the behavior of rippled is to sync to
the most recent ledger on the network, and then backfill. If Clio is extracting ledgers
from rippled, and then rippled is stopped for a significant amount of time and then restarted, rippled
will take time to backfill to the next ledger that Clio wants. The time it takes is proportional
to the amount of time rippled was offline for. Also be aware that the amount rippled backfills
are dependent on the online_delete and ledger_history config values; if these values
are small, and rippled is stopped for a significant amount of time, rippled may never backfill
to the ledger that Clio wants. To avoid this situation, it is advised to keep history
proportional to the amount of time that you expect rippled to be offline. For example, if you
expect rippled to be offline for a few days from time to time, you should keep at least
a few days of history. If you expect rippled to never be offline, then you can keep a very small
amount of history.
Clio can use multiple rippled servers as a data source. Simply add more entries to
the `etl_sources` section. Clio will load balance requests across the servers specified
in this list. As long as one rippled server is up and synced, Clio will continue
extracting ledgers.
In contrast to rippled, Clio will answer RPC requests for the data already in the
database as soon as the server starts. Clio doesn't wait to sync to the network, or
for rippled to sync.
When starting Clio with a fresh database, Clio needs to download a ledger in full.
This can take some time, and depends on database throughput. With a moderately fast
database, this should take less than 10 minutes. If you did not properly set `secure_gateway`
in the `port_grpc` section of rippled, this step will fail. Once the first ledger
is fully downloaded, Clio only needs to extract the changed data for each ledger,
so extraction is much faster and Clio can keep up with rippled in real-time. Even under
intense load, Clio should not lag behind the network, as Clio is not processing the data,
and is simply writing to a database. The throughput of Clio is dependent on the throughput
of your database, but a standard Cassandra or Scylla deployment can handle
the write load of the XRP Ledger without any trouble. Generally the performance considerations
come on the read side, and depends on the number of RPC requests your Clio nodes
are serving. Be aware that very heavy read traffic can impact write throughput. Again, this
is on the database side, so if you are seeing this, upgrade your database.
It is possible to run multiple Clio nodes that share access to the same database.
The Clio nodes don't need to know about each other. You can simply spin up more Clio
nodes pointing to the same database as you wish, and shut them down as you wish.
On startup, each Clio node queries the database for the latest ledger. If this latest
ledger does not change for some time, the Clio node begins extracting ledgers
and writing to the database. If the Clio node detects a ledger that it is trying to
write has already been written, the Clio node will backoff and stop writing. If later
the Clio node sees no ledger written for some time, it will start writing again.
This algorithm ensures that at any given time, one and only one Clio node is writing
to the database.
It is possible to force Clio to only read data, and to never become a writer.
To do this, set `read_only: true` in the config. One common setup is to have a
small number of writer nodes that are inaccessible to clients, with several
read only nodes handling client requests. The number of read only nodes can be scaled
up or down in response to request volume.
When using multiple rippled servers as data sources and multiple Clio nodes,
each Clio node should use the same set of rippled servers as sources. The order doesn't matter.
The only reason not to do this is if you are running servers in different regions, and
you want the Clio nodes to extract from servers in their region. However, if you
are doing this, be aware that database traffic will be flowing across regions,
which can cause high latencies. A possible alternative to this is to just deploy
a database in each region, and the Clio nodes in each region use their region's database.
This is effectively two systems.
Clio supports API versioning as [described here](https://xrpl.org/request-formatting.html#api-versioning).
It's possible to configure `minimum`, `maximum` and `default` version like so:
```json
"api_version": {
"min": 1,
"max": 2,
"default": 1
}
```
All of the above are optional.
Clio will fallback to hardcoded defaults when not specified in the config file or configured values are outside
of the minimum and maximum supported versions hardcoded in `src/rpc/common/APIVersion.h`.
> **Note:** See `example-config.json` for more details.
## Admin rights for requests
By default clio checks admin privileges by IP address from request (only `127.0.0.1` is considered to be an admin).
It is not very secure because the IP could be spoofed.
For a better security `admin_password` could be provided in the `server` section of clio's config:
```json
"server": {
"admin_password": "secret"
}
```
If the password is presented in the config, clio will check the Authorization header (if any) in each request for the password.
The Authorization header should contain type `Password` and the password from the config, e.g. `Password secret`.
Exactly equal password gains admin rights for the request or a websocket connection.
## Prometheus metrics collection
Clio natively supports Prometheus metrics collection. It accepts Prometheus requests on the port configured in `server` section of config.
Prometheus metrics are enabled by default. To disable it add `"prometheus_enabled": false` to the config.
It is important to know that clio responds to Prometheus request only if they are admin requests, so Prometheus should be configured to send admin password in header.
There is an example of docker-compose file, Prometheus and Grafana configs in [examples/infrastructure](examples/infrastructure).
## Using clang-tidy for static analysis
Minimum clang-tidy version required is 16.0.
Clang-tidy could be run by cmake during building the project.
For that provide the option `-o lint=True` for `conan install` command:
## Developing against `rippled` in standalone mode
If you wish you develop against a `rippled` instance running in standalone
mode there are a few quirks of both clio and rippled you need to keep in mind.
You must:
1. Advance the `rippled` ledger to at least ledger 256
2. Wait 10 minutes before first starting clio against this standalone node.
## Logging
Clio provides several logging options, all are configurable via the config file and are detailed below.
`log_level`: The minimum level of severity at which the log message will be outputted by default.
Severity options are `trace`, `debug`, `info`, `warning`, `error`, `fatal`. Defaults to `info`.
`log_format`: The format of log lines produced by clio. Defaults to `"%TimeStamp% (%SourceLocation%) [%ThreadID%] %Channel%:%Severity% %Message%"`.
Each of the variables expands like so
- `TimeStamp`: The full date and time of the log entry
- `SourceLocation`: A partial path to the c++ file and the line number in said file (`source/file/path:linenumber`)
- `ThreadID`: The ID of the thread the log entry is written from
- `Channel`: The channel that this log entry was sent to
- `Severity`: The severity (aka log level) the entry was sent at
- `Message`: The actual log message
`log_channels`: An array of json objects, each overriding properties for a logging `channel`.
At the moment of writing, only `log_level` can be overriden using this mechanism.
Each object is of this format:
```json
{
"channel": "Backend",
"log_level": "fatal"
}
```
If no override is present for a given channel, that channel will log at the severity specified by the global `log_level`.
Overridable log channels: `Backend`, `WebServer`, `Subscriptions`, `RPC`, `ETL` and `Performance`.
> **Note:** See `example-config.json` for more details.
`log_to_console`: Enable/disable log output to console. Options are `true`/`false`. Defaults to true.
`log_directory`: Path to the directory where log files are stored. If such directory doesn't exist, Clio will create it. If not specified, logs are not written to a file.
`log_rotation_size`: The max size of the log file in **megabytes** before it will rotate into a smaller file. Defaults to 2GB.
`log_directory_max_size`: The max size of the log directory in **megabytes** before old log files will be
deleted to free up space. Defaults to 50GB.
`log_rotation_hour_interval`: The time interval in **hours** after the last log rotation to automatically
rotate the current log file. Defaults to 12 hours.
Note, time-based log rotation occurs dependently on size-based log rotation, where if a
size-based log rotation occurs, the timer for the time-based rotation will reset.
`log_tag_style`: Tag implementation to use. Must be one of:
- `uint`: Lock free and threadsafe but outputs just a simple unsigned integer
- `uuid`: Threadsafe and outputs a UUID tag
- `none`: Don't use tagging at all
## Cassandra / Scylla Administration
Since Clio relies on either Cassandra or Scylla for its database backend, here are some important considerations:
- Scylla, by default, will reserve all free RAM on a machine for itself. If you are running `rippled` or other services on the same machine, restrict its memory usage using the `--memory` argument: https://docs.scylladb.com/getting-started/scylla-in-a-shared-environment/
If you have any questions about building, running, contributing, using Clio or any other, you could always start a new [discussion](https://github.com/XRPLF/clio/discussions).
This document contains the release notes for `clio_server`, an XRP Ledger API Server.
To build and run `clio_server`, follow the instructions in [README.md](https://github.com/XRPLF/clio).
If you find issues or have a new idea, please open [an issue](https://github.com/XRPLF/clio/issues).
# Releases
## 0.1.0
Clio is an XRP Ledger API server. Clio is optimized for RPC calls, over websocket or JSON-RPC. Validated historical ledger and transaction data is stored in a more space efficient format, using up to 4 times less space than rippled.
Clio uses Cassandra or ScyllaDB, allowing for scalable read throughput. Multiple clio nodes can share access to the same dataset, allowing for a highly available cluster of clio nodes, without the need for redundant data storage or computation.
**0.1.0** is the first beta of Project Clio. It contains:
-`./src/backend` is the BackendInterface. This provides an abstraction for reading and writing information to a database.
-`./src/etl` is the ReportingETL. The classes in this folder are used to extract information from the P2P network and write it to a database, either locally or over the network.
-`./src/rpc` contains RPC handlers that are called by clients. These handlers should expose the same API as rippled.
-`./src/subscriptions` contains the SubscriptionManager. This manages publishing to clients subscribing to streams or accounts.
-`./src/webserver` contains a flex server that handles both http/s and ws/s traffic on a single port.
-`./unittests` simple unit tests that write to and read from a database to verify that the ETL works.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
