Compare commits

...

39 Commits

Author SHA1 Message Date
Ayaz Salikhov
fa057bd876 ci: Use Nix-based images for all workflows except pre-commit (#3098) 2026-06-16 22:50:43 +01:00
Bryan
e24216c8a9 feat: Add storage and backend for mpt_issuance_history (#3091)
Why
This is the first PR in the broader `mptoken_issuance_history` plan. It
lands the additive schema/backend layer first so later PRs can safely
add live ETL indexing, historical backfill, and finally the RPC handler
gated on backfill completion. Keeping this step dark avoids exposing
partial history while still making the stack reviewable in small pieces.

Summary
Adds the storage and backend primitives for MPT transaction-history
filtering.

- Adds Cassandra tables:

  - mpt_transactions

  - account_mpt_transactions
- Adds MPTTransactionsData and backend interface methods for
writing/fetching both MPT index shapes.
Implements Cassandra write/fetch paths with forward/reverse pagination,
raw-page cursor behavior, and case-insensitive tx_type filtering before
transaction hydration.
- Adds MockBackend coverage for the new pure-virtual methods.
- Adds Cassandra integration tests for round-trip behavior, ordering,
pagination, missing transaction blobs, sparse tx_type filtering, account
fanout, and account-side filtering.
2026-06-16 12:15:45 -07:00
Shawn Xie
0aa7ed4919 feat: Update MPToken ETL to support all transactions types (#3102)
## Summary

Fixes incomplete MPT holder indexing in ETL.

Previously, Clio only indexed MPT holders when an `ltMPTOKEN` ledger
node was created by an `MPTokenAuthorize` transaction. However, with
future amendments, XRPL can also create `ltMPTOKEN` nodes as part of
other successful transactions, such as `Payment`, when a holder receives
an MPT. Those holders were silently skipped and therefore missing from
the `mpt_holders` RPC results.

This change makes MPT holder indexing metadata-driven instead of
transaction-type-driven. ETL now scans successful transaction metadata
for newly created `ltMPTOKEN` nodes and writes every holder it finds.

## What Changed

- MPT holder extraction now works for any successful transaction that
creates an `ltMPTOKEN`.
- Multiple MPT holders created in a single transaction are all indexed.
- The `mpt_holders` table is kept consistent with ledger metadata
instead of relying on assumptions about which transaction types can
create holder nodes.
- Added regression coverage for MPT holders created by a `Payment`
transaction.

## Testing

- `build/clio_tests '--gtest_filter=MPTExtTests.*'`
- `clang-tidy -p build src/etl/MPTHelpers.cpp src/etl/MPTHelpers.hpp
src/etl/impl/ext/MPT.cpp tests/unit/etl/ext/MPTTests.cpp`
2026-06-15 11:46:55 +01:00
dependabot[bot]
b8ac68c6d5 ci: [DEPENDABOT] bump actions/checkout from 6.0.2 to 6.0.3 (#3097) 2026-06-08 09:42:04 +01:00
dependabot[bot]
c53a538a59 ci: [DEPENDABOT] bump codecov/codecov-action from 6.0.1 to 7.0.0 (#3096) 2026-06-08 09:41:09 +01:00
github-actions[bot]
eaf770ccca style: Update pre-commit hooks (#3087)
Co-authored-by: mathbunnyru <12270691+mathbunnyru@users.noreply.github.com>
Co-authored-by: Ayaz Salikhov <mathbunnyru@users.noreply.github.com>
2026-06-01 11:57:14 +01:00
dependabot[bot]
ffca7f9de0 ci: [DEPENDABOT] bump docker/setup-qemu-action from 4.0.0 to 4.1.0 in /.github/actions/build-docker-image (#3088)
Signed-off-by: dependabot[bot] <support@github.com>
2026-06-01 11:44:22 +01:00
Ayaz Salikhov
16c0a64394 chore: Rewrite verify-commits script (#3086) 2026-05-28 11:20:19 +01:00
Ayaz Salikhov
267fe05a8f style: Run shfmt on workflows and actions (#3085) 2026-05-28 11:03:38 +01:00
Ayaz Salikhov
25c0dc6f1c ci: Only push to DockerHub in main repo (#3084) 2026-05-27 14:38:37 +01:00
dependabot[bot]
76043d607f ci: [DEPENDABOT] bump docker/build-push-action from 7.1.0 to 7.2.0 in /.github/actions/build-docker-image (#3083)
Signed-off-by: dependabot[bot] <support@github.com>
2026-05-26 11:33:12 +01:00
dependabot[bot]
8e7f07d1cf ci: [DEPENDABOT] bump docker/login-action from 4.1.0 to 4.2.0 in /.github/actions/build-docker-image (#3082)
Signed-off-by: dependabot[bot] <support@github.com>
2026-05-26 11:33:03 +01:00
dependabot[bot]
43c3732116 ci: [DEPENDABOT] bump docker/metadata-action from 6.0.0 to 6.1.0 in /.github/actions/build-docker-image (#3081)
Signed-off-by: dependabot[bot] <support@github.com>
2026-05-26 11:32:52 +01:00
dependabot[bot]
13f65b6c3c ci: [DEPENDABOT] bump docker/setup-buildx-action from 4.0.0 to 4.1.0 in /.github/actions/build-docker-image (#3080)
Signed-off-by: dependabot[bot] <support@github.com>
2026-05-26 11:32:40 +01:00
dependabot[bot]
aed2240cbd ci: [DEPENDABOT] bump codecov/codecov-action from 6.0.0 to 6.0.1 (#3079)
Signed-off-by: dependabot[bot] <support@github.com>
2026-05-26 11:32:32 +01:00
dependabot[bot]
4da9de304e ci: [DEPENDABOT] bump docker/setup-buildx-action from 4.0.0 to 4.1.0 (#3078)
Signed-off-by: dependabot[bot] <support@github.com>
2026-05-26 11:32:22 +01:00
dependabot[bot]
f189dbf950 ci: [DEPENDABOT] bump docker/login-action from 4.1.0 to 4.2.0 (#3077)
Signed-off-by: dependabot[bot] <support@github.com>
2026-05-26 11:32:16 +01:00
Alex Kremer
cc4f204d9e style: More clang-tidy identifier renaming (#3076) 2026-05-19 15:25:47 +01:00
Ayaz Salikhov
f4dcfe20e9 ci: Run actions/configure-pages only in the main repo (#3075) 2026-05-14 18:14:48 +01:00
Ayaz Salikhov
3112a3d214 ci: Fix how workflows behave in forks (#3074) 2026-05-14 17:59:33 +01:00
Alex Kremer
dbc98894ba refactor: Rename static constants (#3073) 2026-05-13 12:09:06 +01:00
Sergey Kuznetsov
d056c72212 test: Fix integration tests (#3072)
Fix 2 issues in integration tests:
- `127.0.0.2` is forwarded to localhost on macos
- Keyspace is always dropped in test's destructor but it may not exist
2026-05-12 13:49:39 +01:00
Sergey Kuznetsov
a1112bf318 chore: Update libxrpl to 3.1.3 (#3070) 2026-05-11 14:13:22 +01:00
Sergey Kuznetsov
e974e1899f fix: Don't cache responses to admin's requests (#3058) 2026-05-05 17:00:59 +01:00
Ayaz Salikhov
a7090f5b19 ci: Print tidy errors (#3068) 2026-05-05 16:30:49 +01:00
Ayaz Salikhov
f2158cd2d9 ci: Make clang-tidy work on individual files in PRs (#3067) 2026-05-05 15:51:25 +01:00
Alex Kremer
e96ef39857 chore: Fix clang-tidy issues after introducing new checks (#3061) 2026-05-05 15:21:54 +01:00
Ayaz Salikhov
5fe7c4407b ci: Use XRPLF/create-issue (#3066) 2026-05-05 14:21:36 +01:00
Ayaz Salikhov
9b20f1871e style: Update pre-commit hooks (#3057) 2026-05-01 15:32:13 +01:00
Alex Kremer
51244feb4a chore: Enable more clang-tidy checks (#3054) 2026-05-01 15:31:45 +01:00
Ayaz Salikhov
d6bae6c12b ci: Rename print-env -> print-build-env (#3056) 2026-05-01 11:07:32 +01:00
Ayaz Salikhov
fe30c87e2c ci: Use print-env from XRLPF/actions (#3053) 2026-04-29 18:28:45 +01:00
Ayaz Salikhov
c88ffa7c92 chore: Use LLVM tools 21 (#3051) 2026-04-29 16:34:21 +01:00
Ayaz Salikhov
32aeda0ce7 chore: Fix clang-tidy 21 issues (#3052) 2026-04-29 16:17:32 +01:00
Ayaz Salikhov
f174b47f49 ci: Build image with LLVM tools 21 (#3049) 2026-04-29 14:25:10 +01:00
Ayaz Salikhov
80cdb3234a ci: Upload clang-tidy git diff (#3050) 2026-04-29 13:28:46 +01:00
Sergey Kuznetsov
f1460de5d3 feat: Optional log rotation (#3016)
This PR adds an option to disable log rotation.
2026-04-27 15:30:53 +01:00
github-actions[bot]
198773f86a style: clang-tidy auto fixes (#3047)
Fixes #3046. Please review and commit clang-tidy fixes.

Co-authored-by: godexsoft <385326+godexsoft@users.noreply.github.com>
2026-04-27 13:02:05 +01:00
Sergey Kuznetsov
fe0bf736fb refactor: Use error code in make_address() calls (#3044)
Function `ip::make_address()` throws an exception on an invalid IP.
Refactor to a better error handling without exceptions.
2026-04-27 11:32:07 +01:00
514 changed files with 10947 additions and 9742 deletions

View File

@@ -4,6 +4,7 @@ Checks: "-*,
bugprone-assert-side-effect,
bugprone-bad-signal-to-kill-thread,
bugprone-bool-pointer-implicit-conversion,
bugprone-capturing-this-in-member-variable,
bugprone-casting-through-void,
bugprone-chained-comparison,
bugprone-compare-pointer-to-member-virtual-function,
@@ -23,6 +24,7 @@ Checks: "-*,
bugprone-lambda-function-name,
bugprone-macro-parentheses,
bugprone-macro-repeated-side-effects,
bugprone-misleading-setter-of-reference,
bugprone-misplaced-operator-in-strlen-in-alloc,
bugprone-misplaced-pointer-arithmetic-in-alloc,
bugprone-misplaced-widening-cast,
@@ -64,6 +66,7 @@ Checks: "-*,
bugprone-terminating-continue,
bugprone-throw-keyword-missing,
bugprone-too-small-loop-variable,
bugprone-unchecked-optional-access,
bugprone-undefined-memory-manipulation,
bugprone-undelegated-constructor,
bugprone-unhandled-exception-at-new,
@@ -82,6 +85,7 @@ Checks: "-*,
cppcoreguidelines-pro-type-static-cast-downcast,
cppcoreguidelines-rvalue-reference-param-not-moved,
cppcoreguidelines-use-default-member-init,
cppcoreguidelines-use-enum-class,
cppcoreguidelines-virtual-class-destructor,
hicpp-ignored-remove-result,
llvm-namespace-comment,
@@ -105,8 +109,10 @@ Checks: "-*,
modernize-use-emplace,
modernize-use-equals-default,
modernize-use-equals-delete,
modernize-use-nodiscard,
modernize-use-override,
modernize-use-ranges,
modernize-use-scoped-lock,
modernize-use-starts-ends-with,
modernize-use-std-numbers,
modernize-use-using,
@@ -118,6 +124,7 @@ Checks: "-*,
performance-move-constructor-init,
performance-no-automatic-move,
performance-trivially-destructible,
readability-ambiguous-smartptr-reset-call,
readability-avoid-nested-conditional-operator,
readability-avoid-return-with-void-value,
readability-braces-around-statements,
@@ -128,9 +135,9 @@ Checks: "-*,
readability-duplicate-include,
readability-else-after-return,
readability-enum-initial-value,
readability-identifier-naming,
readability-implicit-bool-conversion,
readability-inconsistent-declaration-parameter-name,
readability-identifier-naming,
readability-make-member-function-const,
readability-math-missing-parentheses,
readability-misleading-indentation,
@@ -149,6 +156,11 @@ Checks: "-*,
"
CheckOptions:
bugprone-unsafe-functions.ReportMoreUnsafeFunctions: true
bugprone-unused-return-value.CheckedReturnTypes: ::std::error_code;::std::error_condition;::std::errc
misc-include-cleaner.IgnoreHeaders: '.*/(detail|impl)/.*;.*(expected|unexpected).*;.*ranges_lower_bound\.h;time.h;stdlib.h;__chrono/.*;fmt/chrono.h;boost/uuid/uuid_hash.hpp'
readability-braces-around-statements.ShortStatementLines: 2
readability-identifier-naming.MacroDefinitionCase: UPPER_CASE
readability-identifier-naming.ClassCase: CamelCase
@@ -157,7 +169,7 @@ CheckOptions:
readability-identifier-naming.EnumCase: CamelCase
readability-identifier-naming.EnumConstantCase: CamelCase
readability-identifier-naming.ScopedEnumConstantCase: CamelCase
readability-identifier-naming.GlobalConstantCase: UPPER_CASE
readability-identifier-naming.GlobalConstantCase: CamelCase
readability-identifier-naming.GlobalConstantPrefix: "k"
readability-identifier-naming.GlobalVariableCase: CamelCase
readability-identifier-naming.GlobalVariablePrefix: "g"
@@ -165,27 +177,25 @@ CheckOptions:
readability-identifier-naming.ConstexprMethodCase: camelBack
readability-identifier-naming.ClassMethodCase: camelBack
readability-identifier-naming.ClassMemberCase: camelBack
readability-identifier-naming.ClassConstantCase: UPPER_CASE
readability-identifier-naming.ClassConstantCase: CamelCase
readability-identifier-naming.ClassConstantPrefix: "k"
readability-identifier-naming.StaticConstantCase: UPPER_CASE
readability-identifier-naming.StaticConstantCase: CamelCase
readability-identifier-naming.StaticConstantPrefix: "k"
readability-identifier-naming.StaticVariableCase: UPPER_CASE
readability-identifier-naming.StaticVariablePrefix: "k"
readability-identifier-naming.ConstexprVariableCase: UPPER_CASE
readability-identifier-naming.ConstexprVariablePrefix: "k"
readability-identifier-naming.StaticVariableCase: camelBack
readability-identifier-naming.ConstexprVariableCase: camelBack
readability-identifier-naming.LocalConstantCase: camelBack
readability-identifier-naming.LocalVariableCase: camelBack
readability-identifier-naming.TemplateParameterCase: CamelCase
readability-identifier-naming.ParameterCase: camelBack
readability-identifier-naming.FunctionCase: camelBack
readability-identifier-naming.MemberCase: camelBack
readability-identifier-naming.PrivateMemberCase: camelBack
readability-identifier-naming.PrivateMemberSuffix: _
readability-identifier-naming.ProtectedMemberCase: camelBack
readability-identifier-naming.ProtectedMemberSuffix: _
readability-identifier-naming.PublicMemberCase: camelBack
readability-identifier-naming.PublicMemberSuffix: ""
readability-identifier-naming.FunctionIgnoredRegexp: ".*tag_invoke.*"
bugprone-unsafe-functions.ReportMoreUnsafeFunctions: true
bugprone-unused-return-value.CheckedReturnTypes: ::std::error_code;::std::error_condition;::std::errc
misc-include-cleaner.IgnoreHeaders: '.*/(detail|impl)/.*;.*(expected|unexpected).*;.*ranges_lower_bound\.h;time.h;stdlib.h;__chrono/.*;fmt/chrono.h;boost/uuid/uuid_hash.hpp'
HeaderFilterRegex: '^.*/(src|tests)/.*\.(h|hpp)$'
WarningsAsErrors: "*"

View File

@@ -17,3 +17,6 @@ endfunction()
function(append_coverage_compiler_flags_to_target name mode)
endfunction()
function(patch_nix_binary target)
endfunction()

View File

@@ -26,6 +26,6 @@ runs:
run: |
cd build
cmake \
--build . \
--parallel "${{ steps.nproc.outputs.nproc }}" \
--target ${CMAKE_TARGETS}
--build . \
--parallel "${{ steps.nproc.outputs.nproc }}" \
--target ${CMAKE_TARGETS}

View File

@@ -34,32 +34,32 @@ runs:
steps:
- name: Login to DockerHub
if: ${{ inputs.push_image == 'true' && inputs.dockerhub_repo != '' }}
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
with:
username: ${{ env.DOCKERHUB_USER }}
password: ${{ env.DOCKERHUB_PW }}
- name: Login to GitHub Container Registry
if: ${{ inputs.push_image == 'true' }}
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ env.GITHUB_TOKEN }}
- uses: docker/setup-qemu-action@ce360397dd3f832beb865e1373c09c0e9f86d70a # v4.0.0
- uses: docker/setup-qemu-action@06116385d9baf250c9f4dcb4858b16962ea869c3 # v4.1.0
with:
cache-image: false
- uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
- uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4.1.0
- uses: docker/metadata-action@030e881283bb7a6894de51c315a6bfe6a94e05cf # v6.0.0
- uses: docker/metadata-action@80c7e94dd9b9319bd5eb7a0e0fe9291e23a2a2e9 # v6.1.0
id: meta
with:
images: ${{ inputs.images }}
tags: ${{ inputs.tags }}
- name: Build and push
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
uses: docker/build-push-action@f9f3042f7e2789586610d6e8b85c8f03e5195baf # v7.2.0
with:
context: ${{ inputs.directory }}
platforms: ${{ inputs.platforms }}

View File

@@ -0,0 +1,53 @@
name: Build identifier
description: Generate a unique build identifier and the ccache key derived from it
inputs:
build_type:
description: Current build type (e.g. Release, Debug)
required: true
compiler:
description: Compiler used for the build (e.g. gcc, clang)
required: true
code_coverage:
description: Whether code coverage is on
required: true
sanitizers:
description: Sanitizer to enable, read by the 'sanitizers' conan profile (e.g. 'address', 'thread', 'undefinedbehavior')
required: true
outputs:
build_identifier:
description: Unique identifier for the build configuration (without commit)
value: ${{ steps.build_identifier.outputs.build_identifier }}
cache_key:
description: ccache key, the build identifier suffixed with the common ancestor commit
value: ${{ steps.build_identifier.outputs.build_identifier }}-${{ steps.git_common_ancestor.outputs.commit }}
runs:
using: composite
steps:
- name: Find common commit
id: git_common_ancestor
uses: ./.github/actions/git-common-ancestor
- name: Build identifier
id: build_identifier
shell: bash
env:
RUNNER_OS: ${{ runner.os }}
BUILD_TYPE: ${{ inputs.build_type }}
COMPILER: ${{ inputs.compiler }}
CODE_COVERAGE: ${{ inputs.code_coverage }}
SANITIZERS: ${{ inputs.sanitizers }}
run: |
# Keep the legacy "<os>_<build_type>_<compiler>" layout so standard artifact
# names (e.g. clio_server_Linux_Release_gcc) stay backwards compatible.
# Sanitizer/coverage builds get extra suffixes and may differ from old names.
BUILD_IDENTIFIER="${RUNNER_OS}_${BUILD_TYPE}_${COMPILER}"
if [ "${CODE_COVERAGE}" == "true" ]; then
BUILD_IDENTIFIER+="_code_coverage"
fi
if [ -n "${SANITIZERS}" ]; then
BUILD_IDENTIFIER+="_${SANITIZERS}"
fi
echo "build_identifier=${BUILD_IDENTIFIER}" >>"${GITHUB_OUTPUT}"

View File

@@ -1,41 +0,0 @@
name: Cache key
description: Generate cache key for ccache
inputs:
conan_profile:
description: Conan profile name
required: true
build_type:
description: Current build type (e.g. Release, Debug)
required: true
default: Release
code_coverage:
description: Whether code coverage is on
required: true
default: "false"
outputs:
key:
description: Generated cache key for ccache
value: ${{ steps.key_without_commit.outputs.key }}-${{ steps.git_common_ancestor.outputs.commit }}
restore_keys:
description: Cache restore keys for fallback
value: ${{ steps.key_without_commit.outputs.key }}
runs:
using: composite
steps:
- name: Find common commit
id: git_common_ancestor
uses: ./.github/actions/git-common-ancestor
- name: Set cache key without commit
id: key_without_commit
shell: bash
env:
RUNNER_OS: ${{ runner.os }}
BUILD_TYPE: ${{ inputs.build_type }}
CODE_COVERAGE: ${{ inputs.code_coverage == 'true' && '-code_coverage' || '' }}
CONAN_PROFILE: ${{ inputs.conan_profile }}
run: |
echo "key=clio-ccache-${RUNNER_OS}-${BUILD_TYPE}${CODE_COVERAGE}-${CONAN_PROFILE}-develop" >> "${GITHUB_OUTPUT}"

View File

@@ -6,9 +6,6 @@ inputs:
description: Build directory
required: false
default: "build"
conan_profile:
description: Conan profile name
required: true
build_type:
description: Build type for third-party libraries and clio. Could be 'Release', 'Debug'
required: true
@@ -25,10 +22,6 @@ inputs:
description: Whether to enable code coverage
required: true
default: "false"
static:
description: Whether Clio is to be statically linked
required: true
default: "false"
time_trace:
description: Whether to enable compiler trace reports
required: true
@@ -50,15 +43,9 @@ runs:
env:
BUILD_DIR: "${{ inputs.build_dir }}"
BUILD_TYPE: "${{ inputs.build_type }}"
SANITIZER_OPTION: |-
${{ endsWith(inputs.conan_profile, '.asan') && '-Dsan=address' ||
endsWith(inputs.conan_profile, '.tsan') && '-Dsan=thread' ||
endsWith(inputs.conan_profile, '.ubsan') && '-Dsan=undefined' ||
'' }}
INTEGRATION_TESTS: "${{ inputs.integration_tests == 'true' && 'ON' || 'OFF' }}"
BENCHMARK: "${{ inputs.benchmark == 'true' && 'ON' || 'OFF' }}"
COVERAGE: "${{ inputs.code_coverage == 'true' && 'ON' || 'OFF' }}"
STATIC: "${{ inputs.static == 'true' && 'ON' || 'OFF' }}"
TIME_TRACE: "${{ inputs.time_trace == 'true' && 'ON' || 'OFF' }}"
PACKAGE: "${{ inputs.package == 'true' && 'ON' || 'OFF' }}"
# GitHub creates a merge commit for a PR
@@ -76,16 +63,14 @@ runs:
FORCE_CLIO_VERSION: ${{ inputs.version }}
run: |
cmake \
-B "${BUILD_DIR}" \
-S . \
-G Ninja \
-DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake \
-DCMAKE_BUILD_TYPE="${BUILD_TYPE}" \
"${SANITIZER_OPTION}" \
-Dtests=ON \
-Dintegration_tests="${INTEGRATION_TESTS}" \
-Dbenchmark="${BENCHMARK}" \
-Dcoverage="${COVERAGE}" \
-Dstatic="${STATIC}" \
-Dtime_trace="${TIME_TRACE}" \
-Dpackage="${PACKAGE}"
-B "${BUILD_DIR}" \
-S . \
-G Ninja \
-DCMAKE_TOOLCHAIN_FILE:FILEPATH=build/generators/conan_toolchain.cmake \
-DCMAKE_BUILD_TYPE="${BUILD_TYPE}" \
-Dtests=ON \
-Dintegration_tests="${INTEGRATION_TESTS}" \
-Dbenchmark="${BENCHMARK}" \
-Dcoverage="${COVERAGE}" \
-Dtime_trace="${TIME_TRACE}" \
-Dpackage="${PACKAGE}"

View File

@@ -15,13 +15,13 @@ runs:
shell: bash
run: |
gcovr \
-e benchmarks \
-e tests \
-e src/data/cassandra \
-e src/data/CassandraBackend.hpp \
-e 'src/data/BackendFactory.*' \
--xml build/coverage_report.xml \
-j8 --exclude-throw-branches
-e benchmarks \
-e tests \
-e src/data/cassandra \
-e src/data/CassandraBackend.hpp \
-e 'src/data/BackendFactory.*' \
--xml build/coverage_report.xml \
-j8 --exclude-throw-branches
- name: Archive coverage report
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1

View File

@@ -6,9 +6,6 @@ inputs:
description: Build directory
required: false
default: "build"
conan_profile:
description: Conan profile name
required: true
force_conan_source_build:
description: Whether conan should build all dependencies from source
required: true
@@ -17,6 +14,10 @@ inputs:
description: Build type for third-party libraries and clio. Could be 'Release', 'Debug'
required: true
default: "Release"
sanitizers:
description: Sanitizer to enable, read by the 'sanitizers' conan profile (e.g. 'address', 'thread', 'undefinedbehavior')
required: false
default: ""
runs:
using: composite
@@ -27,11 +28,11 @@ runs:
BUILD_DIR: "${{ inputs.build_dir }}"
CONAN_BUILD_OPTION: "${{ inputs.force_conan_source_build == 'true' && '*' || 'missing' }}"
BUILD_TYPE: "${{ inputs.build_type }}"
CONAN_PROFILE: "${{ inputs.conan_profile }}"
SANITIZERS: "${{ inputs.sanitizers }}"
run: |
conan \
install . \
-of "${BUILD_DIR}" \
-b "${CONAN_BUILD_OPTION}" \
-s "build_type=${BUILD_TYPE}" \
--profile:all "${CONAN_PROFILE}"
install . \
-of "${BUILD_DIR}" \
-b "${CONAN_BUILD_OPTION}" \
-s "build_type=${BUILD_TYPE}" \
--profile:all ci

View File

@@ -1,46 +0,0 @@
name: Create an issue
description: Create an issue
inputs:
title:
description: Issue title
required: true
body:
description: Issue body
required: true
labels:
description: Comma-separated list of labels
required: true
default: "bug"
assignees:
description: Comma-separated list of assignees
required: true
default: "godexsoft,kuznetsss,PeterChen13579,mathbunnyru"
outputs:
created_issue_id:
description: Created issue id
value: ${{ steps.create_issue.outputs.created_issue }}
runs:
using: composite
steps:
- name: Create an issue
id: create_issue
shell: bash
env:
ISSUE_BODY: ${{ inputs.body }}
ISSUE_ASSIGNEES: ${{ inputs.assignees }}
ISSUE_LABELS: ${{ inputs.labels }}
ISSUE_TITLE: ${{ inputs.title }}
run: |
echo -e "${ISSUE_BODY}" > issue.md
gh issue create \
--assignee "${ISSUE_ASSIGNEES}" \
--label "${ISSUE_LABELS}" \
--title "${ISSUE_TITLE}" \
--body-file ./issue.md \
> create_issue.log
created_issue="$(sed 's|.*/||' create_issue.log)"
echo "created_issue=$created_issue" >> $GITHUB_OUTPUT
rm create_issue.log issue.md

View File

@@ -13,4 +13,4 @@ runs:
id: find_common_ancestor
shell: bash
run: |
echo "commit=\"$(git merge-base --fork-point origin/develop)\"" >> $GITHUB_OUTPUT
echo "commit=\"$(git merge-base --fork-point origin/develop)\"" >>$GITHUB_OUTPUT

View File

@@ -0,0 +1,34 @@
name: Set compiler environment
description: "Set CC and CXX environment variables for the given compiler."
inputs:
compiler:
description: 'The compiler to use ("gcc" or "clang").'
required: true
runs:
using: composite
steps:
- name: Set CC and CXX for gcc
if: ${{ inputs.compiler == 'gcc' }}
shell: bash
run: |
echo "CC=gcc" >>"${GITHUB_ENV}"
echo "CXX=g++" >>"${GITHUB_ENV}"
- name: Set CC and CXX for clang
if: ${{ inputs.compiler == 'clang' }}
shell: bash
run: |
echo "CC=clang" >>"${GITHUB_ENV}"
echo "CXX=clang++" >>"${GITHUB_ENV}"
- name: Fail on unknown compiler
if: ${{ inputs.compiler != 'gcc' && inputs.compiler != 'clang' }}
shell: bash
env:
COMPILER: ${{ inputs.compiler }}
run: |
echo "Unknown compiler: $COMPILER" >&2
exit 1

115
.github/dependabot.yml vendored
View File

@@ -1,111 +1,16 @@
version: 2
updates:
- package-ecosystem: github-actions
directory: /
schedule:
interval: weekly
day: monday
time: "04:00"
timezone: Etc/GMT
reviewers:
- XRPLF/clio-dev-team
commit-message:
prefix: "ci: [DEPENDABOT] "
target-branch: develop
- package-ecosystem: github-actions
directory: .github/actions/build-clio/
schedule:
interval: weekly
day: monday
time: "04:00"
timezone: Etc/GMT
reviewers:
- XRPLF/clio-dev-team
commit-message:
prefix: "ci: [DEPENDABOT] "
target-branch: develop
- package-ecosystem: github-actions
directory: .github/actions/build-docker-image/
schedule:
interval: weekly
day: monday
time: "04:00"
timezone: Etc/GMT
reviewers:
- XRPLF/clio-dev-team
commit-message:
prefix: "ci: [DEPENDABOT] "
target-branch: develop
- package-ecosystem: github-actions
directory: .github/actions/cmake/
schedule:
interval: weekly
day: monday
time: "04:00"
timezone: Etc/GMT
reviewers:
- XRPLF/clio-dev-team
commit-message:
prefix: "ci: [DEPENDABOT] "
target-branch: develop
- package-ecosystem: github-actions
directory: .github/actions/code-coverage/
schedule:
interval: weekly
day: monday
time: "04:00"
timezone: Etc/GMT
reviewers:
- XRPLF/clio-dev-team
commit-message:
prefix: "ci: [DEPENDABOT] "
target-branch: develop
- package-ecosystem: github-actions
directory: .github/actions/conan/
schedule:
interval: weekly
day: monday
time: "04:00"
timezone: Etc/GMT
reviewers:
- XRPLF/clio-dev-team
commit-message:
prefix: "ci: [DEPENDABOT] "
target-branch: develop
- package-ecosystem: github-actions
directory: .github/actions/create-issue/
schedule:
interval: weekly
day: monday
time: "04:00"
timezone: Etc/GMT
reviewers:
- XRPLF/clio-dev-team
commit-message:
prefix: "ci: [DEPENDABOT] "
target-branch: develop
- package-ecosystem: github-actions
directory: .github/actions/git-common-ancestor/
schedule:
interval: weekly
day: monday
time: "04:00"
timezone: Etc/GMT
reviewers:
- XRPLF/clio-dev-team
commit-message:
prefix: "ci: [DEPENDABOT] "
target-branch: develop
- package-ecosystem: github-actions
directory: .github/actions/cache-key/
directories:
- /
- .github/actions/build-clio/
- .github/actions/build-docker-image/
- .github/actions/build-identifier/
- .github/actions/cmake/
- .github/actions/code-coverage/
- .github/actions/conan/
- .github/actions/git-common-ancestor/
- .github/actions/set-compiler-env/
schedule:
interval: weekly
day: monday

View File

@@ -3,9 +3,7 @@ import itertools
import json
LINUX_OS = ["heavy", "heavy-arm64"]
LINUX_CONTAINERS = [
'{ "image": "ghcr.io/xrplf/clio-ci:14342e087ceb8b593027198bf9ef06a43833c696" }'
]
LINUX_CONTAINERS = ['{ "image": "ghcr.io/xrplf/xrpld/nix-ubuntu:sha-7b9d553" }']
LINUX_COMPILERS = ["gcc", "clang"]
MACOS_OS = ["macos15"]
@@ -13,7 +11,10 @@ MACOS_CONTAINERS = [""]
MACOS_COMPILERS = ["apple-clang"]
BUILD_TYPES = ["Release", "Debug"]
SANITIZER_EXT = [".asan", ".tsan", ".ubsan", ""]
# Values of the `SANITIZERS` environment variable read by the `sanitizers` conan
# profile. An empty string builds without any sanitizers.
SANITIZERS = ["address", "thread", "undefinedbehavior", ""]
def generate_matrix():
@@ -23,13 +24,13 @@ def generate_matrix():
itertools.product(LINUX_OS, LINUX_CONTAINERS, LINUX_COMPILERS),
itertools.product(MACOS_OS, MACOS_CONTAINERS, MACOS_COMPILERS),
):
for sanitizer_ext, build_type in itertools.product(SANITIZER_EXT, BUILD_TYPES):
for sanitizers, build_type in itertools.product(SANITIZERS, BUILD_TYPES):
configurations.append(
{
"os": os,
"container": container,
"compiler": compiler,
"sanitizer_ext": sanitizer_ext,
"sanitizers": sanitizers,
"build_type": build_type,
}
)

View File

@@ -1,48 +0,0 @@
#!/bin/bash
set -ex
CURRENT_DIR="$(cd "$(dirname "$0")" && pwd)"
REPO_DIR="$(cd "$CURRENT_DIR/../../../" && pwd)"
CONAN_DIR="${CONAN_HOME:-$HOME/.conan2}"
PROFILES_DIR="$CONAN_DIR/profiles"
# When developers' compilers are updated, these profiles might be different
if [[ -z "$CI" ]]; then
APPLE_CLANG_PROFILE="$CURRENT_DIR/apple-clang-17.profile"
else
APPLE_CLANG_PROFILE="$CURRENT_DIR/apple-clang-17.profile"
fi
GCC_PROFILE="$REPO_DIR/docker/ci/conan/gcc.profile"
CLANG_PROFILE="$REPO_DIR/docker/ci/conan/clang.profile"
SANITIZER_TEMPLATE_FILE="$REPO_DIR/docker/ci/conan/sanitizer_template.profile"
rm -rf "$CONAN_DIR"
conan remote add --index 0 xrplf https://conan.ripplex.io
cp "$REPO_DIR/docker/ci/conan/global.conf" "$CONAN_DIR/global.conf"
create_profile_with_sanitizers() {
profile_name="$1"
profile_source="$2"
cp "$profile_source" "$PROFILES_DIR/$profile_name"
cp "$SANITIZER_TEMPLATE_FILE" "$PROFILES_DIR/$profile_name.asan"
cp "$SANITIZER_TEMPLATE_FILE" "$PROFILES_DIR/$profile_name.tsan"
cp "$SANITIZER_TEMPLATE_FILE" "$PROFILES_DIR/$profile_name.ubsan"
}
mkdir -p "$PROFILES_DIR"
if [[ "$(uname)" == "Darwin" ]]; then
create_profile_with_sanitizers "apple-clang" "$APPLE_CLANG_PROFILE"
echo "include(apple-clang)" >"$PROFILES_DIR/default"
else
create_profile_with_sanitizers "clang" "$CLANG_PROFILE"
create_profile_with_sanitizers "gcc" "$GCC_PROFILE"
echo "include(gcc)" >"$PROFILES_DIR/default"
fi

View File

@@ -22,4 +22,4 @@ rm -f conan.lock
# Create a new lockfile that is compatible with macOS.
# It should also work on Linux.
conan lock create . \
--profile:all=.github/scripts/conan/apple-clang-17.profile
--profile:all=./conan/profiles/apple-clang-17.profile

View File

@@ -14,9 +14,9 @@ on:
type: boolean
description: Whether to strip clio binary
default: true
publish_image:
push_image:
type: boolean
description: Whether to publish docker image
description: Whether to push docker image
required: true
workflow_dispatch:
@@ -43,12 +43,12 @@ defaults:
shell: bash
jobs:
build_and_publish_image:
build_and_push_image:
name: Build and publish image
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- name: Download Clio binary from artifact
if: ${{ inputs.artifact_name != null }}
@@ -65,8 +65,8 @@ jobs:
run: |
wget "${BINARY_URL}" -P ./docker/clio/artifact/
if [ "$(sha256sum ./docker/clio/clio_server | awk '{print $1}')" != "${BINARY_SHA256}" ]; then
echo "Binary sha256 sum doesn't match"
exit 1
echo "Binary sha256 sum doesn't match"
exit 1
fi
- name: Unpack binary
run: |
@@ -74,9 +74,9 @@ jobs:
cd docker/clio/artifact
artifact=$(find . -type f)
if [[ $artifact == *.zip ]]; then
unzip $artifact
unzip $artifact
elif [[ $artifact == *.tar.gz ]]; then
tar -xvf $artifact
tar -xvf $artifact
fi
chmod +x ./clio_server
mv ./clio_server ../
@@ -90,7 +90,7 @@ jobs:
- name: Set GHCR_REPO
id: set-ghcr-repo
run: |
echo "GHCR_REPO=$(echo ghcr.io/${{ github.repository_owner }} | tr '[:upper:]' '[:lower:]')" >> ${GITHUB_OUTPUT}
echo "GHCR_REPO=$(echo ghcr.io/${{ github.repository_owner }} | tr '[:upper:]' '[:lower:]')" >>${GITHUB_OUTPUT}
- name: Build Docker image
uses: ./.github/actions/build-docker-image
@@ -102,7 +102,7 @@ jobs:
images: |
ghcr.io/${{ steps.set-ghcr-repo.outputs.GHCR_REPO }}/clio
${{ github.repository_owner == 'XRPLF' && 'rippleci/clio' || '' }}
push_image: ${{ inputs.publish_image }}
push_image: ${{ inputs.push_image }}
directory: docker/clio
tags: ${{ inputs.tags }}
platforms: linux/amd64

View File

@@ -15,7 +15,6 @@ on:
- ".github/actions/**"
- "!.github/actions/build-docker-image/**"
- "!.github/actions/create-issue/**"
- CMakeLists.txt
- conanfile.py
@@ -46,30 +45,24 @@ jobs:
fail-fast: false
matrix:
os: [heavy]
conan_profile: [gcc, clang]
compiler: [gcc, clang]
build_type: [Release, Debug]
container:
[
'{ "image": "ghcr.io/xrplf/clio-ci:14342e087ceb8b593027198bf9ef06a43833c696" }',
]
static: [true]
container: ['{ "image": "ghcr.io/xrplf/xrpld/nix-ubuntu:sha-7b9d553" }']
include:
- os: macos15
conan_profile: apple-clang
compiler: apple-clang
build_type: Release
container: ""
static: false
uses: ./.github/workflows/reusable-build-test.yml
with:
runs_on: ${{ matrix.os }}
container: ${{ matrix.container }}
conan_profile: ${{ matrix.conan_profile }}
compiler: ${{ matrix.compiler }}
build_type: ${{ matrix.build_type }}
download_ccache: true
upload_ccache: true
static: ${{ matrix.static }}
run_unit_tests: true
run_integration_tests: false
upload_clio_server: true
@@ -80,13 +73,12 @@ jobs:
uses: ./.github/workflows/reusable-build.yml
with:
runs_on: heavy
container: '{ "image": "ghcr.io/xrplf/clio-ci:14342e087ceb8b593027198bf9ef06a43833c696" }'
conan_profile: gcc
container: '{ "image": "ghcr.io/xrplf/xrpld/nix-ubuntu:sha-7b9d553" }'
compiler: gcc
build_type: Debug
download_ccache: true
upload_ccache: true
code_coverage: true
static: true
upload_clio_server: false
targets: all
analyze_build_time: false
@@ -98,10 +90,10 @@ jobs:
needs: build-and-test
runs-on: heavy
container:
image: ghcr.io/xrplf/clio-ci:14342e087ceb8b593027198bf9ef06a43833c696
image: ghcr.io/xrplf/xrpld/nix-ubuntu:sha-7b9d553
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:

View File

@@ -10,7 +10,7 @@ concurrency:
cancel-in-progress: true
env:
CONAN_PROFILE: gcc
COMPILER: gcc
defaults:
run:
@@ -21,10 +21,10 @@ jobs:
name: Build Clio / `libXRPL ${{ github.event.client_payload.version }}`
runs-on: heavy
container:
image: ghcr.io/xrplf/clio-ci:14342e087ceb8b593027198bf9ef06a43833c696
image: ghcr.io/xrplf/xrpld/nix-ubuntu:sha-7b9d553
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
fetch-depth: 0
@@ -33,6 +33,17 @@ jobs:
with:
enable_ccache: false
- name: Print build environment
uses: XRPLF/actions/print-build-env@59dec886e4afb05a1724443af08baccbc045b574
- name: Set compiler environment
uses: ./.github/actions/set-compiler-env
with:
compiler: ${{ env.COMPILER }}
- name: Setup conan
run: conan/init.sh
- name: Update libXRPL version requirement
run: |
sed -i.bak -E "s|'xrpl/[a-zA-Z0-9\\.\\-]+'|'xrpl/${{ github.event.client_payload.conan_ref }}'|g" conanfile.py
@@ -40,17 +51,13 @@ jobs:
- name: Update conan lockfile
run: |
conan lock create . --profile:all ${{ env.CONAN_PROFILE }}
conan lock create . --profile:all ci
- name: Run conan
uses: ./.github/actions/conan
with:
conan_profile: ${{ env.CONAN_PROFILE }}
- name: Run CMake
uses: ./.github/actions/cmake
with:
conan_profile: ${{ env.CONAN_PROFILE }}
- name: Build Clio
uses: ./.github/actions/build-clio
@@ -69,7 +76,7 @@ jobs:
needs: build
runs-on: heavy
container:
image: ghcr.io/xrplf/clio-ci:14342e087ceb8b593027198bf9ef06a43833c696
image: ghcr.io/xrplf/xrpld/nix-ubuntu:sha-7b9d553
steps:
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
@@ -92,17 +99,15 @@ jobs:
issues: write
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- name: Create an issue
uses: ./.github/actions/create-issue
env:
GH_TOKEN: ${{ github.token }}
uses: XRPLF/actions/create-issue@2b8bc36af85b88bca0dd7bfac2e2dc05f94ad712
with:
labels: "compatibility,bug"
title: "Proposed libXRPL check failed"
body: >
Clio build or tests failed against `libXRPL ${{ github.event.client_payload.conan_ref }}`.
PR: ${{ github.event.client_payload.pr_url }}
Workflow run: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}/
labels: "compatibility,bug"
assignees: "godexsoft,kuznetsss,mathbunnyru"

View File

@@ -7,4 +7,4 @@ on:
jobs:
check_title:
uses: XRPLF/actions/.github/workflows/check-pr-title.yml@a5d8dd35be543365e90a11358447130c8763871d
uses: XRPLF/actions/.github/workflows/check-pr-title.yml@cba1f0891650baf1a9c88624dc2d72573be2eb81

View File

@@ -1,16 +1,22 @@
name: Clang-tidy check
name: Run clang-tidy on files
on:
push:
branches: [develop]
schedule:
- cron: "0 9 * * 1-5"
workflow_dispatch:
pull_request:
branches: [develop]
paths:
- .github/workflows/clang-tidy.yml
- CMakeLists.txt
- conanfile.py
- conan.lock
- "cmake/**"
- "src/**"
- "tests/**"
- "benchmarks/**"
- .clang_tidy
concurrency:
@@ -19,19 +25,33 @@ concurrency:
cancel-in-progress: true
env:
CONAN_PROFILE: clang
LLVM_TOOLS_VERSION: 20
BUILD_DIR: build
BUILD_TYPE: Debug # Debug so that ASSERTS and such participate in clang-tidy check
OUTPUT_FILE: clang-tidy-output.txt
DIFF_FILE: clang-tidy-git-diff.txt
ISSUE_FILE: clang-tidy-issue.md
COMPILER: clang
defaults:
run:
shell: bash
jobs:
clang_tidy:
if: github.event_name != 'push' || contains(github.event.head_commit.message, 'clang-tidy auto fixes')
determine-files:
if: ${{ github.event_name == 'pull_request' }}
permissions:
contents: read
uses: XRPLF/actions/.github/workflows/determine-tidy-files.yml@224f3c48d3014d082a1129237b8291ff0b0a331f
run-clang-tidy:
name: Run clang tidy
needs: [determine-files]
if: ${{ always() && !cancelled() && (github.event_name != 'pull_request' || needs.determine-files.outputs.cpp_changed_files != '' || needs.determine-files.outputs.clang_tidy_config_changed == 'true') }}
runs-on: heavy
container:
image: ghcr.io/xrplf/clio-ci:14342e087ceb8b593027198bf9ef06a43833c696
image: ghcr.io/xrplf/xrpld/nix-ubuntu:sha-7b9d553
permissions:
contents: write
@@ -39,38 +59,61 @@ jobs:
pull-requests: write
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
fetch-depth: 0
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- name: Prepare runner
uses: XRPLF/actions/prepare-runner@90f11ee655d1687824fb8793db770477d52afbab
with:
enable_ccache: false
- name: Print build environment
uses: XRPLF/actions/print-build-env@59dec886e4afb05a1724443af08baccbc045b574
- name: Set compiler environment
uses: ./.github/actions/set-compiler-env
with:
compiler: ${{ env.COMPILER }}
- name: Setup conan
run: conan/init.sh
- name: Run conan
uses: ./.github/actions/conan
with:
conan_profile: ${{ env.CONAN_PROFILE }}
build_dir: ${{ env.BUILD_DIR }}
build_type: ${{ env.BUILD_TYPE }}
- name: Run CMake
uses: ./.github/actions/cmake
with:
conan_profile: ${{ env.CONAN_PROFILE }}
build_dir: ${{ env.BUILD_DIR }}
build_type: ${{ env.BUILD_TYPE }}
- name: Get number of processors
uses: XRPLF/actions/get-nproc@cf0433aa74563aead044a1e395610c96d65a37cf
id: nproc
- name: Run clang-tidy (several times)
- name: Run clang tidy
continue-on-error: true
id: clang_tidy
id: run_clang_tidy
env:
TARGETS: ${{ (needs.determine-files.outputs.clang_tidy_config_changed != 'true' && github.event_name == 'pull_request') && needs.determine-files.outputs.cpp_changed_files || 'benchmarks src tests' }}
run: |
# We run clang-tidy several times, because some fixes may enable new fixes in subsequent runs.
CLANG_TIDY_COMMAND="run-clang-tidy-${{ env.LLVM_TOOLS_VERSION }} -p build -j ${{ steps.nproc.outputs.nproc }} -fix -quiet"
${CLANG_TIDY_COMMAND} ||
${CLANG_TIDY_COMMAND} ||
${CLANG_TIDY_COMMAND}
set -o pipefail
run-clang-tidy -j ${{ steps.nproc.outputs.nproc }} -p "${BUILD_DIR}" -quiet -fix -allow-no-checks ${TARGETS} 2>&1 | tee "${OUTPUT_FILE}"
- name: Print errors
if: ${{ steps.run_clang_tidy.outcome != 'success' }}
run: |
sed '/error\||/!d' "${OUTPUT_FILE}"
- name: Upload clang-tidy output
if: ${{ github.event.repository.visibility == 'public' && steps.run_clang_tidy.outcome != 'success' }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
path: ${{ env.OUTPUT_FILE }}
archive: false
retention-days: 30
- name: Check for changes
id: files_changed
@@ -78,24 +121,76 @@ jobs:
run: |
git diff --exit-code
- name: Fix local includes and clang-format style
- name: Fix style
if: ${{ steps.files_changed.outcome != 'success' }}
run: |
pre-commit run --all-files fix-local-includes || true
pre-commit run --all-files clang-format || true
pre-commit run --all-files || true
- name: Create an issue
if: ${{ (steps.clang_tidy.outcome != 'success' || steps.files_changed.outcome != 'success') && github.event_name != 'pull_request' }}
id: create_issue
uses: ./.github/actions/create-issue
env:
GH_TOKEN: ${{ github.token }}
- name: Generate git diff
if: ${{ steps.files_changed.outcome != 'success' }}
run: |
git diff | tee "${DIFF_FILE}"
- name: Upload clang-tidy diff output
if: ${{ github.event.repository.visibility == 'public' && steps.files_changed.outcome != 'success' }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
title: "Clang-tidy found bugs in code 🐛"
body: >
Clang-tidy found issues in the code:
path: ${{ env.DIFF_FILE }}
archive: false
retention-days: 30
List of the issues found: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}/
- name: Write issue header
if: ${{ steps.run_clang_tidy.outcome != 'success' }}
run: |
cat >"${ISSUE_FILE}" <<EOF
## Clang-tidy Check Failed
### Clang-tidy Output:
\`\`\`
EOF
- name: Append clang-tidy output to issue body (filter for errors and warnings)
if: ${{ steps.run_clang_tidy.outcome != 'success' }}
run: |
if [ -f "${OUTPUT_FILE}" ]; then
# Extract lines containing 'error:', 'warning:', or 'note:'
grep -E '(error:|warning:|note:)' "${OUTPUT_FILE}" >filtered-output.txt || true
# If filtered output is empty, use original (might be a different error format)
if [ ! -s filtered-output.txt ]; then
cp "${OUTPUT_FILE}" filtered-output.txt
fi
# Truncate if too large
head -c 60000 filtered-output.txt >>"${ISSUE_FILE}"
if [ "$(wc -c <filtered-output.txt)" -gt 60000 ]; then
echo "" >>"${ISSUE_FILE}"
echo "... (output truncated, see artifacts for full output)" >>"${ISSUE_FILE}"
fi
rm filtered-output.txt
else
echo "No output file found" >>"${ISSUE_FILE}"
fi
- name: Append issue footer
if: ${{ steps.run_clang_tidy.outcome != 'success' }}
run: |
cat >>"${ISSUE_FILE}" <<EOF
\`\`\`
---
*This issue was automatically created by the clang-tidy workflow.*
EOF
- name: Create issue
if: ${{ steps.run_clang_tidy.outcome != 'success' && github.event_name != 'pull_request' }}
uses: XRPLF/actions/create-issue@2b8bc36af85b88bca0dd7bfac2e2dc05f94ad712
with:
title: "Clang-tidy check failed"
body_file: ${{ env.ISSUE_FILE }}
labels: "bug"
assignees: "godexsoft,mathbunnyru"
- uses: crazy-max/ghaction-import-gpg@2dc316deee8e90f13e1a351ab510b4d5bc0c82cd # v7.0.0
if: ${{ steps.files_changed.outcome != 'success' && github.event_name != 'pull_request' }}
@@ -118,9 +213,14 @@ jobs:
branch-suffix: timestamp
delete-branch: true
title: "style: clang-tidy auto fixes"
body: "Fixes #${{ steps.create_issue.outputs.created_issue_id }}. Please review and commit clang-tidy fixes."
reviewers: "godexsoft,kuznetsss,PeterChen13579,mathbunnyru"
body: >
Fixes #${{ steps.create_issue.outputs.issue_number }}.
Please review and commit clang-tidy fixes.
reviewers: "godexsoft,kuznetsss,mathbunnyru"
- name: Fail the job
if: ${{ steps.clang_tidy.outcome != 'success' || steps.files_changed.outcome != 'success' }}
run: exit 1
if: ${{ steps.run_clang_tidy.outcome != 'success' }}
run: |
echo "Clang-tidy check failed!"
exit 1

View File

@@ -18,11 +18,11 @@ jobs:
build:
runs-on: ubuntu-latest
container:
image: ghcr.io/xrplf/clio-ci:14342e087ceb8b593027198bf9ef06a43833c696
image: ghcr.io/xrplf/xrpld/nix-ubuntu:sha-7b9d553
steps:
- name: Checkout
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
lfs: true
@@ -31,6 +31,9 @@ jobs:
with:
enable_ccache: false
- name: Print build environment
uses: XRPLF/actions/print-build-env@59dec886e4afb05a1724443af08baccbc045b574
- name: Create build directory
run: mkdir build_docs
@@ -43,6 +46,7 @@ jobs:
run: cmake --build . --target docs
- name: Setup Pages
if: ${{ github.repository == 'XRPLF/clio' && github.event_name == 'push' }}
uses: actions/configure-pages@45bfe0192ca1faeb007ade9deae92b16b8254a0d # v6.0.0
- name: Upload artifact
@@ -52,6 +56,7 @@ jobs:
name: docs-develop
deploy:
if: ${{ github.repository == 'XRPLF/clio' && github.event_name == 'push' }}
needs: build
permissions:
pages: write

View File

@@ -37,7 +37,7 @@ jobs:
- name: Get current date
id: get_date
run: |
echo "date=$(date +'%Y%m%d')" >> $GITHUB_OUTPUT
echo "date=$(date +'%Y%m%d')" >>$GITHUB_OUTPUT
build-and-test:
name: Build and Test
@@ -48,32 +48,29 @@ jobs:
matrix:
include:
- os: macos15
conan_profile: apple-clang
compiler: apple-clang
build_type: Release
static: false
- os: heavy
conan_profile: gcc
compiler: gcc
build_type: Release
static: true
container: '{ "image": "ghcr.io/xrplf/clio-ci:14342e087ceb8b593027198bf9ef06a43833c696" }'
container: '{ "image": "ghcr.io/xrplf/xrpld/nix-ubuntu:sha-7b9d553" }'
- os: heavy
conan_profile: gcc
compiler: gcc
build_type: Debug
static: true
container: '{ "image": "ghcr.io/xrplf/clio-ci:14342e087ceb8b593027198bf9ef06a43833c696" }'
container: '{ "image": "ghcr.io/xrplf/xrpld/nix-ubuntu:sha-7b9d553" }'
- os: heavy
conan_profile: gcc.ubsan
compiler: gcc
sanitizers: undefinedbehavior
build_type: Release
static: false
container: '{ "image": "ghcr.io/xrplf/clio-ci:14342e087ceb8b593027198bf9ef06a43833c696" }'
container: '{ "image": "ghcr.io/xrplf/xrpld/nix-ubuntu:sha-7b9d553" }'
uses: ./.github/workflows/reusable-build-test.yml
with:
runs_on: ${{ matrix.os }}
container: ${{ matrix.container }}
conan_profile: ${{ matrix.conan_profile }}
compiler: ${{ matrix.compiler }}
sanitizers: ${{ matrix.sanitizers }}
build_type: ${{ matrix.build_type }}
static: ${{ matrix.static }}
run_unit_tests: true
run_integration_tests: true
upload_clio_server: true
@@ -88,13 +85,12 @@ jobs:
uses: ./.github/workflows/reusable-build.yml
with:
runs_on: heavy
container: '{ "image": "ghcr.io/xrplf/clio-ci:14342e087ceb8b593027198bf9ef06a43833c696" }'
conan_profile: gcc
container: '{ "image": "ghcr.io/xrplf/xrpld/nix-ubuntu:sha-7b9d553" }'
compiler: gcc
build_type: Release
download_ccache: false
upload_ccache: false
code_coverage: false
static: true
upload_clio_server: false
package: true
version: nightly-${{ needs.get_date.outputs.date }}
@@ -110,23 +106,20 @@ jobs:
matrix:
include:
- os: heavy
conan_profile: clang
container: '{ "image": "ghcr.io/xrplf/clio-ci:14342e087ceb8b593027198bf9ef06a43833c696" }'
static: true
compiler: clang
container: '{ "image": "ghcr.io/xrplf/xrpld/nix-ubuntu:sha-7b9d553" }'
- os: macos15
conan_profile: apple-clang
compiler: apple-clang
container: ""
static: false
uses: ./.github/workflows/reusable-build.yml
with:
runs_on: ${{ matrix.os }}
container: ${{ matrix.container }}
conan_profile: ${{ matrix.conan_profile }}
compiler: ${{ matrix.compiler }}
build_type: Release
download_ccache: false
upload_ccache: false
code_coverage: false
static: ${{ matrix.static }}
upload_clio_server: false
targets: all
analyze_build_time: true
@@ -157,7 +150,7 @@ jobs:
type=raw,value=${{ github.sha }}
artifact_name: clio_server_Linux_Release_gcc
strip_binary: true
publish_image: ${{ github.event_name != 'pull_request' }}
push_image: ${{ github.repository == 'XRPLF/clio' && github.event_name != 'pull_request' }}
create_issue_on_failure:
needs: [build-and-test, nightly_release, build_and_publish_docker_image]
@@ -169,15 +162,12 @@ jobs:
issues: write
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- name: Create an issue
uses: ./.github/actions/create-issue
env:
GH_TOKEN: ${{ github.token }}
uses: XRPLF/actions/create-issue@2b8bc36af85b88bca0dd7bfac2e2dc05f94ad712
with:
title: "Nightly release failed 🌙"
body: >
Nightly release failed:
Workflow: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}/
body: ""
labels: "bug"
assignees: "godexsoft,kuznetsss,mathbunnyru"

View File

@@ -12,11 +12,11 @@ on:
jobs:
auto-update:
uses: XRPLF/actions/.github/workflows/pre-commit-autoupdate.yml@a249154199805d6809359fe99fa8ba09dd804e3d
uses: XRPLF/actions/.github/workflows/pre-commit-autoupdate.yml@5e942d61bf32f7557a7c159cfac4712a687b3e3a
with:
sign_commit: true
committer: "Clio CI <skuznetsov@ripple.com>"
reviewers: "godexsoft,kuznetsss,PeterChen13579,mathbunnyru"
reviewers: "godexsoft,kuznetsss,mathbunnyru"
secrets:
GPG_PRIVATE_KEY: ${{ secrets.ACTIONS_GPG_PRIVATE_KEY }}
GPG_PASSPHRASE: ${{ secrets.ACTIONS_GPG_PASSPHRASE }}

View File

@@ -8,7 +8,7 @@ on:
jobs:
run-hooks:
uses: XRPLF/actions/.github/workflows/pre-commit.yml@9307df762265e15c745ddcdb38a581c989f7f349
uses: XRPLF/actions/.github/workflows/pre-commit.yml@cba1f0891650baf1a9c88624dc2d72573be2eb81
with:
runs_on: heavy
container: '{ "image": "ghcr.io/xrplf/clio-pre-commit:14342e087ceb8b593027198bf9ef06a43833c696" }'

View File

@@ -22,22 +22,19 @@ jobs:
matrix:
include:
- os: macos15
conan_profile: apple-clang
compiler: apple-clang
build_type: Release
static: false
- os: heavy
conan_profile: gcc
compiler: gcc
build_type: Release
static: true
container: '{ "image": "ghcr.io/xrplf/clio-ci:14342e087ceb8b593027198bf9ef06a43833c696" }'
container: '{ "image": "ghcr.io/xrplf/xrpld/nix-ubuntu:sha-7b9d553" }'
uses: ./.github/workflows/reusable-build-test.yml
with:
runs_on: ${{ matrix.os }}
container: ${{ matrix.container }}
conan_profile: ${{ matrix.conan_profile }}
compiler: ${{ matrix.compiler }}
build_type: ${{ matrix.build_type }}
static: ${{ matrix.static }}
run_unit_tests: true
run_integration_tests: true
upload_clio_server: true
@@ -51,13 +48,12 @@ jobs:
uses: ./.github/workflows/reusable-build.yml
with:
runs_on: heavy
container: '{ "image": "ghcr.io/xrplf/clio-ci:14342e087ceb8b593027198bf9ef06a43833c696" }'
conan_profile: gcc
container: '{ "image": "ghcr.io/xrplf/xrpld/nix-ubuntu:sha-7b9d553" }'
compiler: gcc
build_type: Release
download_ccache: false
upload_ccache: false
code_coverage: false
static: true
upload_clio_server: false
package: true
version: ${{ github.event_name == 'push' && github.ref_name || '' }}

View File

@@ -13,11 +13,17 @@ on:
required: true
type: string
conan_profile:
description: Conan profile to use
compiler:
description: 'Compiler to build with ("gcc", "clang" or "apple-clang")'
required: true
type: string
sanitizers:
description: 'Sanitizers to enable ("address", "thread", "undefinedbehavior" or empty)'
required: false
type: string
default: ""
build_type:
description: Build type
required: true
@@ -35,12 +41,6 @@ on:
type: boolean
default: false
static:
description: Whether to build static binaries
required: true
type: boolean
default: true
run_unit_tests:
description: Whether to run unit tests
required: true
@@ -81,12 +81,12 @@ jobs:
with:
runs_on: ${{ inputs.runs_on }}
container: ${{ inputs.container }}
conan_profile: ${{ inputs.conan_profile }}
compiler: ${{ inputs.compiler }}
sanitizers: ${{ inputs.sanitizers }}
build_type: ${{ inputs.build_type }}
download_ccache: ${{ inputs.download_ccache }}
upload_ccache: ${{ inputs.upload_ccache }}
code_coverage: false
static: ${{ inputs.static }}
upload_clio_server: ${{ inputs.upload_clio_server }}
targets: ${{ inputs.targets }}
analyze_build_time: false
@@ -99,7 +99,8 @@ jobs:
with:
runs_on: ${{ inputs.runs_on }}
container: ${{ inputs.container }}
conan_profile: ${{ inputs.conan_profile }}
compiler: ${{ inputs.compiler }}
sanitizers: ${{ inputs.sanitizers }}
build_type: ${{ inputs.build_type }}
run_unit_tests: ${{ inputs.run_unit_tests }}
run_integration_tests: ${{ inputs.run_integration_tests }}

View File

@@ -13,11 +13,17 @@ on:
required: true
type: string
conan_profile:
description: Conan profile to use
compiler:
description: 'Compiler to build with ("gcc", "clang" or "apple-clang")'
required: true
type: string
sanitizers:
description: 'Sanitizers to enable ("address", "thread", "undefinedbehavior" or empty)'
required: false
type: string
default: ""
build_type:
description: Build type
required: true
@@ -40,11 +46,6 @@ on:
required: true
type: boolean
static:
description: Whether to build static binaries
required: true
type: boolean
upload_clio_server:
description: Whether to upload clio_server
required: true
@@ -90,7 +91,7 @@ jobs:
if: ${{ runner.os == 'macOS' }}
uses: XRPLF/actions/cleanup-workspace@c7d9ce5ebb03c752a354889ecd870cadfc2b1cd4
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
fetch-depth: 0
@@ -99,40 +100,47 @@ jobs:
with:
enable_ccache: ${{ inputs.download_ccache }}
- name: Setup conan on macOS
if: ${{ runner.os == 'macOS' }}
run: ./.github/scripts/conan/init.sh
- name: Print build environment
uses: XRPLF/actions/print-build-env@59dec886e4afb05a1724443af08baccbc045b574
- name: Generate cache key
uses: ./.github/actions/cache-key
id: cache_key
- name: Set compiler environment
if: ${{ runner.os == 'Linux' }}
uses: ./.github/actions/set-compiler-env
with:
compiler: ${{ inputs.compiler }}
- name: Setup conan
run: conan/init.sh
- name: Generate build identifier
uses: ./.github/actions/build-identifier
id: build_identifier
with:
conan_profile: ${{ inputs.conan_profile }}
build_type: ${{ inputs.build_type }}
compiler: ${{ inputs.compiler }}
code_coverage: ${{ inputs.code_coverage }}
sanitizers: ${{ inputs.sanitizers }}
- name: Restore ccache cache
if: ${{ inputs.download_ccache && github.ref != 'refs/heads/develop' }}
uses: actions/cache/restore@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ env.CCACHE_DIR }}
key: ${{ steps.cache_key.outputs.key }}
key: ${{ steps.build_identifier.outputs.cache_key }}
restore-keys: |
${{ steps.cache_key.outputs.restore_keys }}
${{ steps.build_identifier.outputs.build_identifier }}
- name: Run conan
uses: ./.github/actions/conan
with:
conan_profile: ${{ inputs.conan_profile }}
build_type: ${{ inputs.build_type }}
sanitizers: ${{ inputs.sanitizers }}
- name: Run CMake
uses: ./.github/actions/cmake
with:
conan_profile: ${{ inputs.conan_profile }}
build_type: ${{ inputs.build_type }}
code_coverage: ${{ inputs.code_coverage }}
static: ${{ inputs.static }}
time_trace: ${{ inputs.analyze_build_time }}
package: ${{ inputs.package }}
version: ${{ inputs.version }}
@@ -146,14 +154,14 @@ jobs:
if: ${{ inputs.analyze_build_time }}
run: |
ClangBuildAnalyzer --all build/ build_time_report.bin
ClangBuildAnalyzer --analyze build_time_report.bin > build_time_report.txt
ClangBuildAnalyzer --analyze build_time_report.bin >build_time_report.txt
cat build_time_report.txt
- name: Upload build time analyze report
if: ${{ inputs.analyze_build_time }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: build_time_report_${{ runner.os }}_${{ inputs.build_type }}_${{ inputs.conan_profile }}
name: build_time_report_${{ steps.build_identifier.outputs.build_identifier }}
path: build_time_report.txt
- name: Show ccache's statistics and zero it
@@ -167,42 +175,42 @@ jobs:
uses: actions/cache/save@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5.0.5
with:
path: ${{ env.CCACHE_DIR }}
key: ${{ steps.cache_key.outputs.key }}
key: ${{ steps.build_identifier.outputs.cache_key }}
- name: Strip unit_tests
if: ${{ !endsWith(inputs.conan_profile, 'san') && !inputs.code_coverage && !inputs.analyze_build_time }}
if: ${{ inputs.sanitizers == '' && !inputs.code_coverage && !inputs.analyze_build_time }}
run: strip build/clio_tests
- name: Strip integration_tests
if: ${{ !endsWith(inputs.conan_profile, 'san') && !inputs.code_coverage && !inputs.analyze_build_time }}
if: ${{ inputs.sanitizers == '' && !inputs.code_coverage && !inputs.analyze_build_time }}
run: strip build/clio_integration_tests
- name: Upload clio_server
if: ${{ inputs.upload_clio_server && !inputs.code_coverage && !inputs.analyze_build_time }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: clio_server_${{ runner.os }}_${{ inputs.build_type }}_${{ inputs.conan_profile }}
name: clio_server_${{ steps.build_identifier.outputs.build_identifier }}
path: build/clio_server
- name: Upload clio_tests
if: ${{ !inputs.code_coverage && !inputs.analyze_build_time && !inputs.package }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: clio_tests_${{ runner.os }}_${{ inputs.build_type }}_${{ inputs.conan_profile }}
name: clio_tests_${{ steps.build_identifier.outputs.build_identifier }}
path: build/clio_tests
- name: Upload clio_integration_tests
if: ${{ !inputs.code_coverage && !inputs.analyze_build_time && !inputs.package }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: clio_integration_tests_${{ runner.os }}_${{ inputs.build_type }}_${{ inputs.conan_profile }}
name: clio_integration_tests_${{ steps.build_identifier.outputs.build_identifier }}
path: build/clio_integration_tests
- name: Upload Clio Linux package
if: ${{ inputs.package }}
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: clio_deb_package_${{ runner.os }}_${{ inputs.build_type }}_${{ inputs.conan_profile }}
name: clio_deb_package_${{ steps.build_identifier.outputs.build_identifier }}
path: build/*.deb
# This is run as part of the build job, because it requires the following:
@@ -224,13 +232,13 @@ jobs:
set -e
EXPECTED_VERSION="clio-${INPUT_VERSION}"
if [[ "${BUILD_TYPE}" == "Debug" ]]; then
EXPECTED_VERSION="${EXPECTED_VERSION}+DEBUG"
EXPECTED_VERSION="${EXPECTED_VERSION}+DEBUG"
fi
actual_version=$(./build/clio_server --version | head -n 1)
if [[ "${actual_version}" != "${EXPECTED_VERSION}" ]]; then
echo "Expected version '${EXPECTED_VERSION}', but got '${actual_version}'"
exit 1
echo "Expected version '${EXPECTED_VERSION}', but got '${actual_version}'"
exit 1
fi
# `codecov/codecov-action` will rerun `gcov` if it's available and build directory is present

View File

@@ -46,7 +46,7 @@ jobs:
release:
runs-on: heavy
container:
image: ghcr.io/xrplf/clio-ci:14342e087ceb8b593027198bf9ef06a43833c696
image: ghcr.io/xrplf/xrpld/nix-ubuntu:sha-7b9d553
env:
GH_REPO: ${{ github.repository }}
GH_TOKEN: ${{ github.token }}
@@ -55,7 +55,7 @@ jobs:
contents: write
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
fetch-depth: 0
@@ -81,9 +81,9 @@ jobs:
env:
RELEASE_HEADER: ${{ inputs.header }}
run: |
echo "# Release notes" > "${RUNNER_TEMP}/release_notes.md"
echo "" >> "${RUNNER_TEMP}/release_notes.md"
printf '%s\n' "${RELEASE_HEADER}" >> "${RUNNER_TEMP}/release_notes.md"
echo "# Release notes" >"${RUNNER_TEMP}/release_notes.md"
echo "" >>"${RUNNER_TEMP}/release_notes.md"
printf '%s\n' "${RELEASE_HEADER}" >>"${RUNNER_TEMP}/release_notes.md"
- name: Generate changelog
if: ${{ inputs.generate_changelog }}
@@ -91,7 +91,7 @@ jobs:
LAST_TAG="$(gh release view --json tagName -q .tagName --repo XRPLF/clio)"
LAST_TAG_COMMIT="$(git rev-parse $LAST_TAG)"
BASE_COMMIT="$(git merge-base HEAD $LAST_TAG_COMMIT)"
git-cliff "${BASE_COMMIT}..HEAD" --ignore-tags "nightly|-b|-rc" >> "${RUNNER_TEMP}/release_notes.md"
git-cliff "${BASE_COMMIT}..HEAD" --ignore-tags "nightly|-b|-rc" >>"${RUNNER_TEMP}/release_notes.md"
- name: Upload release notes
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
@@ -106,10 +106,10 @@ jobs:
run: |
RELEASES_TO_DELETE=$(gh release list --limit 50 --repo "${GH_REPO}" | grep -E "${DELETE_PATTERN}" | awk -F'\t' '{print $3}' || true)
if [ -n "$RELEASES_TO_DELETE" ]; then
for RELEASE in $RELEASES_TO_DELETE; do
echo "Deleting release: $RELEASE"
gh release delete "$RELEASE" --repo "${GH_REPO}" --yes --cleanup-tag
done
for RELEASE in $RELEASES_TO_DELETE; do
echo "Deleting release: $RELEASE"
gh release delete "$RELEASE" --repo "${GH_REPO}" --yes --cleanup-tag
done
fi
- name: Publish release
@@ -121,9 +121,9 @@ jobs:
DRAFT_OPTION: ${{ inputs.draft && '--draft' || '' }}
run: |
gh release create "${RELEASE_VERSION}" \
${PRERELEASE_OPTION} \
--title "${RELEASE_TITLE}" \
--target "${GITHUB_SHA}" \
${DRAFT_OPTION} \
--notes-file "${RUNNER_TEMP}/release_notes.md" \
./release_artifacts/clio_*
${PRERELEASE_OPTION} \
--title "${RELEASE_TITLE}" \
--target "${GITHUB_SHA}" \
${DRAFT_OPTION} \
--notes-file "${RUNNER_TEMP}/release_notes.md" \
./release_artifacts/clio_*

View File

@@ -13,11 +13,17 @@ on:
required: true
type: string
conan_profile:
description: Conan profile to use
compiler:
description: 'Compiler the binaries were built with ("gcc", "clang" or "apple-clang")'
required: true
type: string
sanitizers:
description: 'Sanitizers the binaries were built with ("address", "thread", "undefinedbehavior" or empty)'
required: false
type: string
default: ""
build_type:
description: Build type
required: true
@@ -50,13 +56,23 @@ jobs:
if: ${{ runner.os == 'macOS' }}
uses: XRPLF/actions/cleanup-workspace@c7d9ce5ebb03c752a354889ecd870cadfc2b1cd4
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
fetch-depth: 0
- name: Generate build identifier
uses: ./.github/actions/build-identifier
id: build_identifier
with:
build_type: ${{ inputs.build_type }}
compiler: ${{ inputs.compiler }}
# code_coverage is run inside build environment
code_coverage: false
sanitizers: ${{ inputs.sanitizers }}
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: clio_tests_${{ runner.os }}_${{ inputs.build_type }}_${{ inputs.conan_profile }}
name: clio_tests_${{ steps.build_identifier.outputs.build_identifier }}
- name: Make clio_tests executable
run: chmod +x ./clio_tests
@@ -67,17 +83,13 @@ jobs:
run: ./clio_tests
- name: Create an issue
if: ${{ steps.run_clio_tests.outcome == 'failure' && endsWith(inputs.conan_profile, 'san') }}
uses: ./.github/actions/create-issue
env:
GH_TOKEN: ${{ github.token }}
if: ${{ steps.run_clio_tests.outcome == 'failure' && inputs.sanitizers != '' }}
uses: XRPLF/actions/create-issue@2b8bc36af85b88bca0dd7bfac2e2dc05f94ad712
with:
title: "[${{ inputs.compiler }} ${{ inputs.sanitizers }}] reported issues"
body: "Clio tests failed one or more sanitizers checks when built with `${{ inputs.compiler }}` and the `${{ inputs.sanitizers }}` sanitizers."
labels: "bug"
title: "[${{ inputs.conan_profile }}] reported issues"
body: >
Clio tests failed one or more sanitizer checks when built with `${{ inputs.conan_profile }}`.
Workflow: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}/
assignees: "godexsoft,kuznetsss,mathbunnyru"
- name: Fail the job if clio_tests failed
if: ${{ steps.run_clio_tests.outcome == 'failure' }}
@@ -104,6 +116,20 @@ jobs:
if: ${{ runner.os == 'macOS' }}
uses: XRPLF/actions/cleanup-workspace@c7d9ce5ebb03c752a354889ecd870cadfc2b1cd4
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
fetch-depth: 0
- name: Generate build identifier
uses: ./.github/actions/build-identifier
id: build_identifier
with:
build_type: ${{ inputs.build_type }}
compiler: ${{ inputs.compiler }}
# code_coverage is run inside build environment
code_coverage: false
sanitizers: ${{ inputs.sanitizers }}
- name: Delete and start colima (macOS)
# This is a temporary workaround for colima issues on macOS runners
if: ${{ runner.os == 'macOS' }}
@@ -111,32 +137,38 @@ jobs:
colima delete --force
colima start
- name: Remove leftover scylladb container (macOS)
# A previous run that didn't clean up (e.g. a cancelled job) can leave the
# container behind, otherwise the name is still taken on the persistent runner.
if: ${{ runner.os == 'macOS' }}
run: docker rm --force scylladb || true
- name: Spin up scylladb (macOS)
if: ${{ runner.os == 'macOS' }}
timeout-minutes: 1
run: |
docker run \
--detach \
--name scylladb \
--health-cmd "cqlsh -e 'describe cluster'" \
--health-interval 10s \
--health-timeout 5s \
--health-retries 5 \
--publish 9042:9042 \
--memory 16G \
scylladb/scylla
--detach \
--name scylladb \
--health-cmd "cqlsh -e 'describe cluster'" \
--health-interval 10s \
--health-timeout 5s \
--health-retries 5 \
--publish 9042:9042 \
--memory 16G \
scylladb/scylla
- name: Wait for scylladb container to be healthy (macOS)
if: ${{ runner.os == 'macOS' }}
timeout-minutes: 1
run: |
until [ "$(docker inspect -f '{{.State.Health.Status}}' scylladb)" == "healthy" ]; do
sleep 1
sleep 1
done
- uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: clio_integration_tests_${{ runner.os }}_${{ inputs.build_type }}_${{ inputs.conan_profile }}
name: clio_integration_tests_${{ steps.build_identifier.outputs.build_identifier }}
- name: Run clio_integration_tests
run: |

View File

@@ -16,7 +16,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
with:
fetch-depth: 0
@@ -27,8 +27,8 @@ jobs:
path: build
- name: Upload coverage report
if: ${{ hashFiles('build/coverage_report.xml') != '' }}
uses: codecov/codecov-action@57e3a136b779b570ffcdbf80b3bdc90e7fab3de2 # v6.0.0
if: ${{ github.repository == 'XRPLF/clio' && hashFiles('build/coverage_report.xml') != '' }}
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
with:
files: build/coverage_report.xml
fail_ci_if_error: true

View File

@@ -14,13 +14,12 @@ on:
- ".github/actions/**"
- "!.github/actions/build-docker-image/**"
- "!.github/actions/create-issue/**"
- CMakeLists.txt
- conanfile.py
- conan.lock
- "cmake/**"
# We don't run sanitizer on code change, because it takes too long
# We don't run sanitizers on code change, because it takes too long
# - "src/**"
# - "tests/**"
@@ -37,18 +36,18 @@ jobs:
fail-fast: false
matrix:
compiler: [gcc, clang]
sanitizer_ext: [.asan, .tsan, .ubsan]
sanitizers: [address, thread, undefinedbehavior]
build_type: [Release, Debug]
uses: ./.github/workflows/reusable-build-test.yml
with:
runs_on: heavy
container: '{ "image": "ghcr.io/xrplf/clio-ci:14342e087ceb8b593027198bf9ef06a43833c696" }'
container: '{ "image": "ghcr.io/xrplf/xrpld/nix-ubuntu:sha-7b9d553" }'
download_ccache: false
upload_ccache: false
conan_profile: ${{ matrix.compiler }}${{ matrix.sanitizer_ext }}
compiler: ${{ matrix.compiler }}
sanitizers: ${{ matrix.sanitizers }}
build_type: ${{ matrix.build_type }}
static: false
run_unit_tests: true
run_integration_tests: false
upload_clio_server: false

View File

@@ -1,364 +0,0 @@
name: Update CI docker image
on:
pull_request:
paths:
- .github/workflows/update-docker-ci.yml
- ".github/actions/build-docker-image/**"
- "docker/**"
- "!docker/clio/**"
- "!docker/develop/**"
push:
branches: [develop]
paths:
- .github/workflows/update-docker-ci.yml
- ".github/actions/build-docker-image/**"
- "docker/**"
- "!docker/clio/**"
- "!docker/develop/**"
workflow_dispatch:
concurrency:
# Only matches runs for the current workflow - matches against branch & tags
group: ${{ github.workflow }}-${{ github.ref }}
# We want to execute all builds sequentially in develop
cancel-in-progress: false
env:
CLANG_MAJOR_VERSION: 19
GCC_MAJOR_VERSION: 15
GCC_VERSION: 15.2.0
defaults:
run:
shell: bash
jobs:
repo:
name: Calculate repo name
runs-on: ubuntu-latest
outputs:
GHCR_REPO: ${{ steps.set-ghcr-repo.outputs.GHCR_REPO }}
steps:
- name: Set GHCR_REPO
id: set-ghcr-repo
run: |
echo "GHCR_REPO=$(echo ghcr.io/${{ github.repository_owner }} | tr '[:upper:]' '[:lower:]')" >> ${GITHUB_OUTPUT}
gcc-amd64:
name: Build and push GCC docker image (amd64)
runs-on: heavy
needs: repo
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Get changed files
id: changed-files
uses: tj-actions/changed-files@9426d40962ed5378910ee2e21d5f8c6fcbf2dd96 # v47.0.6
with:
files: "docker/compilers/gcc/**"
- uses: ./.github/actions/build-docker-image
if: ${{ steps.changed-files.outputs.any_changed == 'true' }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
DOCKERHUB_PW: ${{ secrets.DOCKERHUB_PW }}
with:
images: |
${{ needs.repo.outputs.GHCR_REPO }}/clio-gcc
${{ github.repository_owner == 'XRPLF' && 'rippleci/clio_gcc' || '' }}
push_image: ${{ github.event_name != 'pull_request' }}
directory: docker/compilers/gcc
tags: |
type=raw,value=amd64-latest
type=raw,value=amd64-${{ env.GCC_MAJOR_VERSION }}
type=raw,value=amd64-${{ env.GCC_VERSION }}
type=raw,value=amd64-${{ github.sha }}
platforms: linux/amd64
build_args: |
GCC_MAJOR_VERSION=${{ env.GCC_MAJOR_VERSION }}
GCC_VERSION=${{ env.GCC_VERSION }}
dockerhub_repo: ${{ github.repository_owner == 'XRPLF' && 'rippleci/clio_gcc' || '' }}
dockerhub_description: GCC compiler for XRPLF/clio.
gcc-arm64:
name: Build and push GCC docker image (arm64)
runs-on: heavy-arm64
needs: repo
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Get changed files
id: changed-files
uses: tj-actions/changed-files@9426d40962ed5378910ee2e21d5f8c6fcbf2dd96 # v47.0.6
with:
files: "docker/compilers/gcc/**"
- uses: ./.github/actions/build-docker-image
if: ${{ steps.changed-files.outputs.any_changed == 'true' }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
DOCKERHUB_PW: ${{ secrets.DOCKERHUB_PW }}
with:
images: |
${{ needs.repo.outputs.GHCR_REPO }}/clio-gcc
${{ github.repository_owner == 'XRPLF' && 'rippleci/clio_gcc' || '' }}
push_image: ${{ github.event_name != 'pull_request' }}
directory: docker/compilers/gcc
tags: |
type=raw,value=arm64-latest
type=raw,value=arm64-${{ env.GCC_MAJOR_VERSION }}
type=raw,value=arm64-${{ env.GCC_VERSION }}
type=raw,value=arm64-${{ github.sha }}
platforms: linux/arm64
build_args: |
GCC_MAJOR_VERSION=${{ env.GCC_MAJOR_VERSION }}
GCC_VERSION=${{ env.GCC_VERSION }}
dockerhub_repo: ${{ github.repository_owner == 'XRPLF' && 'rippleci/clio_gcc' || '' }}
dockerhub_description: GCC compiler for XRPLF/clio.
gcc-merge:
name: Merge and push multi-arch GCC docker image
runs-on: heavy
needs: [repo, gcc-amd64, gcc-arm64]
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Get changed files
id: changed-files
uses: tj-actions/changed-files@9426d40962ed5378910ee2e21d5f8c6fcbf2dd96 # v47.0.6
with:
files: "docker/compilers/gcc/**"
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
- name: Login to GitHub Container Registry
if: ${{ github.event_name != 'pull_request' }}
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Login to DockerHub
if: ${{ github.repository_owner == 'XRPLF' && github.event_name != 'pull_request' }}
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
with:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_PW }}
- name: Create and push multi-arch manifest
if: ${{ github.event_name != 'pull_request' && steps.changed-files.outputs.any_changed == 'true' }}
run: |
push_image() {
image=$1
docker buildx imagetools create \
-t $image:latest \
-t $image:${{ env.GCC_MAJOR_VERSION }} \
-t $image:${{ env.GCC_VERSION }} \
-t $image:${{ github.sha }} \
$image:arm64-latest \
$image:amd64-latest
}
push_image ${{ needs.repo.outputs.GHCR_REPO }}/clio-gcc
if [[ ${{ github.repository_owner }} == 'XRPLF' ]]; then
push_image rippleci/clio_clang
fi
clang:
name: Build and push Clang docker image
runs-on: heavy
needs: repo
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Get changed files
id: changed-files
uses: tj-actions/changed-files@9426d40962ed5378910ee2e21d5f8c6fcbf2dd96 # v47.0.6
with:
files: "docker/compilers/clang/**"
- uses: ./.github/actions/build-docker-image
if: ${{ steps.changed-files.outputs.any_changed == 'true' }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
DOCKERHUB_PW: ${{ secrets.DOCKERHUB_PW }}
with:
images: |
${{ needs.repo.outputs.GHCR_REPO }}/clio-clang
${{ github.repository_owner == 'XRPLF' && 'rippleci/clio_clang' || '' }}
push_image: ${{ github.event_name != 'pull_request' }}
directory: docker/compilers/clang
tags: |
type=raw,value=latest
type=raw,value=${{ env.CLANG_MAJOR_VERSION }}
type=raw,value=${{ github.sha }}
platforms: linux/amd64,linux/arm64
build_args: |
CLANG_MAJOR_VERSION=${{ env.CLANG_MAJOR_VERSION }}
dockerhub_repo: ${{ github.repository_owner == 'XRPLF' && 'rippleci/clio_clang' || '' }}
dockerhub_description: Clang compiler for XRPLF/clio.
tools-amd64:
name: Build and push tools docker image (amd64)
runs-on: heavy
needs: [repo, gcc-merge]
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Get changed files
id: changed-files
uses: tj-actions/changed-files@9426d40962ed5378910ee2e21d5f8c6fcbf2dd96 # v47.0.6
with:
files: "docker/tools/**"
- uses: ./.github/actions/build-docker-image
if: ${{ steps.changed-files.outputs.any_changed == 'true' }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
images: |
${{ needs.repo.outputs.GHCR_REPO }}/clio-tools
push_image: ${{ github.event_name != 'pull_request' }}
directory: docker/tools
tags: |
type=raw,value=amd64-latest
type=raw,value=amd64-${{ github.sha }}
platforms: linux/amd64
build_args: |
GHCR_REPO=${{ needs.repo.outputs.GHCR_REPO }}
GCC_VERSION=${{ env.GCC_VERSION }}
tools-arm64:
name: Build and push tools docker image (arm64)
runs-on: heavy-arm64
needs: [repo, gcc-merge]
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Get changed files
id: changed-files
uses: tj-actions/changed-files@9426d40962ed5378910ee2e21d5f8c6fcbf2dd96 # v47.0.6
with:
files: "docker/tools/**"
- uses: ./.github/actions/build-docker-image
if: ${{ steps.changed-files.outputs.any_changed == 'true' }}
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
images: |
${{ needs.repo.outputs.GHCR_REPO }}/clio-tools
push_image: ${{ github.event_name != 'pull_request' }}
directory: docker/tools
tags: |
type=raw,value=arm64-latest
type=raw,value=arm64-${{ github.sha }}
platforms: linux/arm64
build_args: |
GHCR_REPO=${{ needs.repo.outputs.GHCR_REPO }}
GCC_VERSION=${{ env.GCC_VERSION }}
tools-merge:
name: Merge and push multi-arch tools docker image
runs-on: heavy
needs: [repo, tools-amd64, tools-arm64]
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- name: Get changed files
id: changed-files
uses: tj-actions/changed-files@9426d40962ed5378910ee2e21d5f8c6fcbf2dd96 # v47.0.6
with:
files: "docker/tools/**"
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
- name: Login to GitHub Container Registry
if: ${{ github.event_name != 'pull_request' }}
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
with:
registry: ghcr.io
username: ${{ github.repository_owner }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Create and push multi-arch manifest
if: ${{ github.event_name != 'pull_request' && steps.changed-files.outputs.any_changed == 'true' }}
run: |
image=${{ needs.repo.outputs.GHCR_REPO }}/clio-tools
docker buildx imagetools create \
-t $image:latest \
-t $image:${{ github.sha }} \
$image:arm64-latest \
$image:amd64-latest
pre-commit:
name: Build and push pre-commit docker image
runs-on: heavy
needs: [repo, tools-merge]
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: ./.github/actions/build-docker-image
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
images: |
${{ needs.repo.outputs.GHCR_REPO }}/clio-pre-commit
push_image: ${{ github.event_name != 'pull_request' }}
directory: docker/pre-commit
tags: |
type=raw,value=latest
type=raw,value=${{ github.sha }}
platforms: linux/amd64,linux/arm64
build_args: |
GHCR_REPO=${{ needs.repo.outputs.GHCR_REPO }}
ci:
name: Build and push CI docker image
runs-on: heavy
needs: [repo, gcc-merge, clang, tools-merge]
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: ./.github/actions/build-docker-image
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
DOCKERHUB_PW: ${{ secrets.DOCKERHUB_PW }}
with:
images: |
${{ needs.repo.outputs.GHCR_REPO }}/clio-ci
${{ github.repository_owner == 'XRPLF' && 'rippleci/clio_ci' || '' }}
push_image: ${{ github.event_name != 'pull_request' }}
directory: docker/ci
tags: |
type=raw,value=latest
type=raw,value=gcc_${{ env.GCC_MAJOR_VERSION }}_clang_${{ env.CLANG_MAJOR_VERSION }}
type=raw,value=${{ github.sha }}
platforms: linux/amd64,linux/arm64
build_args: |
GHCR_REPO=${{ needs.repo.outputs.GHCR_REPO }}
CLANG_MAJOR_VERSION=${{ env.CLANG_MAJOR_VERSION }}
GCC_MAJOR_VERSION=${{ env.GCC_MAJOR_VERSION }}
GCC_VERSION=${{ env.GCC_VERSION }}
dockerhub_repo: ${{ github.repository_owner == 'XRPLF' && 'rippleci/clio_ci' || '' }}
dockerhub_description: CI image for XRPLF/clio.

View File

@@ -52,14 +52,14 @@ jobs:
outputs:
matrix: ${{ steps.set-matrix.outputs.matrix }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- name: Calculate conan matrix
id: set-matrix
run: .github/scripts/conan/generate_matrix.py >> "${GITHUB_OUTPUT}"
run: .github/scripts/conan/generate_matrix.py >>"${GITHUB_OUTPUT}"
upload-conan-deps:
name: Build ${{ matrix.compiler }}${{ matrix.sanitizer_ext }} ${{ matrix.build_type }}
name: Build ${{ matrix.compiler }} ${{ matrix.sanitizers }} ${{ matrix.build_type }}
needs: generate-matrix
@@ -71,39 +71,46 @@ jobs:
runs-on: ${{ matrix.os }}
container: ${{ matrix.container != '' && fromJson(matrix.container) || null }}
env:
CONAN_PROFILE: ${{ matrix.compiler }}${{ matrix.sanitizer_ext }}
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
- uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- name: Prepare runner
uses: XRPLF/actions/prepare-runner@90f11ee655d1687824fb8793db770477d52afbab
with:
enable_ccache: false
- name: Setup conan on macOS
if: ${{ runner.os == 'macOS' }}
run: ./.github/scripts/conan/init.sh
- name: Print build environment
uses: XRPLF/actions/print-build-env@59dec886e4afb05a1724443af08baccbc045b574
- name: Set compiler environment
if: ${{ runner.os == 'Linux' }}
uses: ./.github/actions/set-compiler-env
with:
compiler: ${{ matrix.compiler }}
- name: Setup conan
run: conan/init.sh
- name: Show conan profile
run: conan profile show --profile:all ${{ env.CONAN_PROFILE }}
env:
SANITIZERS: ${{ matrix.sanitizers }}
run: conan profile show --profile:all ci
- name: Run conan
uses: ./.github/actions/conan
with:
conan_profile: ${{ env.CONAN_PROFILE }}
sanitizers: ${{ matrix.sanitizers }}
# We check that everything builds fine from source on scheduled runs
# But we do build and upload packages with build=missing by default
force_conan_source_build: ${{ github.event_name == 'schedule' || github.event.inputs.force_source_build == 'true' }}
build_type: ${{ matrix.build_type }}
- name: Login to Conan
if: ${{ github.repository_owner == 'XRPLF' && github.event_name != 'pull_request' }}
- name: Log into Conan remote
if: ${{ github.repository == 'XRPLF/clio' && github.event_name != 'pull_request' }}
run: conan remote login -p ${{ secrets.CONAN_PASSWORD }} xrplf ${{ secrets.CONAN_USERNAME }}
- name: Upload Conan packages
if: ${{ github.repository_owner == 'XRPLF' && github.event_name != 'pull_request' && github.event_name != 'schedule' }}
if: ${{ github.repository == 'XRPLF/clio' && github.event_name != 'pull_request' && github.event_name != 'schedule' }}
env:
FORCE_OPTION: ${{ github.event.inputs.force_upload == 'true' && '--force' || '' }}
run: conan upload "*" -r=xrplf --confirm ${FORCE_OPTION}

View File

@@ -29,7 +29,7 @@ repos:
# Autoformat: YAML, JSON, Markdown, etc.
- repo: https://github.com/rbubley/mirrors-prettier
rev: c2bc67fe8f8f549cc489e00ba8b45aa18ee713b1 # frozen: v3.8.1
rev: 515f543f5718ebfd6ce22e16708bb32c68ff96e1 # frozen: v3.8.3
hooks:
- id: prettier
@@ -40,7 +40,7 @@ repos:
exclude: LICENSE.md
- repo: https://github.com/hadolint/hadolint
rev: 4e697ba704fd23b2409b947a319c19c3ee54d24f # frozen: v2.14.0
rev: 57e1618d78fd469a92c1e584e8c9313024656623 # frozen: v2.14.0
hooks:
- id: hadolint-docker
# hadolint-docker is a special hook that runs hadolint in a Docker container
@@ -59,15 +59,15 @@ repos:
]
- repo: https://github.com/psf/black-pre-commit-mirror
rev: fa505ab9c3e0fedafe1709fd7ac2b5f8996c670d # frozen: 26.3.1
rev: 4160603246a6b365d4a2af661c6d71b0a0f50478 # frozen: 26.5.1
hooks:
- id: black
- repo: https://github.com/scop/pre-commit-shfmt
rev: e26a818fd47b4f33cefa99035d1265b0849f4b47 # frozen: v3.13.0-1
rev: 05c1426671b9237fb5e1444dd63aa5731bec0dfb # frozen: v3.13.1-1
hooks:
- id: shfmt
args: ["-i", "4", "--write"]
args: [--write, --indent=4, --case-indent=true]
# Running some C++ hooks before clang-format
# to ensure that the style is consistent.
@@ -94,14 +94,14 @@ repos:
language: script
- repo: https://github.com/pre-commit/mirrors-clang-format
rev: 07a0f7667439f60724899f6ae288e4a4f572e0e1 # frozen: v22.1.2
rev: dd18dad857d6133e90bbe478f4f2f22ec0030269 # frozen: v22.1.5
hooks:
- id: clang-format
args: [--style=file]
types: [c++]
- repo: https://github.com/BlankSpruce/gersemi
rev: 79ddf2c9f3a84d766fce4e39fb2f83eac62b34f7 # frozen: 0.26.1
- repo: https://github.com/BlankSpruce/gersemi-pre-commit
rev: e98930bdc210d3387007f9252d8c1694ea7e410f # frozen: 0.27.7
hooks:
- id: gersemi

View File

@@ -16,7 +16,6 @@ option(docs "Generate doxygen docs" FALSE)
option(coverage "Build test coverage report" FALSE)
option(package "Create distribution packages" FALSE)
option(lint "Run clang-tidy checks during compilation" FALSE)
option(static "Statically linked Clio" FALSE)
option(snapshot "Build snapshot tool" FALSE)
option(
time_trace
@@ -25,10 +24,7 @@ option(
)
# ========================================================================== #
set(san "" CACHE STRING "Add sanitizer instrumentation")
set(CMAKE_EXPORT_COMPILE_COMMANDS TRUE)
set_property(CACHE san PROPERTY STRINGS ";undefined;memory;address;thread")
# ========================================================================== #
set(CMAKE_MODULE_PATH "${PROJECT_SOURCE_DIR}/cmake" ${CMAKE_MODULE_PATH})
@@ -42,6 +38,21 @@ add_library(clio_options INTERFACE)
target_compile_features(clio_options INTERFACE cxx_std_23) # Clio needs c++23 but deps can remain c++20 for now
target_include_directories(clio_options INTERFACE ${CMAKE_SOURCE_DIR}/src)
if(
CMAKE_CXX_COMPILER_ID STREQUAL "Clang"
OR CMAKE_CXX_COMPILER_ID STREQUAL "GNU"
)
# Note: -static-libstdc++ can statically link both libstdc++ and libc++
target_link_libraries(
clio_options
INTERFACE -static-libstdc++ -static-libgcc
)
endif()
# Apply sanitizer instrumentation if requested and define SANITIZERS_ENABLED.
# Must come before the modules and subdirectories below that key off it.
include(Sanitizers)
if(verbose)
set(CMAKE_VERBOSE_MAKEFILE TRUE)
endif()
@@ -50,6 +61,7 @@ endif()
include(CheckCompiler)
include(Settings)
include(SourceLocation)
include(PatchNixBinary)
# Clio deps
include(deps/libxrpl)
@@ -68,29 +80,6 @@ if(benchmark)
add_subdirectory(benchmarks)
endif()
# Enable selected sanitizer if enabled via `san`
if(san)
set(SUPPORTED_SANITIZERS "address" "thread" "memory" "undefined")
if(NOT san IN_LIST SUPPORTED_SANITIZERS)
message(
FATAL_ERROR
"Error: Unsupported sanitizer '${san}'. Supported values are: ${SUPPORTED_SANITIZERS}."
)
endif()
# Sanitizers recommend minimum of -O1 for reasonable performance so we enable it for debug builds
set(SAN_OPTIMIZATION_FLAG "")
if(CMAKE_BUILD_TYPE STREQUAL "Debug")
set(SAN_OPTIMIZATION_FLAG -O1)
endif()
target_compile_options(
clio_options
INTERFACE ${SAN_OPTIMIZATION_FLAG} ${SAN_FLAG} -fno-omit-frame-pointer
)
target_link_libraries(clio_options INTERFACE ${SAN_FLAG} ${SAN_LIB})
endif()
# Generate `docs` target for doxygen documentation if enabled Note: use `make docs` to generate the documentation
if(docs)
add_subdirectory(docs)

View File

@@ -26,7 +26,7 @@ using namespace util::config;
namespace {
auto const kCONFIG = ClioConfigDefinition{
auto const kConfig = ClioConfigDefinition{
{"prometheus.compress_reply", ConfigValue{ConfigType::Boolean}.defaultValue(true)},
{"prometheus.enabled", ConfigValue{ConfigType::Boolean}.defaultValue(true)},
{"log.channels.[].channel", Array{ConfigValue{ConfigType::String}}},
@@ -48,10 +48,10 @@ auto const kCONFIG = ClioConfigDefinition{
void
init()
{
static std::once_flag kONCE;
std::call_once(kONCE, [] {
PrometheusService::init(kCONFIG);
(void)util::LogService::init(kCONFIG);
static std::once_flag kOnce;
std::call_once(kOnce, [] {
PrometheusService::init(kConfig);
(void)util::LogService::init(kConfig);
});
}

View File

@@ -169,11 +169,11 @@ public:
static auto
generateData()
{
constexpr auto kTOTAL = 10'000;
constexpr auto kTotal = 10'000;
std::vector<uint64_t> data;
data.reserve(kTOTAL);
data.reserve(kTotal);
util::MTRandomGenerator randomGenerator;
for (auto i = 0; i < kTOTAL; ++i)
for (auto i = 0; i < kTotal; ++i)
data.push_back(randomGenerator.uniform(1, 100'000'000));
return data;

View File

@@ -11,6 +11,7 @@
#include <barrier>
#include <chrono>
#include <cstddef>
#include <cstdint>
#include <filesystem>
#include <memory>
#include <string>
@@ -20,13 +21,19 @@
using namespace util;
static constexpr auto kLOG_FORMAT = "%Y-%m-%d %H:%M:%S.%f %^%3!l:%n%$ - %v";
static constexpr auto kLogFormat = "%Y-%m-%d %H:%M:%S.%f %^%3!l:%n%$ - %v";
struct BenchmarkLoggingInitializer {
[[nodiscard]] static std::shared_ptr<spdlog::sinks::sink>
createFileSink(LogService::FileLoggingParams const& params)
createFileSink(std::string const& logDir, uint32_t sizeMB, uint32_t maxFiles)
{
return LogService::createFileSink(params, kLOG_FORMAT);
return LogService::createFileSink(
LogService::FileLoggingParams{
.logDir = logDir,
.rotation = LogService::RotationParams{.sizeMB = sizeMB, .maxFiles = maxFiles},
},
kLogFormat
);
}
static Logger
@@ -64,15 +71,11 @@ benchmarkConcurrentFileLogging(benchmark::State& state)
state.PauseTiming();
std::filesystem::create_directories(logDir);
static constexpr size_t kQUEUE_SIZE = 8192;
static constexpr size_t kTHREAD_COUNT = 1;
spdlog::init_thread_pool(kQUEUE_SIZE, kTHREAD_COUNT);
static constexpr size_t kQueueSize = 8192;
static constexpr size_t kThreadCount = 1;
spdlog::init_thread_pool(kQueueSize, kThreadCount);
auto fileSink = BenchmarkLoggingInitializer::createFileSink({
.logDir = logDir,
.rotationSizeMB = 5,
.dirMaxFiles = 25,
});
auto fileSink = BenchmarkLoggingInitializer::createFileSink(logDir, 5, 25);
std::vector<std::thread> threads;
threads.reserve(numThreads);

View File

@@ -19,30 +19,3 @@ else()
"Supported compilers: AppleClang 15+, Clang 16+, GCC 12+"
)
endif()
if(san)
string(TOLOWER ${san} san)
set(SAN_FLAG "-fsanitize=${san}")
set(SAN_LIB "")
if(is_gcc)
if(san STREQUAL "address")
set(SAN_LIB "asan")
elseif(san STREQUAL "thread")
set(SAN_LIB "tsan")
elseif(san STREQUAL "memory")
set(SAN_LIB "msan")
elseif(san STREQUAL "undefined")
set(SAN_LIB "ubsan")
endif()
endif()
set(_saved_CRL ${CMAKE_REQUIRED_LIBRARIES})
set(CMAKE_REQUIRED_LIBRARIES "${SAN_FLAG};${SAN_LIB}")
check_cxx_compiler_flag(${SAN_FLAG} COMPILER_SUPPORTS_SAN)
set(CMAKE_REQUIRED_LIBRARIES ${_saved_CRL})
if(NOT COMPILER_SUPPORTS_SAN)
message(
FATAL_ERROR
"${san} sanitizer does not seem to be supported by your compiler"
)
endif()
endif()

View File

@@ -0,0 +1,55 @@
#[===================================================================[
Patch executables to run in non-Nix environments.
The Nix-based CI image links binaries against an ELF interpreter (loader)
that lives in the Nix store, so the resulting binaries don't run elsewhere
(including once installed from the .deb package). `patch_nix_binary` adds a
POST_BUILD step that resets the interpreter to the system default loader and
drops the rpath.
This is only active inside the Nix-based image, detected by the presence of
/tmp/loader-path.sh (shipped by that image, resolves the default loader). It
is skipped for sanitizer builds, whose runtime libraries are resolved through
the rpath. Everywhere else `patch_nix_binary` is a no-op.
#]===================================================================]
include_guard(GLOBAL)
# Provided by the Nix-based CI image; prints the system default ELF loader path.
set(_loader_path_script "/tmp/loader-path.sh")
if(
CMAKE_SYSTEM_NAME STREQUAL "Linux"
AND NOT SANITIZERS_ENABLED
AND EXISTS "${_loader_path_script}"
)
execute_process(
COMMAND "${_loader_path_script}"
OUTPUT_VARIABLE DEFAULT_LOADER_PATH
OUTPUT_STRIP_TRAILING_WHITESPACE
COMMAND_ERROR_IS_FATAL ANY
)
find_program(PATCHELF_COMMAND patchelf REQUIRED)
set(PATCH_NIX_BINARIES TRUE)
message(
STATUS
"Binaries will be patched to use loader '${DEFAULT_LOADER_PATH}'"
)
else()
set(PATCH_NIX_BINARIES FALSE)
endif()
function(patch_nix_binary target)
if(NOT PATCH_NIX_BINARIES)
return()
endif()
add_custom_command(
TARGET ${target}
POST_BUILD
COMMAND
"${PATCHELF_COMMAND}" --set-interpreter "${DEFAULT_LOADER_PATH}"
--remove-rpath "$<TARGET_FILE:${target}>"
COMMENT "Patching ${target}: set default loader, remove rpath"
VERBATIM
)
endfunction()

47
cmake/Sanitizers.cmake Normal file
View File

@@ -0,0 +1,47 @@
#[===================================================================[
Apply sanitizer flags built by the Conan profile.
Parsing, validation, and flag construction are performed in
conan/profiles/sanitizers. This module reads the following CMake variables
injected by the Conan toolchain via extra_variables:
- SANITIZERS: The active sanitizers (e.g. "address").
- SANITIZERS_COMPILER_FLAGS: Space-separated compiler flags.
- SANITIZERS_LINKER_FLAGS: Space-separated linker flags.
It defines SANITIZERS_ENABLED for the rest of the build to key off, and
applies the flags to the 'clio_options' interface library.
#]===================================================================]
include_guard(GLOBAL)
if(NOT DEFINED SANITIZERS)
set(SANITIZERS_ENABLED FALSE)
return()
endif()
set(SANITIZERS_ENABLED TRUE)
message(STATUS "=== Configuring sanitizers ===")
message(STATUS " SANITIZERS: ${SANITIZERS}")
message(STATUS " Compile flags: ${SANITIZERS_COMPILER_FLAGS}")
message(STATUS " Link flags: ${SANITIZERS_LINKER_FLAGS}")
# Flags arrive as space-separated strings; split into CMake lists before use
separate_arguments(
sanitizers_compiler_flags
UNIX_COMMAND
"${SANITIZERS_COMPILER_FLAGS}"
)
separate_arguments(
sanitizers_linker_flags
UNIX_COMMAND
"${SANITIZERS_LINKER_FLAGS}"
)
target_compile_options(
clio_options
INTERFACE
$<$<COMPILE_LANGUAGE:CXX>:${sanitizers_compiler_flags}>
$<$<COMPILE_LANGUAGE:C>:${sanitizers_compiler_flags}>
)
target_link_options(clio_options INTERFACE ${sanitizers_linker_flags})

View File

@@ -40,7 +40,7 @@ if(is_appleclang)
list(APPEND COMPILER_FLAGS -Wreorder-init-list)
endif()
if(san)
if(SANITIZERS_ENABLED)
# When building with sanitizers some compilers will actually produce extra warnings/errors. We don't want this yet,
# at least not until we have fixed all runtime issues reported by the sanitizers. Once that is done we can start
# removing some of these and trying to fix it in our codebase. We can never remove all of below because most of them
@@ -85,3 +85,22 @@ target_compile_options(clio_options INTERFACE ${COMPILER_FLAGS})
# Add debug symbols for all builds, including Release. This is needed to get useful stack traces in production.
target_compile_options(clio_options INTERFACE -g)
# Keep -stdlib=libstdc++ off the compile commands, but preserve it for linking.
#
# Conan turns `compiler.libcxx=libstdc++` into `-stdlib=libstdc++` and puts it in CMAKE_CXX_FLAGS, which CMake passes to
# BOTH compile and link steps. On a normal Clang the compile step consumes it while choosing the C++ stdlib include
# paths. The Nixpkgs Clang wrapper (used by our CI image) supplies those paths itself (via -nostdinc++), so at compile
# time the flag is unused -> Clang errors under our -Werror. At link time the flag IS consumed (it selects the C++
# runtime), so we move it there instead of dropping it entirely.
get_filename_component(_cxx_real "${CMAKE_CXX_COMPILER}" REALPATH)
if(
_cxx_real MATCHES "^/nix/store/"
AND CMAKE_SYSTEM_NAME STREQUAL "Linux"
AND is_clang
AND CMAKE_CXX_FLAGS MATCHES "stdlib=libstdc"
)
string(REPLACE "-stdlib=libstdc++" "" CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS}")
string(STRIP "${CMAKE_CXX_FLAGS}" CMAKE_CXX_FLAGS)
add_link_options($<$<LINK_LANGUAGE:CXX>:-stdlib=libstdc++>)
endif()

View File

@@ -1,4 +1,4 @@
if("${san}" STREQUAL "")
if(NOT SANITIZERS_ENABLED)
target_compile_definitions(clio_options INTERFACE BOOST_STACKTRACE_LINK)
target_compile_definitions(
clio_options

View File

@@ -10,36 +10,36 @@ CLIO_BIN="$CLIO_PREFIX/bin/${CLIO_EXECUTABLE}"
CLIO_CONFIG="$CLIO_PREFIX/etc/config.json"
case "$1" in
configure)
if ! id -u "$USER_NAME" >/dev/null 2>&1; then
# Users who should not have a home directory should have their home directory set to /nonexistent
# https://www.debian.org/doc/debian-policy/ch-opersys.html#non-existent-home-directories
useradd \
--system \
--home-dir /nonexistent \
--no-create-home \
--shell /usr/sbin/nologin \
--comment "system user for ${CLIO_EXECUTABLE}" \
--user-group \
${USER_NAME}
fi
configure)
if ! id -u "$USER_NAME" >/dev/null 2>&1; then
# Users who should not have a home directory should have their home directory set to /nonexistent
# https://www.debian.org/doc/debian-policy/ch-opersys.html#non-existent-home-directories
useradd \
--system \
--home-dir /nonexistent \
--no-create-home \
--shell /usr/sbin/nologin \
--comment "system user for ${CLIO_EXECUTABLE}" \
--user-group \
${USER_NAME}
fi
install -d -o "$USER_NAME" -g "$GROUP_NAME" /var/log/clio
install -d -o "$USER_NAME" -g "$GROUP_NAME" /var/log/clio
if [ -f "$CLIO_CONFIG" ]; then
chown "$USER_NAME:$GROUP_NAME" "$CLIO_CONFIG"
fi
if [ -f "$CLIO_CONFIG" ]; then
chown "$USER_NAME:$GROUP_NAME" "$CLIO_CONFIG"
fi
chown -R "$USER_NAME:$GROUP_NAME" "$CLIO_PREFIX"
chown -R "$USER_NAME:$GROUP_NAME" "$CLIO_PREFIX"
ln -sf "$CLIO_BIN" "/usr/bin/${CLIO_EXECUTABLE}"
ln -sf "$CLIO_BIN" "/usr/bin/${CLIO_EXECUTABLE}"
;;
abort-upgrade | abort-remove | abort-deconfigure) ;;
*)
echo "postinst called with unknown argument \`$1'" >&2
exit 1
;;
;;
abort-upgrade | abort-remove | abort-deconfigure) ;;
*)
echo "postinst called with unknown argument \`$1'" >&2
exit 1
;;
esac
exit 0

View File

@@ -1,17 +1,17 @@
{
"version": "0.5",
"requires": [
"zlib/1.3.1#b8bc2603263cf7eccbd6e17e66b0ed76%1765850150.075",
"zlib/1.3.1#cac0f6daea041b0ccf42934163defb20%1774439233.809",
"xxhash/0.8.3#681d36a0a6111fc56e5e45ea182c19cc%1765850149.987",
"xrpl/3.1.0#3d408ab8c8020014fa7dd52bc7cc7ea8%1769706825.165",
"xrpl/3.1.3#eabc6838b4553671bdde4b3aa1f8e3e6%1778495466.414",
"sqlite3/3.49.1#8631739a4c9b93bd3d6b753bac548a63%1765850149.926",
"spdlog/1.17.0#bcbaaf7147bda6ad24ffbd1ac3d7142c%1768312128.781",
"soci/4.0.3#a9f8d773cd33e356b5879a4b0564f287%1765850149.46",
"soci/4.0.3#fe32b9ad5eb47e79ab9e45a68f363945%1774450067.231",
"re2/20230301#ca3b241baec15bd31ea9187150e0b333%1765850148.103",
"rapidjson/cci.20220822#1b9d8c2256876a154172dc5cfbe447c6%1754325007.656",
"protobuf/3.21.12#44ee56c0a6eea0c19aeeaca680370b88%1764175361.456",
"openssl/1.1.1w#a8f0792d7c5121b954578a7149d23e03%1756223730.729",
"nudb/2.0.9#0432758a24204da08fee953ec9ea03cb%1769436073.32",
"nudb/2.0.9#11149c73f8f2baff9a0198fe25971fc7%1775040983.408",
"minizip/1.2.13#9e87d57804bd372d6d1e32b1871517a3%1754325004.374",
"lz4/1.10.0#59fc63cac7f10fbe8e05c7e62c2f3504%1765850143.914",
"libuv/1.46.0#dc28c1f653fa197f00db5b577a6f6011%1754325003.592",
@@ -25,18 +25,18 @@
"doctest/2.4.11#a4211dfc329a16ba9f280f9574025659%1756234220.819",
"date/3.0.4#862e11e80030356b53c2c38599ceb32b%1765850143.772",
"cassandra-cpp-driver/2.17.0#bd3934138689482102c265d01288a316%1764175359.611",
"c-ares/1.34.5#5581c2b62a608b40bb85d965ab3ec7c8%1765850144.336",
"c-ares/1.34.6#545240bb1c40e2cacd4362d6b8967650%1774439234.681",
"bzip2/1.0.8#c470882369c2d95c5c77e970c0c7e321%1765850143.837",
"boost/1.83.0#91d8b1572534d2c334d6790e3c34d0c1%1764175359.61",
"benchmark/1.9.4#ce4403f7a24d3e1f907cd9da4b678be4%1754578869.672",
"benchmark/1.9.5#b885dc73ad67b40a55d45684d1c88ad1%1774363287.434",
"abseil/20230802.1#90ba607d4ee8fb5fb157c3db540671fc%1764175359.429"
],
"build_requires": [
"zlib/1.3.1#b8bc2603263cf7eccbd6e17e66b0ed76%1765850150.075",
"zlib/1.3.1#cac0f6daea041b0ccf42934163defb20%1774439233.809",
"protobuf/3.21.12#44ee56c0a6eea0c19aeeaca680370b88%1764175361.456",
"cmake/4.2.0#ae0a44f44a1ef9ab68fd4b3e9a1f8671%1765850153.937",
"cmake/3.31.10#313d16a1aa16bbdb2ca0792467214b76%1765850153.479",
"b2/5.3.3#107c15377719889654eb9a162a673975%1765850144.355"
"cmake/4.3.0#b939a42e98f593fb34d3a8c5cc860359%1774439249.183",
"cmake/3.31.11#f325c933f618a1fcebc1e1c0babfd1ba%1774439246.719",
"b2/5.4.2#ffd6084a119587e70f11cd45d1a386e2%1774439233.447"
],
"python_requires": [],
"overrides": {
@@ -51,10 +51,10 @@
"lz4/1.9.4": [
"lz4/1.10.0"
],
"boost/1.90.0": [
"boost/[>=1.83.0 <1.91.0]": [
"boost/1.83.0"
],
"sqlite3/3.44.2": [
"sqlite3/[>=3.44 <4]": [
"sqlite3/3.49.1"
]
},

7
conan/global.conf Normal file
View File

@@ -0,0 +1,7 @@
# Global configuration for Conan. This is used to set the number of parallel
# downloads and uploads.
core:non_interactive=True
core.download:parallel={{ os.cpu_count() }}
core.upload:parallel={{ os.cpu_count() }}
tools.files.download:retry=5
tools.files.download:retry_wait=10

30
conan/init.sh Executable file
View File

@@ -0,0 +1,30 @@
#!/bin/bash
set -ex
CURRENT_DIR="$(cd "$(dirname "$0")" && pwd)"
PROFILES_SRC_DIR="$CURRENT_DIR/profiles"
CONAN_DIR="${CONAN_HOME:-$HOME/.conan2}"
PROFILES_DIR="$CONAN_DIR/profiles"
rm -rf "$CONAN_DIR"
conan remote add --index 0 --force xrplf https://conan.ripplex.io
cp "$CURRENT_DIR/global.conf" "$CONAN_DIR/global.conf"
mkdir -p "$PROFILES_DIR"
# The compiler is selected via the `CC`/`CXX` environment variables (see
# `.github/actions/set-compiler-env`) and the sanitizers via the `SANITIZERS`
# environment variable. Builds always use the `ci` profile, which includes
# `sanitizers` and `default`.
cp "$PROFILES_SRC_DIR/ci" "$PROFILES_DIR/ci"
cp "$PROFILES_SRC_DIR/sanitizers" "$PROFILES_DIR/sanitizers"
if [[ "$(uname)" == "Darwin" ]]; then
cp "$PROFILES_SRC_DIR/apple-clang-17.profile" "$PROFILES_DIR/default"
else
cp "$PROFILES_SRC_DIR/default" "$PROFILES_DIR/default"
fi

8
conan/profiles/ci Normal file
View File

@@ -0,0 +1,8 @@
{% set os = detect_api.detect_os() %}
include(sanitizers)
[conf]
{% if os == "Linux" %}
user.package:libc_version=2.31
tools.info.package_id:confs+=["user.package:libc_version"]
{% endif %}

28
conan/profiles/default Normal file
View File

@@ -0,0 +1,28 @@
{% set os = detect_api.detect_os() %}
{% set arch = detect_api.detect_arch() %}
{% set compiler, version, compiler_exe = detect_api.detect_default_compiler() %}
{% set compiler_version = version %}
{% if os == "Linux" %}
{% set compiler_version = detect_api.default_compiler_version(compiler, version) %}
{% endif %}
[settings]
os={{ os }}
arch={{ arch }}
build_type=Debug
compiler={{compiler}}
compiler.version={{ compiler_version }}
compiler.cppstd=20
{% if os == "Windows" %}
compiler.runtime=static
{% else %}
compiler.libcxx={{detect_api.detect_libcxx(compiler, version, compiler_exe)}}
{% endif %}
[conf]
{% if compiler == "gcc" and compiler_version < 13 %}
tools.build:cxxflags+=['-Wno-restrict']
{% endif %}
{% if compiler == "clang" %}
grpc/1.50.1:tools.build:cxxflags+=['-Wno-missing-template-arg-list-after-template-kw']
{% endif %}

120
conan/profiles/sanitizers Normal file
View File

@@ -0,0 +1,120 @@
include(default)
{% set compiler, version, compiler_exe = detect_api.detect_default_compiler() %}
{% set arch = detect_api.detect_arch() %}
{% set sanitizers = os.getenv("SANITIZERS") %}
{% if not sanitizers %}
{# Sanitizers not configured; no additional settings needed #}
{% else %}
{% if compiler == "msvc" %}
{{ "Sanitizers are not supported on Windows/MSVC. Please unset the SANITIZERS environment variable." }}
{% endif %}
{% set known_sanitizers = ["address", "thread", "undefinedbehavior"] %}
{% set provided_sanitizers = [] %}
{% for san in sanitizers.split(",") %}
{% set san = san.strip() %}
{% if san not in known_sanitizers %}
{{ "Unknown sanitizer in SANITIZERS: " ~ san }}
{% endif %}
{% set _ = provided_sanitizers.append(san) %}
{% endfor %}
{% set enable_asan = "address" in provided_sanitizers %}
{% set enable_tsan = "thread" in provided_sanitizers %}
{% set enable_ubsan = "undefinedbehavior" in provided_sanitizers %}
{% if enable_asan and enable_tsan %}
{{ "AddressSanitizer and ThreadSanitizer are incompatible and cannot be enabled simultaneously." }}
{% endif %}
{% set sanitizer_types = [] %}
{% set defines = [] %}
{% if enable_asan %}
{% set _ = sanitizer_types.append("address") %}
{% set _ = defines.append("BOOST_USE_ASAN") %}
{% set _ = defines.append("BOOST_USE_UCONTEXT") %}
{% elif enable_tsan %}
{% set _ = sanitizer_types.append("thread") %}
{% set _ = defines.append("BOOST_USE_TSAN") %}
{% set _ = defines.append("BOOST_USE_UCONTEXT") %}
{% endif %}
{% if enable_ubsan %}
{% set _ = sanitizer_types.append("undefined") %}
{% set _ = sanitizer_types.append("float-divide-by-zero") %}
{# Clang supports additional UB checks beyond the GCC baseline #}
{% if compiler == "clang" or compiler == "apple-clang" %}
{% set _ = sanitizer_types.append("unsigned-integer-overflow") %}
{% endif %}
{% endif %}
{# Frame pointer required for meaningful stack traces; -O1 for reasonable performance #}
{% set sanitizer_compiler_flags = ["-fno-omit-frame-pointer", "-O1"] %}
{% if compiler == "gcc" %}
{# Suppress false positive warnings with GCC #}
{% set _ = sanitizer_compiler_flags.append("-Wno-stringop-overflow") %}
{% set relocation_flags = [] %}
{% if arch == "x86_64" and enable_asan %}
{# Large code model prevents relocation errors in instrumented ASAN binaries #}
{% set _ = sanitizer_compiler_flags.append("-mcmodel=large") %}
{% set _ = relocation_flags.append("-mcmodel=large") %}
{% elif enable_tsan %}
{# GCC doesn't support atomic_thread_fence with TSAN; suppress warnings #}
{% set _ = sanitizer_compiler_flags.append("-Wno-tsan") %}
{% if arch == "x86_64" %}
{# Medium code model for TSAN; large is incompatible #}
{% set _ = sanitizer_compiler_flags.append("-mcmodel=medium") %}
{% set _ = relocation_flags.append("-mcmodel=medium") %}
{% endif %}
{% endif %}
{% set fsanitize = "-fsanitize=" ~ ",".join(sanitizer_types) %}
{% set _ = sanitizer_compiler_flags.append(fsanitize) %}
{% set _ = relocation_flags.append(fsanitize) %}
{% set sanitizer_linker_flags = relocation_flags %}
{% elif compiler == "clang" or compiler == "apple-clang" %}
{% set fsanitize = "-fsanitize=" ~ ",".join(sanitizer_types) %}
{% set _ = sanitizer_compiler_flags.append(fsanitize) %}
{% set sanitizer_linker_flags = [fsanitize] %}
{% endif %}
[conf]
tools.build:defines+={{defines}}
tools.build:cxxflags+={{sanitizer_compiler_flags}}
tools.build:sharedlinkflags+={{sanitizer_linker_flags}}
tools.build:exelinkflags+={{sanitizer_linker_flags}}
tools.info.package_id:confs+=["tools.build:cxxflags", "tools.build:exelinkflags", "tools.build:sharedlinkflags", "tools.build:defines"]
# &: means "apply only to the consumer/root package"
&:tools.cmake.cmaketoolchain:extra_variables={"SANITIZERS": "{{sanitizers}}", "SANITIZERS_COMPILER_FLAGS": "{{sanitizer_compiler_flags | join(' ')}}", "SANITIZERS_LINKER_FLAGS": "{{sanitizer_linker_flags | join(' ')}}"}
[options]
{% if enable_asan %}
# Build Boost.Context with ucontext backend (not fcontext) so that
# ASAN fiber-switching annotations (__sanitizer_start/finish_switch_fiber)
# are compiled into the library. fcontext (assembly) has no ASAN support.
# define=BOOST_USE_ASAN=1 is critical: it must be defined when building
# Boost.Context itself so the ucontext backend compiles in the ASAN annotations.
boost/*:extra_b2_flags=context-impl=ucontext address-sanitizer=on define=BOOST_USE_ASAN=1
boost/*:without_context=False
# Boost stacktrace fails to build with some sanitizers
boost/*:without_stacktrace=True
{% elif enable_tsan %}
# Build Boost.Context with ucontext backend for TSAN. fcontext (assembly)
# has no TSAN annotations, so without this the BOOST_USE_TSAN/BOOST_USE_UCONTEXT
# defines in [conf] would be ineffective.
boost/*:extra_b2_flags=context-impl=ucontext thread-sanitizer=on define=BOOST_USE_TSAN=1
boost/*:without_context=False
boost/*:without_stacktrace=True
{% endif %}
{% endif %}

View File

@@ -1,6 +1,7 @@
from conan import ConanFile
from conan.tools.cmake import CMake, CMakeToolchain, cmake_layout
from conan import ConanFile
class ClioConan(ConanFile):
name = "clio"
@@ -19,7 +20,7 @@ class ClioConan(ConanFile):
"openssl/1.1.1w",
"protobuf/3.21.12",
"spdlog/1.17.0",
"xrpl/3.1.0",
"xrpl/3.1.3",
"zlib/1.3.1",
]
@@ -44,7 +45,7 @@ class ClioConan(ConanFile):
def requirements(self):
self.requires("boost/1.83.0", force=True)
self.requires("gtest/1.17.0")
self.requires("benchmark/1.9.4")
self.requires("benchmark/1.9.5")
def configure(self):
if self.settings.compiler == "apple-clang":

View File

@@ -1,132 +0,0 @@
ARG GHCR_REPO=invalid
ARG CLANG_MAJOR_VERSION=invalid
ARG GCC_VERSION=invalid
FROM ${GHCR_REPO}/clio-gcc:${GCC_VERSION} AS clio-gcc
FROM ${GHCR_REPO}/clio-tools:latest AS clio-tools
FROM ${GHCR_REPO}/clio-clang:${CLANG_MAJOR_VERSION}
ARG DEBIAN_FRONTEND=noninteractive
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
# Using root by default is not very secure but github checkout action doesn't work with any other user
# https://github.com/actions/checkout/issues/956
# And Github Actions doc recommends using root
# https://docs.github.com/en/actions/sharing-automations/creating-actions/dockerfile-support-for-github-actions#user
# hadolint ignore=DL3002
USER root
WORKDIR /root
# Install common tools and dependencies
RUN apt-get update \
&& apt-get install -y --no-install-recommends --no-install-suggests \
curl \
dpkg-dev \
file \
git \
git-lfs \
gnupg \
graphviz \
jq \
# libgmp, libmpfr and libncurses are gdb dependencies
libgmp-dev \
libmpfr-dev \
libncurses-dev \
make \
wget \
zip \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
# Install Python tools
RUN apt-get update \
&& apt-get install -y --no-install-recommends --no-install-suggests \
python3 \
python3-pip \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
RUN pip install -q --no-cache-dir \
# TODO: Remove this once we switch to newer Ubuntu base image
# lxml 6.0.0 is not compatible with our image
'lxml<6.0.0' \
cmake \
conan==2.24.0 \
gcovr \
# We're adding pre-commit to this image as well,
# because clang-tidy workflow requires it
pre-commit
# Install LLVM tools
ARG LLVM_TOOLS_VERSION=20
RUN echo "deb http://apt.llvm.org/focal/ llvm-toolchain-focal-${LLVM_TOOLS_VERSION} main" >> /etc/apt/sources.list \
&& wget --progress=dot:giga -O - https://apt.llvm.org/llvm-snapshot.gpg.key | apt-key add -
RUN apt-get update \
&& apt-get install -y --no-install-recommends --no-install-suggests \
clang-tidy-${LLVM_TOOLS_VERSION} \
clang-tools-${LLVM_TOOLS_VERSION} \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
ARG GCC_MAJOR_VERSION=invalid
# Install custom-built gcc and make ldconfig aware of the new libstdc++ location (for gcc)
# Note: Clang is using libc++ instead
COPY --from=clio-gcc /gcc${GCC_MAJOR_VERSION}.deb /
RUN apt-get update \
&& apt-get install -y --no-install-recommends --no-install-suggests \
binutils \
libc6-dev \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* \
&& dpkg -i /gcc${GCC_MAJOR_VERSION}.deb \
&& rm -rf /gcc${GCC_MAJOR_VERSION}.deb \
&& ldconfig
# Rewire to use our custom-built gcc as default compiler
RUN update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-${GCC_MAJOR_VERSION} 100 \
&& update-alternatives --install /usr/bin/c++ c++ /usr/bin/g++-${GCC_MAJOR_VERSION} 100 \
&& update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-${GCC_MAJOR_VERSION} 100 \
&& update-alternatives --install /usr/bin/cc cc /usr/bin/gcc-${GCC_MAJOR_VERSION} 100 \
&& update-alternatives --install /usr/bin/gcov gcov /usr/bin/gcov-${GCC_MAJOR_VERSION} 100 \
&& update-alternatives --install /usr/bin/gcov-dump gcov-dump /usr/bin/gcov-dump-${GCC_MAJOR_VERSION} 100 \
&& update-alternatives --install /usr/bin/gcov-tool gcov-tool /usr/bin/gcov-tool-${GCC_MAJOR_VERSION} 100
COPY --from=clio-tools \
/usr/local/bin/mold \
/usr/local/bin/ld.mold \
/usr/local/bin/ccache \
/usr/local/bin/doxygen \
/usr/local/bin/ClangBuildAnalyzer \
/usr/local/bin/git-cliff \
/usr/local/bin/gh \
/usr/local/bin/gdb \
/usr/local/bin/ninja \
/usr/local/bin/
WORKDIR /root
# Setup conan
RUN conan remote add --index 0 xrplf https://conan.ripplex.io
WORKDIR /root/.conan2
COPY conan/global.conf ./global.conf
WORKDIR /root/.conan2/profiles
COPY conan/clang.profile ./clang
COPY conan/sanitizer_template.profile ./clang.asan
COPY conan/sanitizer_template.profile ./clang.tsan
COPY conan/sanitizer_template.profile ./clang.ubsan
COPY conan/gcc.profile ./gcc
COPY conan/sanitizer_template.profile ./gcc.asan
COPY conan/sanitizer_template.profile ./gcc.tsan
COPY conan/sanitizer_template.profile ./gcc.ubsan
WORKDIR /root

View File

@@ -1,25 +0,0 @@
# CI image for XRPLF/clio
This image contains an environment to build [Clio](https://github.com/XRPLF/clio), check code and documentation.
It is used in [Clio Github Actions](https://github.com/XRPLF/clio/actions) but can also be used to compile Clio locally.
The image is based on Ubuntu 20.04 and contains:
- ccache 4.12.2
- Clang 19
- ClangBuildAnalyzer 1.6.0
- Conan 2.24.0
- Doxygen 1.16.1
- GCC 15.2.0
- GDB 17.1
- gh 2.83.2
- git-cliff 2.11.0
- mold 2.40.4
- Ninja 1.13.2
- Python 3.8
- and some other useful tools
Conan is set up to build Clio without any additional steps.
There are two preset conan profiles: `clang` and `gcc` to use corresponding compiler.
`ASan`, `TSan` and `UBSan` sanitizer builds are enabled via conan profiles for each of the supported compilers.
These can be selected using the following pattern (all lowercase): `[compiler].[sanitizer]` (e.g. `--profile:all gcc.tsan`).

View File

@@ -1,12 +0,0 @@
[settings]
arch={{detect_api.detect_arch()}}
build_type=Release
compiler=clang
compiler.cppstd=20
compiler.libcxx=libc++
compiler.version=19
os=Linux
[conf]
tools.build:compiler_executables={"c": "/usr/bin/clang-19", "cpp": "/usr/bin/clang++-19"}
grpc/1.50.1:tools.build:cxxflags+=["-Wno-missing-template-arg-list-after-template-kw"]

View File

@@ -1,11 +0,0 @@
[settings]
arch={{detect_api.detect_arch()}}
build_type=Release
compiler=gcc
compiler.cppstd=20
compiler.libcxx=libstdc++11
compiler.version=15
os=Linux
[conf]
tools.build:compiler_executables={"c": "/usr/bin/gcc-15", "cpp": "/usr/bin/g++-15"}

View File

@@ -1,2 +0,0 @@
core.download:parallel={{os.cpu_count()}}
core.upload:parallel={{os.cpu_count()}}

View File

@@ -1,37 +0,0 @@
{% set compiler, sani = profile_name.split('.') %}
{% set sanitizer_opt_map = {"asan": "address", "tsan": "thread", "ubsan": "undefined"} %}
{% set sanitizer = sanitizer_opt_map[sani] %}
{% set sanitizer_b2_flags_map = {
"address": "context-impl=ucontext address-sanitizer=norecover",
"thread": "context-impl=ucontext thread-sanitizer=norecover",
"undefined": "undefined-sanitizer=norecover"
} %}
{% set sanitizer_b2_flags_str = sanitizer_b2_flags_map[sanitizer] %}
{% set sanitizer_build_flags_str = "-fsanitize=" ~ sanitizer ~ " -g -O1 -fno-omit-frame-pointer" %}
{% set sanitizer_build_flags = sanitizer_build_flags_str.split(' ') %}
{% set sanitizer_link_flags_str = "-fsanitize=" ~ sanitizer %}
{% set sanitizer_link_flags = sanitizer_link_flags_str.split(' ') %}
include({{ compiler }})
[options]
boost/*:extra_b2_flags="{{ sanitizer_b2_flags_str }}"
boost/*:without_context=False
boost/*:without_stacktrace=True
[conf]
tools.build:cflags+={{ sanitizer_build_flags }}
tools.build:cxxflags+={{ sanitizer_build_flags }}
tools.build:exelinkflags+={{ sanitizer_link_flags }}
tools.build:sharedlinkflags+={{ sanitizer_link_flags }}
{% if sanitizer == "address" %}
tools.build:defines+=["BOOST_USE_ASAN", "BOOST_USE_UCONTEXT"]
{% elif sanitizer == "thread" %}
tools.build:defines+=["BOOST_USE_TSAN", "BOOST_USE_UCONTEXT"]
{% endif %}
tools.info.package_id:confs+=["tools.build:cflags", "tools.build:cxxflags", "tools.build:exelinkflags", "tools.build:sharedlinkflags", "tools.build:defines"]

View File

@@ -1,32 +0,0 @@
FROM ubuntu:20.04
ARG DEBIAN_FRONTEND=noninteractive
SHELL ["/bin/bash", "-c"]
# hadolint ignore=DL3002
USER root
WORKDIR /root
RUN apt-get update \
&& apt-get install -y --no-install-recommends --no-install-suggests \
wget \
software-properties-common \
gnupg \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
ARG CLANG_MAJOR_VERSION=invalid
# Bump this version to force rebuild of the image
ARG BUILD_VERSION=1
RUN wget --progress=dot:giga https://apt.llvm.org/llvm.sh \
&& chmod +x llvm.sh \
&& ./llvm.sh ${CLANG_MAJOR_VERSION} \
&& rm -rf llvm.sh \
&& apt-get update \
&& apt-get install -y --no-install-recommends --no-install-suggests \
libc++-${CLANG_MAJOR_VERSION}-dev \
libc++abi-${CLANG_MAJOR_VERSION}-dev \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*

View File

@@ -1,3 +0,0 @@
# Clang compiler
This image contains clang compiler to build <https://github.com/XRPLF/clio>.

View File

@@ -1,120 +0,0 @@
ARG UBUNTU_VERSION=20.04
ARG GCC_MAJOR_VERSION=invalid
FROM ubuntu:$UBUNTU_VERSION AS build
ARG UBUNTU_VERSION
ARG GCC_MAJOR_VERSION
ARG BUILD_VERSION=0
ARG DEBIAN_FRONTEND=noninteractive
ARG TARGETARCH
RUN apt-get update \
&& apt-get install -y --no-install-recommends --no-install-suggests \
build-essential \
file \
flex \
libz-dev \
libzstd-dev \
software-properties-common \
wget \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
ARG GCC_VERSION
WORKDIR /
RUN wget --progress=dot:giga https://gcc.gnu.org/pub/gcc/releases/gcc-$GCC_VERSION/gcc-$GCC_VERSION.tar.gz \
&& tar xf gcc-$GCC_VERSION.tar.gz
WORKDIR /gcc-$GCC_VERSION
RUN ./contrib/download_prerequisites
# hadolint ignore=DL3059
RUN mkdir /gcc-build
WORKDIR /gcc-build
RUN /gcc-$GCC_VERSION/configure \
--with-pkgversion="clio-build-$BUILD_VERSION https://github.com/XRPLF/clio" \
--enable-languages=c,c++ \
--prefix=/usr \
--with-gcc-major-version-only \
--program-suffix=-${GCC_MAJOR_VERSION} \
--enable-shared \
--enable-linker-build-id \
--libexecdir=/usr/lib \
--without-included-gettext \
--enable-threads=posix \
--libdir=/usr/lib \
--disable-nls \
--enable-clocale=gnu \
--enable-libstdcxx-backtrace=yes \
--enable-libstdcxx-debug \
--enable-libstdcxx-time=yes \
--with-default-libstdcxx-abi=new \
--enable-gnu-unique-object \
--disable-vtable-verify \
--enable-plugin \
--enable-default-pie \
--with-system-zlib \
--enable-libphobos-checking=release \
--with-target-system-zlib=auto \
--disable-werror \
--enable-cet \
--disable-multilib \
--without-cuda-driver \
--enable-checking=release
RUN make -j "$(nproc)"
RUN make install-strip DESTDIR=/gcc-$GCC_VERSION-$BUILD_VERSION-ubuntu-$UBUNTU_VERSION
RUN export GDB_AUTOLOAD_DIR="/gcc-$GCC_VERSION-$BUILD_VERSION-ubuntu-$UBUNTU_VERSION/usr/share/gdb/auto-load/usr/lib64" \
&& mkdir -p "$GDB_AUTOLOAD_DIR" \
&& mv \
/gcc-$GCC_VERSION-$BUILD_VERSION-ubuntu-$UBUNTU_VERSION/usr/lib64/libstdc++.so.*-gdb.py \
$GDB_AUTOLOAD_DIR/
# Generate deb
WORKDIR /
COPY control.m4 /
COPY ld.so.conf /gcc-$GCC_VERSION-$BUILD_VERSION-ubuntu-$UBUNTU_VERSION/etc/ld.so.conf.d/1-gcc-${GCC_MAJOR_VERSION}.conf
RUN mkdir /gcc-$GCC_VERSION-$BUILD_VERSION-ubuntu-$UBUNTU_VERSION/DEBIAN \
&& m4 \
-P \
-DUBUNTU_VERSION=$UBUNTU_VERSION \
-DVERSION=$GCC_VERSION-$BUILD_VERSION \
-DTARGETARCH=$TARGETARCH \
control.m4 > /gcc-$GCC_VERSION-$BUILD_VERSION-ubuntu-$UBUNTU_VERSION/DEBIAN/control \
&& dpkg-deb \
--build \
--root-owner-group \
/gcc-$GCC_VERSION-$BUILD_VERSION-ubuntu-$UBUNTU_VERSION \
/gcc${GCC_MAJOR_VERSION}.deb
# Create final image
FROM ubuntu:$UBUNTU_VERSION
ARG GCC_MAJOR_VERSION
COPY --from=build /gcc${GCC_MAJOR_VERSION}.deb /
# Install gcc-${GCC_MAJOR_VERSION}, but also leave gcc${GCC_MAJOR_VERSION}.deb for others to copy if needed
RUN apt-get update \
&& apt-get install -y --no-install-recommends --no-install-suggests \
binutils \
libc6-dev \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/* \
&& dpkg -i /gcc${GCC_MAJOR_VERSION}.deb
RUN update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-${GCC_MAJOR_VERSION} 100 \
&& update-alternatives --install /usr/bin/c++ c++ /usr/bin/g++-${GCC_MAJOR_VERSION} 100 \
&& update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-${GCC_MAJOR_VERSION} 100 \
&& update-alternatives --install /usr/bin/cc cc /usr/bin/gcc-${GCC_MAJOR_VERSION} 100 \
&& update-alternatives --install /usr/bin/gcov gcov /usr/bin/gcov-${GCC_MAJOR_VERSION} 100 \
&& update-alternatives --install /usr/bin/gcov-dump gcov-dump /usr/bin/gcov-dump-${GCC_MAJOR_VERSION} 100 \
&& update-alternatives --install /usr/bin/gcov-tool gcov-tool /usr/bin/gcov-tool-${GCC_MAJOR_VERSION} 100

View File

@@ -1,3 +0,0 @@
# GCC compiler
This image contains GCC compiler to build <https://github.com/XRPLF/clio>.

View File

@@ -1,7 +0,0 @@
Package: gcc-15-ubuntu-UBUNTUVERSION
Version: VERSION
Architecture: TARGETARCH
Maintainer: Alex Kremer <akremer@ripple.com>
Uploaders: Ayaz Salikhov <asalikhov@ripple.com>
Description: GCC VERSION build for ubuntu UBUNTUVERSION
Depends: binutils, libc6-dev

View File

@@ -1,2 +0,0 @@
# Path to the directory containing libstdc++.so.6
/usr/lib64

View File

@@ -1,6 +1,6 @@
services:
clio_develop:
image: ghcr.io/xrplf/clio-ci:14342e087ceb8b593027198bf9ef06a43833c696
image: ghcr.io/xrplf/xrpld/nix-ubuntu:sha-7b9d553
volumes:
- clio_develop_conan_data:/root/.conan2/p
- clio_develop_ccache:/root/.ccache

View File

@@ -41,26 +41,26 @@ EOF
}
case $1 in
-h | --help)
print_help
;;
-h | --help)
print_help
;;
-t | --terminal)
open_terminal
;;
-t | --terminal)
open_terminal
;;
-s | --stop)
stop_container
;;
-s | --stop)
stop_container
;;
-*)
echo "Unknown option: $1"
print_help
;;
-*)
echo "Unknown option: $1"
print_help
;;
*)
run "$@"
;;
*)
run "$@"
;;
esac
popd >/dev/null

View File

@@ -1,112 +0,0 @@
ARG GHCR_REPO=invalid
ARG GCC_VERSION=invalid
FROM ${GHCR_REPO}/clio-gcc:${GCC_VERSION}
ARG DEBIAN_FRONTEND=noninteractive
ARG TARGETARCH
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
ARG BUILD_VERSION=0
RUN apt-get update \
&& apt-get install -y --no-install-recommends --no-install-suggests \
python3 \
python3-pip \
software-properties-common \
wget \
&& pip3 install -q --no-cache-dir \
cmake \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /tmp
ARG NINJA_VERSION=1.13.2
RUN wget --progress=dot:giga "https://github.com/ninja-build/ninja/archive/refs/tags/v${NINJA_VERSION}.tar.gz" \
&& tar xf "v${NINJA_VERSION}.tar.gz" \
&& cd "ninja-${NINJA_VERSION}" \
&& ./configure.py --bootstrap \
&& mv ninja /usr/local/bin/ninja \
&& rm -rf /tmp/* /var/tmp/*
ARG MOLD_VERSION=2.40.4
RUN wget --progress=dot:giga "https://github.com/rui314/mold/archive/refs/tags/v${MOLD_VERSION}.tar.gz" \
&& tar xf "v${MOLD_VERSION}.tar.gz" \
&& cd "mold-${MOLD_VERSION}" \
&& mkdir build \
&& cd build \
&& cmake -GNinja -DCMAKE_BUILD_TYPE=Release .. \
&& ninja install \
&& rm -rf /tmp/* /var/tmp/*
ARG CCACHE_VERSION=4.12.2
RUN wget --progress=dot:giga "https://github.com/ccache/ccache/releases/download/v${CCACHE_VERSION}/ccache-${CCACHE_VERSION}.tar.gz" \
&& tar xf "ccache-${CCACHE_VERSION}.tar.gz" \
&& cd "ccache-${CCACHE_VERSION}" \
&& mkdir build \
&& cd build \
&& cmake -GNinja -DCMAKE_BUILD_TYPE=Release -DENABLE_TESTING=False .. \
&& ninja install \
&& rm -rf /tmp/* /var/tmp/*
RUN apt-get update \
&& apt-get install -y --no-install-recommends --no-install-suggests \
bison \
flex \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
ARG DOXYGEN_VERSION=1.16.1
RUN wget --progress=dot:giga "https://github.com/doxygen/doxygen/releases/download/Release_${DOXYGEN_VERSION//./_}/doxygen-${DOXYGEN_VERSION}.src.tar.gz" \
&& tar xf "doxygen-${DOXYGEN_VERSION}.src.tar.gz" \
&& cd "doxygen-${DOXYGEN_VERSION}" \
&& mkdir build \
&& cd build \
&& cmake -GNinja -DCMAKE_BUILD_TYPE=Release .. \
&& ninja install \
&& rm -rf /tmp/* /var/tmp/*
ARG CLANG_BUILD_ANALYZER_VERSION=1.6.0
RUN wget --progress=dot:giga "https://github.com/aras-p/ClangBuildAnalyzer/archive/refs/tags/v${CLANG_BUILD_ANALYZER_VERSION}.tar.gz" \
&& tar xf "v${CLANG_BUILD_ANALYZER_VERSION}.tar.gz" \
&& cd "ClangBuildAnalyzer-${CLANG_BUILD_ANALYZER_VERSION}" \
&& mkdir build \
&& cd build \
&& cmake -GNinja -DCMAKE_BUILD_TYPE=Release .. \
&& ninja install \
&& rm -rf /tmp/* /var/tmp/*
ARG GIT_CLIFF_VERSION=2.11.0
RUN wget --progress=dot:giga "https://github.com/orhun/git-cliff/releases/download/v${GIT_CLIFF_VERSION}/git-cliff-${GIT_CLIFF_VERSION}-x86_64-unknown-linux-musl.tar.gz" \
&& tar xf git-cliff-${GIT_CLIFF_VERSION}-x86_64-unknown-linux-musl.tar.gz \
&& mv git-cliff-${GIT_CLIFF_VERSION}/git-cliff /usr/local/bin/git-cliff \
&& rm -rf /tmp/* /var/tmp/*
ARG GH_VERSION=2.83.2
RUN wget --progress=dot:giga "https://github.com/cli/cli/releases/download/v${GH_VERSION}/gh_${GH_VERSION}_linux_${TARGETARCH}.tar.gz" \
&& tar xf gh_${GH_VERSION}_linux_${TARGETARCH}.tar.gz \
&& mv gh_${GH_VERSION}_linux_${TARGETARCH}/bin/gh /usr/local/bin/gh \
&& rm -rf /tmp/* /var/tmp/*
RUN apt-get update \
&& apt-get install -y --no-install-recommends --no-install-suggests \
libgmp-dev \
libmpfr-dev \
libncurses-dev \
make \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
ARG GDB_VERSION=17.1
RUN wget --progress=dot:giga "https://sourceware.org/pub/gdb/releases/gdb-${GDB_VERSION}.tar.gz" \
&& tar xf "gdb-${GDB_VERSION}.tar.gz" \
&& cd "gdb-${GDB_VERSION}" \
&& ./configure --prefix=/usr/local \
&& make -j "$(nproc)" \
&& make install-gdb \
&& rm -rf /tmp/* /var/tmp/*
WORKDIR /root

View File

@@ -11,8 +11,8 @@
- [**Optional**] [GCovr](https://gcc.gnu.org/onlinedocs/gcc/Gcov.html): needed for code coverage generation
- [**Optional**] [CCache](https://ccache.dev/): speeds up compilation if you are going to compile Clio often
We use our Docker image `ghcr.io/XRPLF/clio-ci` to build `Clio`, see [Building Clio with Docker](#building-clio-with-docker).
You can find information about exact compiler versions and tools in the [image's README](https://github.com/XRPLF/clio/blob/develop/docker/ci/README.md).
We use the Nix-based Docker image `ghcr.io/xrplf/xrpld/nix-ubuntu` to build `Clio`, see [Building Clio with Docker](#building-clio-with-docker).
This image is produced by [rippled](https://github.com/XRPLF/rippled) and ships the compilers and tools listed below.
The following compiler version are guaranteed to work.
Any compiler with lower version may not be able to build Clio:
@@ -30,7 +30,7 @@ You can change it by using `$CONAN_HOME` env variable.
[More info about Conan home](https://docs.conan.io/2/reference/environment.html#conan-home).
> [!TIP]
> To setup Conan automatically, you can run `.github/scripts/conan/init.sh`.
> To setup Conan automatically, you can run `conan/init.sh`.
> This will delete Conan home directory (if it exists), set up profiles and add Artifactory remote.
The instruction below assumes that `$CONAN_HOME` is not set.
@@ -175,7 +175,7 @@ Open the `index.html` file in your browser to see the documentation pages.
It is also possible to build Clio using [Docker](https://www.docker.com/) if you don't want to install all the dependencies on your machine.
```sh
docker run -it ghcr.io/xrplf/clio-ci:14342e087ceb8b593027198bf9ef06a43833c696
docker run -it ghcr.io/xrplf/xrpld/nix-ubuntu:sha-7b9d553
git clone https://github.com/XRPLF/clio
cd clio
```

View File

@@ -230,7 +230,7 @@ This document provides a list of all available Clio configuration properties in
- **Required**: False
- **Type**: string
- **Default value**: None
- **Constraints**: None
- **Constraints**: The value must be a valid IP address.
- **Description**: The list of IP addresses to whitelist for DOS protection.
### dos_guard.max_fetches
@@ -342,7 +342,7 @@ This document provides a list of all available Clio configuration properties in
- **Required**: True
- **Type**: string
- **Default value**: None
- **Constraints**: None
- **Constraints**: The value must be a valid IP address.
- **Description**: List of proxy ip addresses. When Clio receives a request from proxy it will use `Forwarded` value (if any) as client ip. When this option is used together with `server.proxy.tokens` Clio will identify proxy by ip or by token.
### server.proxy.tokens.[]
@@ -561,6 +561,14 @@ Documentation can be found at: <https://github.com/gabime/spdlog/wiki/Custom-for
- **Constraints**: The minimum value is `1`. The maximum value is `4294967295`.
- **Description**: The maximum number of log files in the directory.
### log.rotate
- **Required**: True
- **Type**: boolean
- **Default value**: `True`
- **Constraints**: None
- **Description**: Enables or disables log file rotation. When disabled, a single log file is used without size-based rotation. Useful when rotation is managed externally (e.g., via logrotate).
### log.tag_style
- **Required**: True

View File

@@ -1,51 +1,56 @@
#!/bin/bash
# git for-each-ref refs/tags # see which tags are annotated and which are lightweight. Annotated tags are "tag" objects.
# # Set these so your commits and tags are always signed
# git config commit.gpgsign true
# git config tag.gpgsign true
# Annotated tags have object type "tag"; lightweight tags have type "commit".
# To inspect tags: git for-each-ref refs/tags
#
# To always sign commits and tags, configure:
# git config --global commit.gpgsign true
# git config --global tag.gpgsign true
verify_commit_signed() {
if git verify-commit HEAD &>/dev/null; then
:
# echo "HEAD commit seems signed..."
echo "HEAD commit is signed."
else
echo "HEAD commit isn't signed!"
echo "HEAD commit is not signed!"
exit 1
fi
}
verify_tag() {
if git describe --exact-match --tags HEAD &>/dev/null; then
: # You might be ok to push
# echo "Tag is annotated."
return 0
verify_tag_annotated() {
local version="$1"
# git cat-file -t returns "tag" for annotated tags, "commit" for lightweight.
if [[ "$(git cat-file -t "$version")" == "tag" ]]; then
echo "Tag '$version' is annotated."
else
echo "Tag for [$version] not an annotated tag."
echo "Tag '$version' is not annotated!"
echo "Re-create it with: git tag -a -s -m \"$version\" \"$version\""
exit 1
fi
}
verify_tag_signed() {
local version="$1"
if git verify-tag "$version" &>/dev/null; then
: # ok, I guess we'll let you push
# echo "Tag appears signed"
return 0
echo "Tag '$version' is signed."
else
echo "$version tag isn't signed"
echo "Sign it with [git tag -ams\"$version\" $version]"
echo "Tag '$version' is not signed!"
echo "Sign it with: git tag -a -s -m \"$version\" \"$version\""
exit 1
fi
}
# Check some things if we're pushing a branch called "release/"
if echo "$PRE_COMMIT_REMOTE_BRANCH" | grep ^refs\/heads\/release\/ &>/dev/null; then
version=$(git tag --points-at HEAD)
echo "Looks like you're trying to push a $version release..."
echo "Making sure you've signed and tagged it."
if verify_commit_signed && verify_tag && verify_tag_signed; then
: # Ok, I guess you can push
else
# Enforce signing and annotated tags when pushing to a release branch.
if echo "$PRE_COMMIT_REMOTE_BRANCH" | grep -q "^refs/heads/release/"; then
# git describe --exact-match guarantees a single tag; git tag --points-at HEAD
# can return multiple newline-separated values, which breaks downstream commands.
version=$(git describe --exact-match --tags HEAD 2>/dev/null)
if [[ -z "$version" ]]; then
echo "No tag found at HEAD — cannot push an untagged release commit!"
exit 1
fi
echo "Looks like you're trying to push a '$version' release..."
echo "Verifying the commit is signed and the tag is annotated and signed."
verify_commit_signed
verify_tag_annotated "$version"
verify_tag_signed "$version"
fi

View File

@@ -28,7 +28,7 @@ CliArgs::parse(int argc, char const* argv[])
description.add_options()
("help,h", "Print help message and exit")
("version,v", "Print version and exit")
("conf,c", po::value<std::string>()->default_value(kDEFAULT_CONFIG_PATH), "Configuration file")
("conf,c", po::value<std::string>()->default_value(kDefaultConfigPath), "Configuration file")
("ng-web-server,w", "Use ng-web-server")
("migrate", po::value<std::string>(), "Start migration helper")
("verify", "Checks the validity of config values")

View File

@@ -16,7 +16,7 @@ public:
/**
* @brief Default configuration path.
*/
static constexpr char kDEFAULT_CONFIG_PATH[] = "/etc/opt/clio/config.json";
static constexpr char kDefaultConfigPath[] = "/etc/opt/clio/config.json";
/**
* @brief An action parsed from the command line.

View File

@@ -92,9 +92,14 @@ ClioApplication::run(bool const useNgWebServer)
boost::asio::io_context ioc{threads};
// Rate limiter, to prevent abuse
auto whitelistHandler = web::dosguard::WhitelistHandler{config_};
auto whitelistHandler = web::dosguard::WhitelistHandler::create(config_);
if (not whitelistHandler.has_value()) {
LOG(util::LogService::fatal()) << whitelistHandler.error();
return EXIT_FAILURE;
}
auto const dosguardWeights = web::dosguard::Weights::make(config_);
auto dosGuard = web::dosguard::DOSGuard{config_, whitelistHandler, dosguardWeights};
auto dosGuard = web::dosguard::DOSGuard{config_, *whitelistHandler, dosguardWeights};
auto sweepHandler = web::dosguard::IntervalSweepHandler{config_, ioc, dosGuard};
auto cache = data::LedgerCache{};
@@ -222,10 +227,15 @@ ClioApplication::run(bool const useNgWebServer)
config_, backend, rpcEngine, etl, dosGuard
);
auto const httpServer = web::makeHttpServer(config_, ioc, dosGuard, handler, cache);
auto const expectedHttpServer = web::makeHttpServer(config_, ioc, dosGuard, handler, cache);
if (not expectedHttpServer.has_value()) {
LOG(util::LogService::fatal()) << expectedHttpServer.error();
return EXIT_FAILURE;
}
appStopper_.setOnStop(
Stopper::makeOnStopCallback(
*httpServer,
**expectedHttpServer,
*balancer,
*etl,
*subscriptions,

View File

@@ -28,7 +28,7 @@ parseConfig(std::string_view configPath)
}
auto const errors = getClioConfig().parse(json.value());
if (errors.has_value()) {
for (auto const& err : errors.value()) {
for (auto const& err : *errors) {
std::cerr << "Issues found in provided config '" << configPath << "':\n";
std::cerr << err.error << std::endl;
}

View File

@@ -83,7 +83,7 @@ MetricsHandler::operator()(
ASSERT(onTaskComplete.has_value(), "Coroutine group can't be full");
bool const postSuccessful = workQueue_.get().postCoro(
[this, &request, &response, &onTaskComplete = onTaskComplete.value(), &connectionMetadata](
[this, &request, &response, &onTaskComplete = *onTaskComplete, &connectionMetadata]( // NOLINT(bugprone-unchecked-optional-access)
boost::asio::yield_context
) mutable {
auto const maybeHttpRequest = request.asHttpRequest();
@@ -94,7 +94,7 @@ MetricsHandler::operator()(
httpRequest, adminVerifier_->isAdmin(httpRequest, connectionMetadata.ip())
);
ASSERT(maybeResponse.has_value(), "Got unexpected request for Prometheus");
response = web::ng::Response{std::move(maybeResponse).value(), request};
response = web::ng::Response{*std::move(maybeResponse), request};
// notify the coroutine group that the foreign task is done
onTaskComplete();
},
@@ -114,7 +114,7 @@ MetricsHandler::operator()(
coroutineGroup.asyncWait(yield);
ASSERT(response.has_value(), "Woke up coroutine without setting response");
return std::move(response).value();
return *std::move(response); // NOLINT(bugprone-unchecked-optional-access)
}
web::ng::Response
@@ -125,7 +125,7 @@ HealthCheckHandler::operator()(
boost::asio::yield_context
)
{
static constexpr auto kHEALTH_CHECK_HTML = R"html(
static constexpr auto kHealthCheckHtml = R"html(
<!DOCTYPE html>
<html>
<head><title>Test page for Clio</title></head>
@@ -133,7 +133,7 @@ HealthCheckHandler::operator()(
</html>
)html";
return web::ng::Response{boost::beast::http::status::ok, kHEALTH_CHECK_HTML, request};
return web::ng::Response{boost::beast::http::status::ok, kHealthCheckHtml, request};
}
web::ng::Response
@@ -144,7 +144,7 @@ CacheStateHandler::operator()(
boost::asio::yield_context
)
{
static constexpr auto kCACHE_CHECK_LOADED_HTML = R"html(
static constexpr auto kCacheCheckLoadedHtml = R"html(
<!DOCTYPE html>
<html>
<head><title>Cache state</title></head>
@@ -152,7 +152,7 @@ CacheStateHandler::operator()(
</html>
)html";
static constexpr auto kCACHE_CHECK_NOT_LOADED_HTML = R"html(
static constexpr auto kCacheCheckNotLoadedHtml = R"html(
<!DOCTYPE html>
<html>
<head><title>Cache state</title></head>
@@ -161,10 +161,10 @@ CacheStateHandler::operator()(
)html";
if (cache_.get().isFull())
return web::ng::Response{boost::beast::http::status::ok, kCACHE_CHECK_LOADED_HTML, request};
return web::ng::Response{boost::beast::http::status::ok, kCacheCheckLoadedHtml, request};
return web::ng::Response{
boost::beast::http::status::service_unavailable, kCACHE_CHECK_NOT_LOADED_HTML, request
boost::beast::http::status::service_unavailable, kCacheCheckNotLoadedHtml, request
};
}

View File

@@ -22,12 +22,11 @@ namespace cluster {
namespace {
struct JsonFields {
static constexpr std::string_view const kUPDATE_TIME = "update_time";
static constexpr std::string_view const kDB_ROLE = "db_role";
static constexpr std::string_view const kETL_STARTED = "etl_started";
static constexpr std::string_view const kCACHE_IS_FULL = "cache_is_full";
static constexpr std::string_view const kCACHE_IS_CURRENTLY_LOADING =
"cache_is_currently_loading";
static constexpr std::string_view const kUpdateTime = "update_time";
static constexpr std::string_view const kDbRole = "db_role";
static constexpr std::string_view const kEtlStarted = "etl_started";
static constexpr std::string_view const kCacheIsFull = "cache_is_full";
static constexpr std::string_view const kCacheIsCurrentlyLoading = "cache_is_currently_loading";
};
} // namespace
@@ -67,11 +66,11 @@ void
tag_invoke(boost::json::value_from_tag, boost::json::value& jv, ClioNode const& node)
{
jv = {
{JsonFields::kUPDATE_TIME, util::systemTpToUtcStr(node.updateTime, ClioNode::kTIME_FORMAT)},
{JsonFields::kDB_ROLE, static_cast<int64_t>(node.dbRole)},
{JsonFields::kETL_STARTED, node.etlStarted},
{JsonFields::kCACHE_IS_FULL, node.cacheIsFull},
{JsonFields::kCACHE_IS_CURRENTLY_LOADING, node.cacheIsCurrentlyLoading}
{JsonFields::kUpdateTime, util::systemTpToUtcStr(node.updateTime, ClioNode::kTimeFormat)},
{JsonFields::kDbRole, static_cast<int64_t>(node.dbRole)},
{JsonFields::kEtlStarted, node.etlStarted},
{JsonFields::kCacheIsFull, node.cacheIsFull},
{JsonFields::kCacheIsCurrentlyLoading, node.cacheIsCurrentlyLoading}
};
}
@@ -79,16 +78,16 @@ ClioNode
tag_invoke(boost::json::value_to_tag<ClioNode>, boost::json::value const& jv)
{
auto const& obj = jv.as_object();
auto const& updateTimeStr = obj.at(JsonFields::kUPDATE_TIME).as_string();
auto const& updateTimeStr = obj.at(JsonFields::kUpdateTime).as_string();
auto const updateTime =
util::systemTpFromUtcStr(std::string(updateTimeStr), ClioNode::kTIME_FORMAT);
util::systemTpFromUtcStr(std::string(updateTimeStr), ClioNode::kTimeFormat);
if (!updateTime.has_value()) {
throw std::runtime_error("Failed to parse update time");
}
// Each field has a default value for backward compatibility
auto dbRole = ClioNode::DbRole::Fallback;
if (auto const* v = obj.if_contains(JsonFields::kDB_ROLE)) {
if (auto const* v = obj.if_contains(JsonFields::kDbRole)) {
auto const dbRoleValue = v->as_int64();
if (dbRoleValue > static_cast<int64_t>(ClioNode::DbRole::Max))
throw std::runtime_error("Invalid db_role value");
@@ -96,19 +95,18 @@ tag_invoke(boost::json::value_to_tag<ClioNode>, boost::json::value const& jv)
}
auto const etlStarted =
obj.contains(JsonFields::kETL_STARTED) ? obj.at(JsonFields::kETL_STARTED).as_bool() : true;
auto const cacheIsFull = obj.contains(JsonFields::kCACHE_IS_FULL)
? obj.at(JsonFields::kCACHE_IS_FULL).as_bool()
: true;
auto const cacheIsCurrentlyLoading = obj.contains(JsonFields::kCACHE_IS_CURRENTLY_LOADING)
? obj.at(JsonFields::kCACHE_IS_CURRENTLY_LOADING).as_bool()
obj.contains(JsonFields::kEtlStarted) ? obj.at(JsonFields::kEtlStarted).as_bool() : true;
auto const cacheIsFull =
obj.contains(JsonFields::kCacheIsFull) ? obj.at(JsonFields::kCacheIsFull).as_bool() : true;
auto const cacheIsCurrentlyLoading = obj.contains(JsonFields::kCacheIsCurrentlyLoading)
? obj.at(JsonFields::kCacheIsCurrentlyLoading).as_bool()
: false;
return ClioNode{
// Json data doesn't contain uuid so leaving it empty here. It will be filled outside of
// this parsing
.uuid = std::make_shared<boost::uuids::uuid>(),
.updateTime = updateTime.value(),
.updateTime = *updateTime,
.dbRole = dbRole,
.etlStarted = etlStarted,
.cacheIsFull = cacheIsFull,

View File

@@ -19,7 +19,7 @@ struct ClioNode {
/**
* @brief The format of the time to store in the database.
*/
static constexpr char const* kTIME_FORMAT = "%Y-%m-%dT%H:%M:%SZ";
static constexpr char const* kTimeFormat = "%Y-%m-%dT%H:%M:%SZ";
/**
* @brief Database role of a node in the cluster.

View File

@@ -36,8 +36,8 @@ class ClusterCommunicationService : public ClusterCommunicationServiceTag {
CacheLoaderDecider cacheLoaderDecider_;
public:
static constexpr std::chrono::milliseconds kDEFAULT_READ_INTERVAL{1000};
static constexpr std::chrono::milliseconds kDEFAULT_WRITE_INTERVAL{1000};
static constexpr std::chrono::milliseconds kDefaultReadInterval{1000};
static constexpr std::chrono::milliseconds kDefaultWriteInterval{1000};
/**
* @brief Construct a new Cluster Communication Service object.
@@ -52,8 +52,8 @@ public:
std::shared_ptr<data::BackendInterface> backend,
std::unique_ptr<etl::WriterStateInterface> writerState,
std::unique_ptr<data::LedgerCacheLoadingStateInterface> cacheLoadingState,
std::chrono::steady_clock::duration readInterval = kDEFAULT_READ_INTERVAL,
std::chrono::steady_clock::duration writeInterval = kDEFAULT_WRITE_INTERVAL
std::chrono::steady_clock::duration readInterval = kDefaultReadInterval,
std::chrono::steady_clock::duration writeInterval = kDefaultWriteInterval
);
~ClusterCommunicationService() override;

View File

@@ -63,7 +63,7 @@ namespace cluster {
*/
class WriterDecider {
public:
static constexpr std::chrono::steady_clock::duration kRECOVERY_TIME = std::chrono::hours{1};
static constexpr std::chrono::steady_clock::duration kRecoveryTime = std::chrono::hours{1};
private:
/** @brief Thread pool for spawning asynchronous tasks */
@@ -90,12 +90,12 @@ public:
* @param ctx Thread pool for executing asynchronous operations
* @param writerState Writer state interface for controlling write operations
* @param recoveryTime How long to wait in Fallback before attempting recovery
* (defaults to `kRECOVERY_TIME`; pass a short duration in tests)
* (defaults to `kRecoveryTime`; pass a short duration in tests)
*/
WriterDecider(
boost::asio::thread_pool& ctx,
std::unique_ptr<etl::WriterStateInterface> writerState,
std::chrono::steady_clock::duration recoveryTime = kRECOVERY_TIME
std::chrono::steady_clock::duration recoveryTime = kRecoveryTime
);
/**

View File

@@ -33,8 +33,8 @@ namespace {
std::unordered_set<std::string>&
supportedAmendments()
{
static std::unordered_set<std::string> kAMENDMENTS = {};
return kAMENDMENTS;
static std::unordered_set<std::string> kAmendments = {};
return kAmendments;
}
bool

View File

@@ -159,6 +159,7 @@ struct Amendments {
REGISTER(fix1523);
REGISTER(fix1528);
REGISTER(fixBatchInnerSigs);
REGISTER(fixCleanup3_1_3);
// NOLINTEND(readability-identifier-naming)
/** @endcond */
};

View File

@@ -17,7 +17,7 @@ namespace data {
namespace {
std::vector<std::int64_t> const kHISTOGRAM_BUCKETS{1, 2, 5, 10, 20, 50, 100, 200, 500, 700, 1000};
std::vector<std::int64_t> const kHistogramBuckets{1, 2, 5, 10, 20, 50, 100, 200, 500, 700, 1000};
std::int64_t
durationInMillisecondsSince(std::chrono::steady_clock::time_point const startTime)
@@ -60,7 +60,7 @@ BackendCounters::BackendCounters()
PrometheusService::histogramInt(
"backend_duration_milliseconds_histogram",
Labels({Label{"operation", "read"}}),
kHISTOGRAM_BUCKETS,
kHistogramBuckets,
"The duration of backend read operations including retries"
)
)
@@ -68,7 +68,7 @@ BackendCounters::BackendCounters()
PrometheusService::histogramInt(
"backend_duration_milliseconds_histogram",
Labels({Label{"operation", "write"}}),
kHISTOGRAM_BUCKETS,
kHistogramBuckets,
"The duration of backend write operations including retries"
)
)

View File

@@ -130,7 +130,7 @@ public:
*
* @return The report
*/
boost::json::object
[[nodiscard]] boost::json::object
report() const;
private:
@@ -152,7 +152,7 @@ private:
void
registerError(std::uint64_t count);
boost::json::object
[[nodiscard]] boost::json::object
report() const;
private:

View File

@@ -188,7 +188,7 @@ BackendInterface::fetchBookOffers(
auto mid2 = std::chrono::system_clock::now();
numSucc++;
succMillis += getMillis(mid2 - mid1);
if (!offerDir || offerDir->key >= bookEnd) {
if (not offerDir.has_value() || offerDir->key >= bookEnd) {
LOG(log_.trace()) << "offerDir.has_value() " << offerDir.has_value() << " breaking";
break;
}
@@ -208,8 +208,10 @@ BackendInterface::fetchBookOffers(
auto nextKey = ripple::keylet::page(uTipIndex, next);
auto nextDir = fetchLedgerObject(nextKey.key, ledgerSequence, yield);
ASSERT(nextDir.has_value(), "Next dir must exist");
// NOLINTBEGIN(bugprone-unchecked-optional-access)
offerDir->blob = *nextDir;
offerDir->key = nextKey.key;
// NOLINTEND(bugprone-unchecked-optional-access)
}
auto mid3 = std::chrono::system_clock::now();
pageMillis += getMillis(mid3 - mid2);
@@ -309,9 +311,10 @@ BackendInterface::fetchLedgerPage(
ripple::uint256 const& curCursor = [&]() {
if (!keys.empty())
return keys.back();
return (cursor ? *cursor : kFIRST_KEY);
return (cursor ? *cursor : kFirstKey);
}();
// NOLINTNEXTLINE(bugprone-unchecked-optional-access)
std::uint32_t const seq = outOfOrder ? range_->maxSequence : ledgerSequence;
auto succ = fetchSuccessorKey(curCursor, seq, yield);

View File

@@ -42,14 +42,14 @@ public:
/**
* @return The error message as a C string
*/
char const*
[[nodiscard]] char const*
what() const throw() override
{
return "Database read timed out. Please retry the request";
}
};
static constexpr std::size_t kDEFAULT_WAIT_BETWEEN_RETRY = 500;
static constexpr std::size_t kDefaultWaitBetweenRetry = 500;
/**
* @brief A helper function that catches DatabaseTimeout exceptions and retries indefinitely.
*
@@ -60,7 +60,7 @@ static constexpr std::size_t kDEFAULT_WAIT_BETWEEN_RETRY = 500;
*/
template <typename FnType>
auto
retryOnTimeout(FnType func, size_t waitMs = kDEFAULT_WAIT_BETWEEN_RETRY)
retryOnTimeout(FnType func, size_t waitMs = kDefaultWaitBetweenRetry)
{
static util::Logger const log{"Backend"}; // NOLINT(readability-identifier-naming)
@@ -408,6 +408,60 @@ public:
boost::asio::yield_context yield
) const = 0;
/**
* @brief Fetches transactions for a particular MPTokenIssuance ID.
*
* Returns one page of transactions for this issuance, newest-first (or oldest-first when
* @p forward is set). Each row of the mptoken_issuance_transactions table holds only a
* transaction hash and its ledger position, not the transaction itself or its type. So the
* query cannot filter by type. When a type filter is requested, the handler looks up each full
* transaction by hash and drops rows whose type does not match, the same way account_tx does.
*
* @param mptIssuanceID The 24-byte MPTokenIssuance ID.
* @param limit The maximum number of transactions per result page.
* @param forward Whether to fetch the page forwards or backwards from the given cursor.
* @param cursorIn The cursor to resume fetching from.
* @param yield The coroutine context.
* @return Results and a cursor to resume from.
*/
virtual TransactionsAndCursor
fetchMPTokenIssuanceTransactions(
ripple::uint192 const& mptIssuanceID,
std::uint32_t limit,
bool forward,
std::optional<TransactionsCursor> const& cursorIn,
boost::asio::yield_context yield
) const = 0;
/**
* @brief Fetches transactions for a particular MPTokenIssuance ID involving a particular
* account.
*
* Returns one page of transactions for this issuance and account, newest-first (or oldest-first
* when @p forward is set). Each row of the account_mptoken_issuance_transactions table holds
* only a transaction hash and its ledger position, not the transaction itself or its type. So
* the query cannot filter by type. When a type filter is requested, the handler looks up each
* full transaction by hash and drops rows whose type does not match, the same way account_tx
* does.
*
* @param mptIssuanceID The 24-byte MPTokenIssuance ID.
* @param account The account that must be affected by the transaction.
* @param limit The maximum number of transactions per result page.
* @param forward Whether to fetch the page forwards or backwards from the given cursor.
* @param cursorIn The cursor to resume fetching from.
* @param yield The coroutine context.
* @return Results and a cursor to resume from.
*/
virtual TransactionsAndCursor
fetchAccountMPTokenIssuanceTransactions(
ripple::uint192 const& mptIssuanceID,
ripple::AccountID const& account,
std::uint32_t limit,
bool forward,
std::optional<TransactionsCursor> const& cursorIn,
boost::asio::yield_context yield
) const = 0;
/**
* @brief Fetches a specific ledger object.
*
@@ -725,6 +779,28 @@ public:
virtual void
writeNFTTransactions(std::vector<NFTTransactionsData> const& data) = 0;
/**
* @brief Write MPTokenIssuance transaction index rows to the `mptoken_issuance_transactions`
* table.
*
* @param data A vector of MPTokenIssuanceTransactionsData objects.
*/
virtual void
writeMPTokenIssuanceTransactions(std::vector<MPTokenIssuanceTransactionsData> const& data) = 0;
/**
* @brief Write MPTokenIssuance transaction index rows to the
* `account_mptoken_issuance_transactions` table.
*
* One row is written per affected account in each record.
*
* @param data A vector of MPTokenIssuanceTransactionsData objects.
*/
virtual void
writeAccountMPTokenIssuanceTransactions(
std::vector<MPTokenIssuanceTransactionsData> const& data
) = 0;
/**
* @brief Write accounts that started holding onto a MPT.
*

View File

@@ -99,7 +99,7 @@ public:
return true;
}
NFTsAndCursor
[[nodiscard]] NFTsAndCursor
fetchNFTsByIssuer(
ripple::AccountID const& issuer,
std::optional<std::uint32_t> const& taxon,
@@ -190,7 +190,7 @@ public:
return ret;
}
std::vector<ripple::uint256>
[[nodiscard]] std::vector<ripple::uint256>
fetchAccountRoots(
std::uint32_t number,
std::uint32_t pageSize,

View File

@@ -199,6 +199,20 @@ struct MPTHolderData {
ripple::AccountID holder;
};
/**
* @brief Represents a transaction link for an MPTokenIssuance.
*
* @note Writing one of these records inserts into two tables:
* mptoken_issuance_transactions and account_mptoken_issuance_transactions.
*/
struct MPTokenIssuanceTransactionsData {
ripple::uint192 mptIssuanceID;
boost::container::flat_set<ripple::AccountID> accounts;
std::uint32_t ledgerSequence{};
std::uint32_t transactionIndex{};
ripple::uint256 txHash;
};
/**
* @brief Check whether the supplied object is a dir node.
*
@@ -209,13 +223,13 @@ template <typename T>
inline bool
isDirNode(T const& object)
{
static constexpr auto kMIN_SIZE_REQUIRED = 3;
if (std::size(object) < kMIN_SIZE_REQUIRED)
static constexpr auto kMinSizeRequired = 3;
if (std::size(object) < kMinSizeRequired)
return false;
static constexpr short kDIR_NODE_SPACE_KEY = 0x0064;
static constexpr short kDirNodeSpaceKey = 0x0064;
short const spaceKey = (object.data()[1] << 8) | object.data()[2];
return spaceKey == kDIR_NODE_SPACE_KEY;
return spaceKey == kDirNodeSpaceKey;
}
/**
@@ -246,12 +260,12 @@ template <typename T>
inline ripple::uint256
getBookBase(T const& key)
{
static constexpr size_t kEY_SIZE = 24;
static constexpr size_t kEySize = 24;
ASSERT(key.size() == ripple::uint256::size(), "Invalid key size {}", key.size());
ripple::uint256 ret;
for (size_t i = 0; i < kEY_SIZE; ++i)
for (size_t i = 0; i < kEySize; ++i)
ret.data()[i] = key.data()[i];
return ret;
@@ -270,4 +284,4 @@ uint256ToString(ripple::uint256 const& input)
}
/** @brief The ripple epoch start timestamp. Midnight on 1st January 2000. */
static constexpr std::uint32_t kRIPPLE_EPOCH_START = 946684800;
static constexpr std::uint32_t kRippleEpochStart = 946684800;

View File

@@ -102,7 +102,7 @@ public:
return true;
}
NFTsAndCursor
[[nodiscard]] NFTsAndCursor
fetchNFTsByIssuer(
ripple::AccountID const& issuer,
std::optional<std::uint32_t> const& taxon,
@@ -162,7 +162,7 @@ public:
* @param yield The coroutine context.
* @return A vector of ripple::uint256 representing the account root hashes.
*/
std::vector<ripple::uint256>
[[nodiscard]] std::vector<ripple::uint256>
fetchAccountRoots(
[[maybe_unused]] std::uint32_t number,
[[maybe_unused]] std::uint32_t pageSize,
@@ -175,7 +175,7 @@ public:
}
private:
std::vector<ripple::uint256>
[[nodiscard]] std::vector<ripple::uint256>
fetchNFTIDsByTaxon(
ripple::AccountID const& issuer,
std::uint32_t const taxon,
@@ -198,7 +198,7 @@ private:
return nftIDs;
}
std::vector<ripple::uint256>
[[nodiscard]] std::vector<ripple::uint256>
fetchNFTIDsWithoutTaxon(
ripple::AccountID const& issuer,
std::uint32_t const limit,
@@ -242,7 +242,7 @@ private:
* @brief Takes a list of NFT IDs, fetches their full data, and assembles the final result with
* a cursor.
*/
NFTsAndCursor
[[nodiscard]] NFTsAndCursor
populateNFTsAndCreateCursor(
std::vector<ripple::uint256> const& nftIDs,
std::uint32_t const ledgerSequence,

View File

@@ -55,7 +55,7 @@ public:
* @param seq The sequence to fetch for
* @return If found in cache, will return the cached Blob; otherwise nullopt is returned
*/
virtual std::optional<Blob>
[[nodiscard]] virtual std::optional<Blob>
get(ripple::uint256 const& key, uint32_t seq) const = 0;
/**
@@ -65,7 +65,7 @@ public:
* @param seq The sequence to fetch for
* @return If found in deleted cache, will return the cached Blob; otherwise nullopt is returned
*/
virtual std::optional<Blob>
[[nodiscard]] virtual std::optional<Blob>
getDeleted(ripple::uint256 const& key, uint32_t seq) const = 0;
/**
@@ -77,7 +77,7 @@ public:
* @param seq The sequence to fetch for
* @return If found in cache, will return the cached successor; otherwise nullopt is returned
*/
virtual std::optional<LedgerObject>
[[nodiscard]] virtual std::optional<LedgerObject>
getSuccessor(ripple::uint256 const& key, uint32_t seq) const = 0;
/**
@@ -89,7 +89,7 @@ public:
* @param seq The sequence to fetch for
* @return If found in cache, will return the cached predcessor; otherwise nullopt is returned
*/
virtual std::optional<LedgerObject>
[[nodiscard]] virtual std::optional<LedgerObject>
getPredecessor(ripple::uint256 const& key, uint32_t seq) const = 0;
/**
@@ -101,7 +101,7 @@ public:
/**
* @return true if the cache is disabled; false otherwise
*/
virtual bool
[[nodiscard]] virtual bool
isDisabled() const = 0;
/**
@@ -117,33 +117,33 @@ public:
/**
* @return The latest ledger sequence for which cache is available.
*/
virtual uint32_t
[[nodiscard]] virtual uint32_t
latestLedgerSequence() const = 0;
/**
* @return true if the cache has all data for the most recent ledger; false otherwise
*/
virtual bool
[[nodiscard]] virtual bool
isFull() const = 0;
/**
* @return The total size of the cache.
*/
virtual size_t
[[nodiscard]] virtual size_t
size() const = 0;
/**
* @return A number representing the success rate of hitting an object in the cache versus
* missing it.
*/
virtual float
[[nodiscard]] virtual float
getObjectHitRate() const = 0;
/**
* @return A number representing the success rate of hitting a successor in the cache versus
* missing it.
*/
virtual float
[[nodiscard]] virtual float
getSuccessorHitRate() const = 0;
/**

View File

@@ -312,13 +312,13 @@ struct AmendmentKey {
operator<=>(AmendmentKey const& other) const = default;
};
constexpr ripple::uint256 kFIRST_KEY{
constexpr ripple::uint256 kFirstKey{
"0000000000000000000000000000000000000000000000000000000000000000"
};
constexpr ripple::uint256 kLAST_KEY{
constexpr ripple::uint256 kLastKey{
"FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF"
};
constexpr ripple::uint256 kHI192{
constexpr ripple::uint256 kHi192{
"0000000000000000000000000000000000000000000000001111111111111111"
};

View File

@@ -77,6 +77,13 @@ protected:
// TODO: move to interface level
mutable FetchLedgerCacheType ledgerCache_{};
static constexpr std::size_t kTransactionCursorBindIndex = 1;
static constexpr std::size_t kTransactionLimitBindIndex = 2;
static constexpr std::size_t kMPTokenIssuanceTxCursorBindIndex = 1;
static constexpr std::size_t kMPTokenIssuanceTxLimitBindIndex = 2;
static constexpr std::size_t kAccountMPTokenIssuanceTxCursorBindIndex = 2;
static constexpr std::size_t kAccountMPTokenIssuanceTxLimitBindIndex = 3;
public:
/**
* @brief Create a new cassandra/scylla backend instance.
@@ -154,14 +161,16 @@ public:
auto cursor = txnCursor;
if (cursor) {
statement.bindAt(1, cursor->asTuple());
statement.bindAt(kTransactionCursorBindIndex, cursor->asTuple());
LOG(log_.debug()) << "account = " << ripple::strHex(account)
<< " tuple = " << cursor->ledgerSequence << cursor->transactionIndex;
} else {
auto const seq = forward ? rng->minSequence : rng->maxSequence;
auto const placeHolder = forward ? 0u : std::numeric_limits<std::uint32_t>::max();
statement.bindAt(1, std::make_tuple(placeHolder, placeHolder));
statement.bindAt(
kTransactionCursorBindIndex, std::make_tuple(placeHolder, placeHolder)
);
LOG(log_.debug()) << "account = " << ripple::strHex(account) << " idx = " << seq
<< " tuple = " << placeHolder;
}
@@ -169,7 +178,7 @@ public:
// FIXME: Limit is a hack to support uint32_t properly for the time
// being. Should be removed later and schema updated to use proper
// types.
statement.bindAt(2, Limit{limit});
statement.bindAt(kTransactionLimitBindIndex, Limit{limit});
auto const res = executor_.read(yield, statement);
auto const& results = res.value();
if (not results.hasRows()) {
@@ -435,19 +444,21 @@ public:
auto cursor = cursorIn;
if (cursor) {
statement.bindAt(1, cursor->asTuple());
statement.bindAt(kTransactionCursorBindIndex, cursor->asTuple());
LOG(log_.debug()) << "token_id = " << ripple::strHex(tokenID)
<< " tuple = " << cursor->ledgerSequence << cursor->transactionIndex;
} else {
auto const seq = forward ? rng->minSequence : rng->maxSequence;
auto const placeHolder = forward ? 0 : std::numeric_limits<std::uint32_t>::max();
statement.bindAt(1, std::make_tuple(placeHolder, placeHolder));
statement.bindAt(
kTransactionCursorBindIndex, std::make_tuple(placeHolder, placeHolder)
);
LOG(log_.debug()) << "token_id = " << ripple::strHex(tokenID) << " idx = " << seq
<< " tuple = " << placeHolder;
}
statement.bindAt(2, Limit{limit});
statement.bindAt(kTransactionLimitBindIndex, Limit{limit});
auto const res = executor_.read(yield, statement);
auto const& results = res.value();
@@ -485,6 +496,59 @@ public:
return {txns, {}};
}
TransactionsAndCursor
fetchMPTokenIssuanceTransactions(
ripple::uint192 const& mptIssuanceID,
std::uint32_t const limit,
bool const forward,
std::optional<TransactionsCursor> const& cursorIn,
boost::asio::yield_context yield
) const override
{
auto const statement = [this, forward, &mptIssuanceID]() {
if (forward)
return schema_->selectMPTokenIssuanceTxForward.bind(mptIssuanceID);
return schema_->selectMPTokenIssuanceTx.bind(mptIssuanceID);
}();
return fetchMPTokenIssuanceTransactionsImpl(
statement,
kMPTokenIssuanceTxCursorBindIndex,
kMPTokenIssuanceTxLimitBindIndex,
limit,
forward,
cursorIn,
yield
);
}
TransactionsAndCursor
fetchAccountMPTokenIssuanceTransactions(
ripple::uint192 const& mptIssuanceID,
ripple::AccountID const& account,
std::uint32_t const limit,
bool const forward,
std::optional<TransactionsCursor> const& cursorIn,
boost::asio::yield_context yield
) const override
{
auto const statement = [this, forward, &mptIssuanceID, &account]() {
if (forward)
return schema_->selectAccountMPTokenIssuanceTxForward.bind(mptIssuanceID, account);
return schema_->selectAccountMPTokenIssuanceTx.bind(mptIssuanceID, account);
}();
return fetchMPTokenIssuanceTransactionsImpl(
statement,
kAccountMPTokenIssuanceTxCursorBindIndex,
kAccountMPTokenIssuanceTxLimitBindIndex,
limit,
forward,
cursorIn,
yield
);
}
MPTHoldersAndCursor
fetchMPTHolders(
ripple::uint192 const& mptID,
@@ -564,7 +628,7 @@ public:
<< ", key = " << ripple::to_string(key);
if (auto const res = executor_.read(yield, schema_->selectObject, key, sequence); res) {
if (auto const result = res->template get<Blob, std::uint32_t>(); result) {
auto [_, seq] = result.value();
auto [_, seq] = *result;
return seq;
}
LOG(log_.debug()) << "Could not fetch ledger object sequence - no rows";
@@ -603,7 +667,7 @@ public:
if (auto const res = executor_.read(yield, schema_->selectSuccessor, key, ledgerSequence);
res) {
if (auto const result = res->template get<ripple::uint256>(); result) {
if (*result == kLAST_KEY)
if (*result == kLastKey)
return std::nullopt;
return result;
}
@@ -877,6 +941,55 @@ public:
executor_.write(std::move(statements));
}
void
writeMPTokenIssuanceTransactions(
std::vector<MPTokenIssuanceTransactionsData> const& data
) override
{
std::vector<Statement> statements;
statements.reserve(data.size());
std::ranges::transform(data, std::back_inserter(statements), [this](auto const& record) {
return schema_->insertMPTokenIssuanceTx.bind(
record.mptIssuanceID,
std::make_tuple(record.ledgerSequence, record.transactionIndex),
record.txHash
);
});
executor_.write(std::move(statements));
}
void
writeAccountMPTokenIssuanceTransactions(
std::vector<MPTokenIssuanceTransactionsData> const& data
) override
{
std::size_t numStatements = 0u;
for (auto const& record : data)
numStatements += record.accounts.size();
std::vector<Statement> statements;
statements.reserve(numStatements);
for (auto const& record : data) {
std::ranges::transform(
record.accounts,
std::back_inserter(statements),
[this, &record](auto const& account) {
return schema_->insertAccountMPTokenIssuanceTx.bind(
record.mptIssuanceID,
account,
std::make_tuple(record.ledgerSequence, record.transactionIndex),
record.txHash
);
}
);
}
executor_.write(std::move(statements));
}
void
writeTransaction(
std::string&& hash,
@@ -923,14 +1036,15 @@ public:
record.tokenID
));
statements.push_back(schema_->insertNFTURI.bind(
record.tokenID, record.ledgerSequence, record.uri.value()
record.tokenID, record.ledgerSequence, *record.uri
));
}
} else {
// only uri changed, we update the uri table only
statements.push_back(schema_->insertNFTURI.bind(
record.tokenID, record.ledgerSequence, record.uri.value()
));
statements.push_back(
// NOLINTNEXTLINE(bugprone-unchecked-optional-access)
schema_->insertNFTURI.bind(record.tokenID, record.ledgerSequence, *record.uri)
);
}
}
@@ -1002,7 +1116,7 @@ protected:
return false;
}
if (not maybeSuccess.value()) {
if (not *maybeSuccess) {
LOG(log_.warn()) << "Update failed. Checking if DB state is what we expect";
// error may indicate that another writer wrote something.
@@ -1015,6 +1129,86 @@ protected:
return true;
}
/**
* @brief Shared implementation of the two MPTokenIssuance transaction-index fetchers.
*
* @note The forward path queries with an inclusive seq_idx >=,
* so the returned cursor's transaction index is advanced
* by one to avoid re-reading the last row on the next page.
*
* @param statement The statement already bound with the partition-key columns.
* @param cursorIdx The bind index for the `seq_idx` cursor tuple.
* @param limitIdx The bind index for the `LIMIT`.
* @param limit The maximum number of transactions per result page.
* @param forward Whether the page is fetched forwards or backwards.
* @param cursorIn The cursor to resume fetching from.
* @param yield The coroutine context.
* @return Results and a cursor to resume from.
*/
TransactionsAndCursor
fetchMPTokenIssuanceTransactionsImpl(
Statement const& statement,
std::size_t const cursorIdx,
std::size_t const limitIdx,
std::uint32_t const limit,
bool const forward,
std::optional<TransactionsCursor> const& cursorIn,
boost::asio::yield_context yield
) const
{
auto rng = fetchLedgerRange();
if (!rng)
return {.txns = {}, .cursor = {}};
auto cursor = cursorIn;
if (cursor.has_value()) {
statement.bindAt(cursorIdx, cursor->asTuple());
} else {
// Forward uses the nft_history-style inclusive lower bound; reverse starts just past
// the latest validated ledger so its exclusive `<` query includes that ledger's rows.
auto const ledgerSequence = forward ? rng->minSequence : rng->maxSequence;
auto const transactionIndex = forward ? 0u : std::numeric_limits<std::uint32_t>::max();
statement.bindAt(cursorIdx, std::make_tuple(ledgerSequence, transactionIndex));
}
statement.bindAt(limitIdx, Limit{limit});
auto const res = executor_.read(yield, statement);
auto const& results = res.value();
if (not results.hasRows()) {
LOG(log_.debug()) << "No rows returned";
return {};
}
std::vector<ripple::uint256> hashes = {};
auto numRows = results.numRows();
for (auto const& [hash, data] :
extract<ripple::uint256, std::tuple<uint32_t, uint32_t>>(results)) {
hashes.push_back(hash);
if (--numRows == 0) {
LOG(log_.debug()) << "Setting cursor";
cursor = data;
// forward queries by ledger/tx sequence `>=`
// so we have to advance the index by one
if (forward)
++cursor->transactionIndex;
}
}
auto txns = fetchTransactions(hashes, yield);
LOG(log_.debug()) << "MPTokenIssuance Txns = " << txns.size();
if (txns.size() == limit) {
LOG(log_.debug()) << "Returning cursor";
return {std::move(txns), cursor};
}
return {std::move(txns), {}};
}
};
} // namespace data::cassandra

View File

@@ -65,7 +65,7 @@ public:
/**
* @return The final error message as a std::string
*/
std::string
[[nodiscard]] std::string
message() const
{
return message_;
@@ -74,7 +74,7 @@ public:
/**
* @return The error code
*/
uint32_t
[[nodiscard]] uint32_t
code() const
{
return code_;
@@ -83,7 +83,7 @@ public:
/**
* @return true if the wrapped error is considered a timeout; false otherwise
*/
bool
[[nodiscard]] bool
isTimeout() const
{
return code_ == CASS_ERROR_LIB_NO_HOSTS_AVAILABLE or
@@ -94,7 +94,7 @@ public:
/**
* @return true if the wrapped error is an invalid query; false otherwise
*/
bool
[[nodiscard]] bool
isInvalidQuery() const
{
return code_ == CASS_ERROR_SERVER_INVALID_QUERY;

View File

@@ -114,7 +114,7 @@ public:
*
* @return Possibly an error
*/
[[maybe_unused]] MaybeErrorType
[[nodiscard]] [[maybe_unused]] MaybeErrorType
disconnect() const;
/**
@@ -162,7 +162,7 @@ public:
* @return The result or an error
*/
template <typename... Args>
[[maybe_unused]] ResultOrErrorType
[[nodiscard]] [[maybe_unused]] ResultOrErrorType
execute(std::string_view query, Args&&... args) const
{
return asyncExecute<Args...>(query, std::forward<Args>(args)...).get();
@@ -189,7 +189,7 @@ public:
* @param statements The statements to execute
* @return Possibly an error
*/
[[maybe_unused]] MaybeErrorType
[[nodiscard]] [[maybe_unused]] MaybeErrorType
executeEach(std::vector<StatementType> const& statements) const;
/**
@@ -250,7 +250,7 @@ public:
* @param statement The statement to execute
* @return The result or an error
*/
[[maybe_unused]] ResultOrErrorType
[[nodiscard]] [[maybe_unused]] ResultOrErrorType
execute(StatementType const& statement) const;
/**
@@ -270,7 +270,7 @@ public:
* @param statements The statements to execute
* @return Possibly an error
*/
[[maybe_unused]] MaybeErrorType
[[nodiscard]] [[maybe_unused]] MaybeErrorType
execute(std::vector<StatementType> const& statements) const;
/**

View File

@@ -278,9 +278,42 @@ public:
R"(
CREATE TABLE IF NOT EXISTS {}
(
mpt_id blob,
holder blob,
PRIMARY KEY (mpt_id, holder)
mptoken_issuance_id blob,
seq_idx tuple<bigint, bigint>,
hash blob,
PRIMARY KEY (mptoken_issuance_id, seq_idx)
)
WITH CLUSTERING ORDER BY (seq_idx DESC)
)",
qualifiedTableName(settingsProvider_.get(), "mptoken_issuance_transactions")
)
);
statements.emplace_back(
fmt::format(
R"(
CREATE TABLE IF NOT EXISTS {}
(
mptoken_issuance_id blob,
account blob,
seq_idx tuple<bigint, bigint>,
hash blob,
PRIMARY KEY ((mptoken_issuance_id, account), seq_idx)
)
WITH CLUSTERING ORDER BY (seq_idx DESC)
)",
qualifiedTableName(settingsProvider_.get(), "account_mptoken_issuance_transactions")
)
);
statements.emplace_back(
fmt::format(
R"(
CREATE TABLE IF NOT EXISTS {}
(
mpt_id blob,
holder blob,
PRIMARY KEY (mpt_id, holder)
)
WITH CLUSTERING ORDER BY (holder ASC)
)",
@@ -474,6 +507,34 @@ public:
);
}();
PreparedStatement insertMPTokenIssuanceTx = [this]() {
return handle_.get().prepare(
fmt::format(
R"(
INSERT INTO {}
(mptoken_issuance_id, seq_idx, hash)
VALUES (?, ?, ?)
)",
qualifiedTableName(settingsProvider_.get(), "mptoken_issuance_transactions")
)
);
}();
PreparedStatement insertAccountMPTokenIssuanceTx = [this]() {
return handle_.get().prepare(
fmt::format(
R"(
INSERT INTO {}
(mptoken_issuance_id, account, seq_idx, hash)
VALUES (?, ?, ?, ?)
)",
qualifiedTableName(
settingsProvider_.get(), "account_mptoken_issuance_transactions"
)
)
);
}();
PreparedStatement insertMPTHolder = [this]() {
return handle_.get().prepare(
fmt::format(
@@ -740,6 +801,76 @@ public:
);
}();
PreparedStatement selectMPTokenIssuanceTx = [this]() {
return handle_.get().prepare(
fmt::format(
R"(
SELECT hash, seq_idx
FROM {}
WHERE mptoken_issuance_id = ?
AND seq_idx < ?
ORDER BY seq_idx DESC
LIMIT ?
)",
qualifiedTableName(settingsProvider_.get(), "mptoken_issuance_transactions")
)
);
}();
PreparedStatement selectMPTokenIssuanceTxForward = [this]() {
return handle_.get().prepare(
fmt::format(
R"(
SELECT hash, seq_idx
FROM {}
WHERE mptoken_issuance_id = ?
AND seq_idx >= ?
ORDER BY seq_idx ASC
LIMIT ?
)",
qualifiedTableName(settingsProvider_.get(), "mptoken_issuance_transactions")
)
);
}();
PreparedStatement selectAccountMPTokenIssuanceTx = [this]() {
return handle_.get().prepare(
fmt::format(
R"(
SELECT hash, seq_idx
FROM {}
WHERE mptoken_issuance_id = ?
AND account = ?
AND seq_idx < ?
ORDER BY seq_idx DESC
LIMIT ?
)",
qualifiedTableName(
settingsProvider_.get(), "account_mptoken_issuance_transactions"
)
)
);
}();
PreparedStatement selectAccountMPTokenIssuanceTxForward = [this]() {
return handle_.get().prepare(
fmt::format(
R"(
SELECT hash, seq_idx
FROM {}
WHERE mptoken_issuance_id = ?
AND account = ?
AND seq_idx >= ?
ORDER BY seq_idx ASC
LIMIT ?
)",
qualifiedTableName(
settingsProvider_.get(), "account_mptoken_issuance_transactions"
)
)
);
}();
PreparedStatement selectNFTIDsByIssuerTaxon = [this]() {
return handle_.get().prepare(
fmt::format(

View File

@@ -90,13 +90,13 @@ SettingsProvider::parseSettings() const
if (config_.getValueView("connect_timeout").hasValue()) {
auto const connectTimeoutSecond = config_.get<uint32_t>("connect_timeout");
settings.connectionTimeout =
std::chrono::milliseconds{connectTimeoutSecond * util::kMILLISECONDS_PER_SECOND};
std::chrono::milliseconds{connectTimeoutSecond * util::kMillisecondsPerSecond};
}
if (config_.getValueView("request_timeout").hasValue()) {
auto const requestTimeoutSecond = config_.get<uint32_t>("request_timeout");
settings.requestTimeout =
std::chrono::milliseconds{requestTimeoutSecond * util::kMILLISECONDS_PER_SECOND};
std::chrono::milliseconds{requestTimeoutSecond * util::kMillisecondsPerSecond};
}
settings.certificate = parseOptionalCertificate();

View File

@@ -12,7 +12,7 @@
#include <vector>
namespace {
constexpr auto kBATCH_DELETER = [](CassBatch* ptr) { cass_batch_free(ptr); };
constexpr auto kBatchDeleter = [](CassBatch* ptr) { cass_batch_free(ptr); };
} // namespace
namespace data::cassandra::impl {
@@ -29,7 +29,7 @@ namespace data::cassandra::impl {
* https://docs.datastax.com/en/developer/cpp-driver-dse/1.10/features/basics/batches/index.html
*/
Batch::Batch(std::vector<Statement> const& statements)
: ManagedObject{cass_batch_new(CASS_BATCH_TYPE_UNLOGGED), kBATCH_DELETER}
: ManagedObject{cass_batch_new(CASS_BATCH_TYPE_UNLOGGED), kBatchDeleter}
{
cass_batch_set_is_idempotent(*this, cass_true);

View File

@@ -14,13 +14,13 @@
namespace {
constexpr auto kCLUSTER_DELETER = [](CassCluster* ptr) { cass_cluster_free(ptr); };
constexpr auto kClusterDeleter = [](CassCluster* ptr) { cass_cluster_free(ptr); };
}; // namespace
namespace data::cassandra::impl {
Cluster::Cluster(Settings const& settings) : ManagedObject{cass_cluster_new(), kCLUSTER_DELETER}
Cluster::Cluster(Settings const& settings) : ManagedObject{cass_cluster_new(), kClusterDeleter}
{
using std::to_string;
@@ -123,8 +123,8 @@ Cluster::setupContactPoints(Settings::ContactPoints const& points)
}
if (points.port) {
auto const rc = cass_cluster_set_port(*this, points.port.value());
throwErrorIfNeeded(rc, "port", to_string(points.port.value()));
auto const rc = cass_cluster_set_port(*this, *points.port);
throwErrorIfNeeded(rc, "port", to_string(*points.port));
}
}
@@ -158,10 +158,8 @@ Cluster::setupCredentials(Settings const& settings)
if (not settings.username || not settings.password)
return;
LOG(log_.debug()) << "Set credentials; username: " << settings.username.value();
cass_cluster_set_credentials(
*this, settings.username.value().c_str(), settings.password.value().c_str()
);
LOG(log_.debug()) << "Set credentials; username: " << *settings.username;
cass_cluster_set_credentials(*this, settings.username->c_str(), settings.password->c_str());
}
} // namespace data::cassandra::impl

View File

@@ -35,11 +35,11 @@ providerFromString(std::string const& provider)
* @brief Bundles all cassandra settings in one place.
*/
struct Settings {
static constexpr std::size_t kDEFAULT_CONNECTION_TIMEOUT = 10000;
static constexpr uint32_t kDEFAULT_MAX_WRITE_REQUESTS_OUTSTANDING = 10'000;
static constexpr uint32_t kDEFAULT_MAX_READ_REQUESTS_OUTSTANDING = 100'000;
static constexpr std::size_t kDEFAULT_BATCH_SIZE = 20;
static constexpr Provider kDEFAULT_PROVIDER = Provider::Cassandra;
static constexpr std::size_t kDefaultConnectionTimeout = 10000;
static constexpr uint32_t kDefaultMaxWriteRequestsOutstanding = 10'000;
static constexpr uint32_t kDefaultMaxReadRequestsOutstanding = 100'000;
static constexpr std::size_t kDefaultBatchSize = 20;
static constexpr Provider kDefaultProvider = Provider::Cassandra;
/**
* @brief Represents the configuration of contact points for cassandra.
@@ -61,7 +61,7 @@ struct Settings {
/** @brief Connect timeout specified in milliseconds */
std::chrono::milliseconds connectionTimeout =
std::chrono::milliseconds{kDEFAULT_CONNECTION_TIMEOUT};
std::chrono::milliseconds{kDefaultConnectionTimeout};
/** @brief Request timeout specified in milliseconds */
std::chrono::milliseconds requestTimeout = std::chrono::milliseconds{0}; // no timeout at all
@@ -73,19 +73,19 @@ struct Settings {
uint32_t threads = std::thread::hardware_concurrency();
/** @brief The maximum number of outstanding write requests at any given moment */
uint32_t maxWriteRequestsOutstanding = kDEFAULT_MAX_WRITE_REQUESTS_OUTSTANDING;
uint32_t maxWriteRequestsOutstanding = kDefaultMaxWriteRequestsOutstanding;
/** @brief The maximum number of outstanding read requests at any given moment */
uint32_t maxReadRequestsOutstanding = kDEFAULT_MAX_READ_REQUESTS_OUTSTANDING;
uint32_t maxReadRequestsOutstanding = kDefaultMaxReadRequestsOutstanding;
/** @brief The number of connection per host to always have active */
uint32_t coreConnectionsPerHost = 3u;
/** @brief Size of batches when writing */
std::size_t writeBatchSize = kDEFAULT_BATCH_SIZE;
std::size_t writeBatchSize = kDefaultBatchSize;
/** @brief Provider to know if we are using scylladb or keyspace */
Provider provider = kDEFAULT_PROVIDER;
Provider provider = kDefaultProvider;
/** @brief Size of the IO queue */
std::optional<uint32_t> queueSizeIO =

View File

@@ -14,7 +14,7 @@
namespace data::cassandra::impl {
class Collection : public ManagedObject<CassCollection> {
static constexpr auto kDELETER = [](CassCollection* ptr) { cass_collection_free(ptr); };
static constexpr auto kDeleter = [](CassCollection* ptr) { cass_collection_free(ptr); };
static void
throwErrorIfNeeded(CassError const rc, std::string_view const label)
@@ -30,7 +30,7 @@ public:
template <typename Type>
explicit Collection(std::vector<Type> const& value)
: ManagedObject{cass_collection_new(CASS_COLLECTION_TYPE_LIST, value.size()), kDELETER}
: ManagedObject{cass_collection_new(CASS_COLLECTION_TYPE_LIST, value.size()), kDeleter}
{
bind(value);
}

Some files were not shown because too many files have changed in this diff Show More