diff --git a/.github/workflows/nightly_notes.md b/.github/workflows/nightly_notes.md index b8e26fd3..8f777788 100644 --- a/.github/workflows/nightly_notes.md +++ b/.github/workflows/nightly_notes.md @@ -1,5 +1,7 @@ +# Release notes + > **Note:** Please remember that this is a development release and it is not recommended for production use. -Changelog (including previous releases): https://github.com/XRPLF/clio/commits/nightly +Changelog (including previous releases): ## SHA256 checksums diff --git a/.markdownlint.yaml b/.markdownlint.yaml new file mode 100644 index 00000000..4bce02df --- /dev/null +++ b/.markdownlint.yaml @@ -0,0 +1,6 @@ +# Default state for all rules +default: true + +# MD013/line-length - Line length +MD013: + line_length: 1000 diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index b1fafbb9..5424c087 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -39,3 +39,9 @@ repos: types: [go] language: golang description: "Runs `gofmt`, requires golang" + + - repo: https://github.com/igorshubovych/markdownlint-cli + rev: v0.44.0 + hooks: + - id: markdownlint-fix + exclude: LICENSE.md diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a11d21c1..72f0e5e6 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,6 +2,8 @@ Thank you for your interest in contributing to the `clio` project 🙏 +## Workflow + To contribute, please: 1. Fork the repository under your own user. @@ -15,7 +17,7 @@ To contribute, please: > **Note:** Please read the [Style guide](#style-guide). -## Install git hooks +### Install git hooks Please run the following command in order to use git hooks that are helpful for `clio` development. @@ -23,14 +25,14 @@ Please run the following command in order to use git hooks that are helpful for git config --local core.hooksPath .githooks ``` -## Git hooks dependencies +### Git hooks dependencies The pre-commit hook requires `clang-format >= 19.0.0` and `cmake-format` to be installed on your machine. `clang-format` can be installed using `brew` on macOS and default package manager on Linux. `cmake-format` can be installed using `pip`. The hook will also attempt to automatically use `doxygen` to verify that everything public in the codebase is covered by doc comments. If `doxygen` is not installed, the hook will raise a warning suggesting to install `doxygen` for future commits. -## Git commands +### Git commands This sections offers a detailed look at the git commands you will need to use to get your PR submitted. Please note that there are more than one way to do this and these commands are provided for your convenience. @@ -75,17 +77,17 @@ git commit --amend -S git push --force ``` -## Use ccache (optional) +### Use ccache (optional) Clio uses `ccache` to speed up compilation. If you want to use it, please make sure it is installed on your machine. CMake will automatically detect it and use it if it is available. -## Opening a pull request +### Opening a pull request When a pull request is open CI will perform checks on the new code. Title of the pull request and squashed commit should follow [conventional commits specification](https://www.conventionalcommits.org/en/v1.0.0/). -## Fixing issues found during code review +### Fixing issues found during code review While your code is in review, it's possible that some changes will be requested by reviewer(s). This section describes the process of adding your fixes. @@ -104,7 +106,7 @@ git commit -S -m "[FOLD] Your commit message" git push ``` -## After code review +### After code review When your PR is approved and ready to merge, use `Squash and merge`. The button for that is near the bottom of the PR's page on GitHub. @@ -112,23 +114,23 @@ The button for that is near the bottom of the PR's page on GitHub. > **Important:** Please leave the automatically-generated mention/link to the PR in the subject line **and** in the description field add `"Fix #ISSUE_ID"` (replacing `ISSUE_ID` with yours) if the PR fixes an issue. > **Note:** See [issues](https://github.com/XRPLF/clio/issues) to find the `ISSUE_ID` for the feature/bug you were working on. -# Style guide +## Style guide This is a non-exhaustive list of recommended style guidelines. These are not always strictly enforced and serve as a way to keep the codebase coherent. -## Formatting +### Formatting Code must conform to `clang-format` version 19, unless the result would be unreasonably difficult to read or maintain. In most cases the pre-commit hook will take care of formatting and will fix any issues automatically. To manually format your code, use `clang-format -i ` for C++ files and `cmake-format -i ` for CMake files. -## Documentation +### Documentation All public namespaces, classes and functions must be covered by doc (`doxygen`) comments. Everything that is not within a nested `impl` namespace is considered public. > **Note:** Keep in mind that this is enforced by Clio's CI and your build will fail if newly added public code lacks documentation. -## Avoid +### Avoid - Proliferation of nearly identical code. - Proliferation of new files and classes unless it improves readability or/and compilation time. @@ -138,7 +140,7 @@ All public namespaces, classes and functions must be covered by doc (`doxygen`) - CPU or architecture-specific code unless there is a good reason to include it, and where it is used guard it with macros and provide explanatory comments. - Importing new libraries unless there is a very good reason to do so. -## Seek to +### Seek to - Extend functionality of existing code rather than creating new code. - Prefer readability over terseness where important logic is concerned. @@ -147,28 +149,28 @@ All public namespaces, classes and functions must be covered by doc (`doxygen`) - Use TitleCase for classes, structs and filenames, camelCase for function and variable names, lower case for namespaces and folders. - Provide as many comments as you feel that a competent programmer would need to understand what your code does. -# Maintainers +## Maintainers Maintainers are ecosystem participants with elevated access to the repository. They are able to push new code, make decisions on when a release should be made, etc. -## Code Review +### Code Review A PR must be reviewed and approved by at least one of the maintainers before it can be merged. -## Adding and Removing +### Adding and Removing New maintainers can be proposed by two existing maintainers, subject to a vote by a quorum of the existing maintainers. A minimum of 50% support and a 50% participation is required. In the event of a tie vote, the addition of the new maintainer will be rejected. Existing maintainers can resign, or be subject to a vote for removal at the behest of two existing maintainers. A minimum of 60% agreement and 50% participation are required. The XRP Ledger Foundation will have the ability, for cause, to remove an existing maintainer without a vote. -## Existing Maintainers +### Existing Maintainers - [godexsoft](https://github.com/godexsoft) (Ripple) - [kuznetsss](https://github.com/kuznetsss) (Ripple) - [legleux](https://github.com/legleux) (Ripple) - [PeterChen13579](https://github.com/PeterChen13579) (Ripple) -## Honorable ex-Maintainers +### Honorable ex-Maintainers - [cindyyan317](https://github.com/cindyyan317) (ex-Ripple) - [cjcobb23](https://github.com/cjcobb23) (ex-Ripple) diff --git a/README.md b/README.md index fa7dae0b..d7629461 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,4 @@ -# Clio +# Clio [![Build status](https://github.com/XRPLF/clio/actions/workflows/build.yml/badge.svg?branch=develop)](https://github.com/XRPLF/clio/actions/workflows/build.yml?query=branch%3Adevelop) [![Nightly release status](https://github.com/XRPLF/clio/actions/workflows/nightly.yml/badge.svg?branch=develop)](https://github.com/XRPLF/clio/actions/workflows/nightly.yml?query=branch%3Adevelop) diff --git a/docs/build-clio.md b/docs/build-clio.md index 0fd0b8d2..5177c6e9 100644 --- a/docs/build-clio.md +++ b/docs/build-clio.md @@ -23,9 +23,9 @@ Clio does not require anything other than `compiler.cppstd=20` in your (`~/.cona > [!NOTE] > Although Clio is built using C++23, it's required to set `compiler.cppstd=20` for the time being as some of Clio's dependencies are not yet capable of building under C++23. -> Mac example: +**Mac example**: -``` +```text [settings] os=Macos os_build=Macos @@ -40,9 +40,9 @@ compiler.cppstd=20 tools.build:cxxflags+=["-DBOOST_ASIO_DISABLE_CONCEPTS"] ``` -> Linux example: +**Linux example**: -``` +```text [settings] os=Linux os_build=Linux @@ -93,6 +93,8 @@ If successful, `conan install` will find the required packages and `cmake` will > [!TIP] > To generate a Code Coverage report, include `-o coverage=True` in the `conan install` command above, along with `-o tests=True` to enable tests. After running the `cmake` commands, execute `make clio_tests-ccov`. The coverage report will be found at `clio_tests-llvm-cov/index.html`. + + > [!NOTE] > If you've built Clio before and the build is now failing, it's likely due to updated dependencies. Try deleting the build folder and then rerunning the Conan and CMake commands mentioned above. diff --git a/docs/config-description.md b/docs/config-description.md index 86a52a7a..2d895563 100644 --- a/docs/config-description.md +++ b/docs/config-description.md @@ -445,7 +445,7 @@ This document provides a list of all available Clio configuration properties in - **Type**: string - **Default value**: `%TimeStamp% (%SourceLocation%) [%ThreadID%] %Channel%:%Severity% %Message%` - **Constraints**: None -- **Description**: The format string for log messages. The format is described here: https://www.boost.org/doc/libs/1_83_0/libs/log/doc/html/log/tutorial/formatters.html. +- **Description**: The format string for log messages. The format is described here: . ### log_to_console diff --git a/docs/trouble_shooting.md b/docs/trouble_shooting.md index aa9f2b19..2c54aa5a 100644 --- a/docs/trouble_shooting.md +++ b/docs/trouble_shooting.md @@ -35,7 +35,7 @@ If you see the error log message `Failed to fetch ETL state from...`, this means If you would like to run Clio without an avaliable rippled node, you can add below setting to Clio's configuration file: -``` +```text "allow_no_etl": true ``` diff --git a/src/data/README.md b/src/data/README.md index 0274dcf1..ab00534c 100644 --- a/src/data/README.md +++ b/src/data/README.md @@ -58,11 +58,11 @@ Their schemas and how they work are detailed in the following sections. ### ledger_transactions -``` +```sql CREATE TABLE clio.ledger_transactions ( - ledger_sequence bigint, # The sequence number of the ledger version - hash blob, # Hash of all the transactions on this ledger version - PRIMARY KEY (ledger_sequence, hash) + ledger_sequence bigint, # The sequence number of the ledger version + hash blob, # Hash of all the transactions on this ledger version + PRIMARY KEY (ledger_sequence, hash) ) WITH CLUSTERING ORDER BY (hash ASC) ... ``` @@ -70,13 +70,13 @@ This table stores the hashes of all transactions in a given ledger sequence and ### transactions -``` +```sql CREATE TABLE clio.transactions ( - hash blob PRIMARY KEY, # The transaction hash - date bigint, # Date of the transaction - ledger_sequence bigint, # The sequence that the transaction was validated - metadata blob, # Metadata of the transaction - transaction blob # Data of the transaction + hash blob PRIMARY KEY, # The transaction hash + date bigint, # Date of the transaction + ledger_sequence bigint, # The sequence that the transaction was validated + metadata blob, # Metadata of the transaction + transaction blob # Data of the transaction ) ... ``` @@ -86,10 +86,10 @@ To lookup all the transactions that were validated in a ledger version with sequ ### ledger_hashes -``` +```sql CREATE TABLE clio.ledger_hashes ( - hash blob PRIMARY KEY, # Hash of entire ledger version's data - sequence bigint # The sequence of the ledger version + hash blob PRIMARY KEY, # Hash of entire ledger version's data + sequence bigint # The sequence of the ledger version ) ... ``` @@ -97,10 +97,10 @@ This table stores the hash of all ledger versions by their sequences. ### ledger_range -``` +```sql CREATE TABLE clio.ledger_range ( - is_latest boolean PRIMARY KEY, # Whether this sequence is the stopping range - sequence bigint # The sequence number of the starting/stopping range + is_latest boolean PRIMARY KEY, # Whether this sequence is the stopping range + sequence bigint # The sequence number of the starting/stopping range ) ... ``` @@ -108,12 +108,12 @@ This table marks the range of ledger versions that is stored on this specific Ca ### objects -``` +```sql CREATE TABLE clio.objects ( - key blob, # Object index of the object - sequence bigint, # The sequence this object was last updated - object blob, # Data of the object - PRIMARY KEY (key, sequence) + key blob, # Object index of the object + sequence bigint, # The sequence this object was last updated + object blob, # Data of the object + PRIMARY KEY (key, sequence) ) WITH CLUSTERING ORDER BY (sequence DESC) ... ``` @@ -123,10 +123,10 @@ The table is updated when all data for a given ledger sequence has been written ### ledgers -``` +```sql CREATE TABLE clio.ledgers ( - sequence bigint PRIMARY KEY, # Sequence of the ledger version - header blob # Data of the header + sequence bigint PRIMARY KEY, # Sequence of the ledger version + header blob # Data of the header ) ... ``` @@ -134,11 +134,11 @@ This table stores the ledger header data of specific ledger versions by their se ### diff -``` +```sql CREATE TABLE clio.diff ( - seq bigint, # Sequence of the ledger version - key blob, # Hash of changes in the ledger version - PRIMARY KEY (seq, key) + seq bigint, # Sequence of the ledger version + key blob, # Hash of changes in the ledger version + PRIMARY KEY (seq, key) ) WITH CLUSTERING ORDER BY (key ASC) ... ``` @@ -146,12 +146,12 @@ This table stores the object index of all the changes in each ledger version. ### account_tx -``` +```sql CREATE TABLE clio.account_tx ( - account blob, - seq_idx frozen>, # Tuple of (ledger_index, transaction_index) - hash blob, # Hash of the transaction - PRIMARY KEY (account, seq_idx) + account blob, + seq_idx frozen>, # Tuple of (ledger_index, transaction_index) + hash blob, # Hash of the transaction + PRIMARY KEY (account, seq_idx) ) WITH CLUSTERING ORDER BY (seq_idx DESC) ... ``` @@ -159,12 +159,12 @@ This table stores the list of transactions affecting a given account. This inclu ### successor -``` +```sql CREATE TABLE clio.successor ( - key blob, # Object index - seq bigint, # The sequnce that this ledger object's predecessor and successor was updated - next blob, # Index of the next object that existed in this sequence - PRIMARY KEY (key, seq) + key blob, # Object index + seq bigint, # The sequnce that this ledger object's predecessor and successor was updated + next blob, # Index of the next object that existed in this sequence + PRIMARY KEY (key, seq) ) WITH CLUSTERING ORDER BY (seq ASC) ... ``` @@ -195,13 +195,13 @@ In `rippled` NFTs are stored in `NFTokenPage` ledger objects. This object is imp ### nf_tokens -``` +```sql CREATE TABLE clio.nf_tokens ( - token_id blob, # The NFT's ID - sequence bigint, # Sequence of ledger version - owner blob, # The account ID of the owner of this NFT at this ledger - is_burned boolean, # True if token was burned in this ledger - PRIMARY KEY (token_id, sequence) + token_id blob, # The NFT's ID + sequence bigint, # Sequence of ledger version + owner blob, # The account ID of the owner of this NFT at this ledger + is_burned boolean, # True if token was burned in this ledger + PRIMARY KEY (token_id, sequence) ) WITH CLUSTERING ORDER BY (sequence DESC) ... ``` @@ -209,7 +209,7 @@ This table indexes NFT IDs with their owner at a given ledger. The example query below shows how you could search for the owner of token `N` at ledger `Y` and see whether the token was burned. -``` +```sql SELECT * FROM nf_tokens WHERE token_id = N AND seq <= Y ORDER BY seq DESC LIMIT 1; @@ -219,12 +219,12 @@ If the token is burned, the owner field indicates the account that owned the tok ### issuer_nf_tokens_v2 -``` +```sql CREATE TABLE clio.issuer_nf_tokens_v2 ( - issuer blob, # The NFT issuer's account ID - taxon bigint, # The NFT's token taxon - token_id blob, # The NFT's ID - PRIMARY KEY (issuer, taxon, token_id) + issuer blob, # The NFT issuer's account ID + taxon bigint, # The NFT's token taxon + token_id blob, # The NFT's ID + PRIMARY KEY (issuer, taxon, token_id) ) WITH CLUSTERING ORDER BY (taxon ASC, token_id ASC) ... ``` @@ -233,12 +233,12 @@ combination. This is useful for determining all the NFTs a specific account issu ### nf_token_uris -``` +```sql CREATE TABLE clio.nf_token_uris ( - token_id blob, # The NFT's ID - sequence bigint, # Sequence of ledger version - uri blob, # The NFT's URI - PRIMARY KEY (token_id, sequence) + token_id blob, # The NFT's ID + sequence bigint, # Sequence of ledger version + uri blob, # The NFT's URI + PRIMARY KEY (token_id, sequence) ) WITH CLUSTERING ORDER BY (sequence DESC) ... ``` @@ -252,12 +252,12 @@ A given NFT will have only one entry in this table (see caveat below), and will ### nf_token_transactions -``` +```sql CREATE TABLE clio.nf_token_transactions ( - token_id blob, # The NFT's ID - seq_idx tuple, # Tuple of (ledger_index, transaction_index) - hash blob, # Hash of the transaction - PRIMARY KEY (token_id, seq_idx) + token_id blob, # The NFT's ID + seq_idx tuple, # Tuple of (ledger_index, transaction_index) + hash blob, # Hash of the transaction + PRIMARY KEY (token_id, seq_idx) ) WITH CLUSTERING ORDER BY (seq_idx DESC) ... ``` @@ -265,7 +265,7 @@ The `nf_token_transactions` table serves as the NFT counterpart to `account_tx`, ### migrator_status -``` +```sql CREATE TABLE clio.migrator_status ( migrator_name TEXT, # The name of the migrator status TEXT, # The status of the migrator diff --git a/src/migration/README.md b/src/migration/README.md index 167503d8..ae01b880 100644 --- a/src/migration/README.md +++ b/src/migration/README.md @@ -6,9 +6,10 @@ Clio maintains the off-chain data of XRPL and multiple indexes tables to powerin Clio provides a migration command-line tool to migrate data in database. -> Note: We need a **configuration file** to run the migration tool. This configuration file has the same format as the configuration file of the Clio server, ensuring consistency and ease of use. It reads the database configuration from the same session as the server's configuration, eliminating the need for separate setup or additional configuration files. Be aware that migration-specific configuration is under `.migration` session. +> [!NOTE] +> We need a **configuration file** to run the migration tool. This configuration file has the same format as the configuration file of the Clio server, ensuring consistency and ease of use. It reads the database configuration from the same session as the server's configuration, eliminating the need for separate setup or additional configuration files. Be aware that migration-specific configuration is under `.migration` session. -### To query migration status: +### Query migration status ./clio_server --migrate status ~/config/migrator.json @@ -17,7 +18,7 @@ This command returns the current migration status of each migrator. The example Current Migration Status: Migrator: ExampleMigrator - Feature v1, Clio v3 - not migrated -### To start a migration: +### Start a migration ./clio_server --migrate ExampleMigrator ~/config/migrator.json @@ -51,7 +52,8 @@ Sometimes migrator isn't able to query the historical data by table's partition Most indexes are based on either ledger states or transactions. We provide the `objects` and `transactions` scanner. Developers only need to implement the callback function to receive the historical data. Please find the examples in `tests/integration/migration/cassandra/ExampleTransactionsMigrator.cpp` and `tests/integration/migration/cassandra/ExampleObjectsMigrator.cpp`. -> **Note** The full table scanner splits the table into multiple ranges by token(https://opensource.docs.scylladb.com/stable/cql/functions.html#token). A few of rows maybe read 2 times if its token happens to be at the edge of ranges. **Deduplication is needed** in the callback function. +> [!NOTE] +> The full table scanner splits the table into multiple ranges by token(). A few of rows maybe read 2 times if its token happens to be at the edge of ranges. **Deduplication is needed** in the callback function. ## How to write a full table scan adapter (Only for Cassandra/ScyllaDB) @@ -68,7 +70,7 @@ If you need to do full scan against other table, you can follow below steps: Please take ObjectsAdapter/TransactionsAdapter as example. -## Examples: +## Examples We have some example migrators under `tests/integration/migration/cassandra` folder. diff --git a/src/util/async/README.md b/src/util/async/README.md index e19f3513..8905ed77 100644 --- a/src/util/async/README.md +++ b/src/util/async/README.md @@ -108,7 +108,7 @@ Note: the original context is taken in by reference. See examples of use below. -#### AnyOperation +#### AnyOperation Wraps any type of operations including regular, stoppable and scheduled. @@ -221,7 +221,7 @@ EXPECT_TRUE(err.message.ends_with("test")); EXPECT_TRUE(std::string{err}.ends_with("test")); ``` -### Strand +### Strand The APIs are basically the same as with the parent `ExecutionContext`. @@ -234,7 +234,7 @@ auto res = strand.execute([] { return 42; }); EXPECT_EQ(res.get().value(), 42); ``` -### Type erasure +### Type erasure #### Simple use diff --git a/src/util/newconfig/ConfigDescription.hpp b/src/util/newconfig/ConfigDescription.hpp index 7bf52f11..3a8f068f 100644 --- a/src/util/newconfig/ConfigDescription.hpp +++ b/src/util/newconfig/ConfigDescription.hpp @@ -266,7 +266,7 @@ private: "explicitly defined logging level."}, KV{.key = "log_format", .value = "The format string for log messages. The format is described here: " - "https://www.boost.org/doc/libs/1_83_0/libs/log/doc/html/log/tutorial/formatters.html."}, + "."}, KV{.key = "log_to_console", .value = "Enables or disables logging to the console."}, KV{.key = "log_directory", .value = "The directory path for the log files."}, KV{.key = "log_rotation_size", diff --git a/tests/unit/README.md b/tests/unit/README.md index 06d17ca4..9a38277f 100644 --- a/tests/unit/README.md +++ b/tests/unit/README.md @@ -6,6 +6,6 @@ The correctness of new implementations can be verified via running unit tests. B To run the unit tests, first build Clio as normal, then execute `./clio_tests` to run all unit tests. -# Adding Unit Tests +## Adding Unit Tests To add unit tests, please create a separate file for the component you are trying to cover (unless it already exists) and use any other existing unit test file as an example.