From efcbbd8f4817ab1aca739c8482f6e58a75ba993c Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Mon, 22 Nov 2021 15:32:29 -0800 Subject: [PATCH 1/2] Shards updates for 1.8.0 --- .../ledger-history/history-sharding.md | 17 +-- .../admin-rippled-methods.md | 3 +- .../crawl_shards.md | 9 +- .../node_to_shard.md | 122 ++++++++++++++++++ dactyl-config.yml | 5 + 5 files changed, 143 insertions(+), 13 deletions(-) create mode 100644 content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/node_to_shard.md diff --git a/content/concepts/the-rippled-server/ledger-history/history-sharding.md b/content/concepts/the-rippled-server/ledger-history/history-sharding.md index ec68d302ab..0be42715d4 100644 --- a/content/concepts/the-rippled-server/ledger-history/history-sharding.md +++ b/content/concepts/the-rippled-server/ledger-history/history-sharding.md @@ -10,11 +10,9 @@ labels: [Introduced in: rippled 0.90.0][] -As servers run, they naturally produce a database containing data about the ledgers they witnessed or acquired during network runtime. Each `rippled` server stores that ledger data in its ledger store, but the online delete logic rotates these databases when the number of stored ledgers exceeds configured space limitations. +As XRP Ledger servers run, they naturally produce a database containing data about the ledgers they witnessed or acquired during network runtime. Each server stores that ledger data in its _ledger store_, but [online deletion](online-deletion.html) removes old ledgers' data automatically over time. History sharding provides a separate storage system for older ledger history so that the network can divide up the work of recording the entire (multiple terabyte) history of the XRP Ledger. -Historical sharding distributes the transaction history of the XRP Ledger into segments, called shards, across servers in the XRP Ledger network. A shard is a range of ledgers. A `rippled` server stores ledgers in both the ledger store and the shard store in the same way. - -Using the history sharding feature, individual `rippled` servers can contribute to storing historical data without needing to store the entire (multiple terabyte) history. A shard store does not replace a ledger store, but implements a reliable path towards distributed ledger history across the XRP Ledger Network. +Historical sharding distributes the transaction history of the XRP Ledger into segments, called shards, across servers in the XRP Ledger network. A shard is a range of ledgers. A server uses mostly the same format for ledgers in both the ledger store and the shard store, but the two stores are separate. [![XRP Ledger Network: Ledger Store and Shard Store Diagram](img/xrp-ledger-network-ledger-store-and-shard-store.png)](img/xrp-ledger-network-ledger-store-and-shard-store.png) @@ -22,15 +20,18 @@ Using the history sharding feature, individual `rippled` servers can contribute ## Acquiring and Sharing History Shards -`rippled` servers acquire and store history shards only if configured to do so. For those servers, acquiring shards begins after synchronizing with the network and backfilling ledger history to the configured number of recent ledgers. During this time of lower network activity, a `rippled` server set to maintain a `shard_db` randomly chooses a shard to add to its shard store. To increase the probability for an even distribution of the network ledger history, shards are randomly selected for acquisition, and the current shard is given no special consideration. +Servers acquire and store history shards only if configured to do so. Acquiring shards begins after synchronizing with the network and backfilling ledger history to the configured number of recent ledgers. During this time of lower network activity, a server set to maintain a shard database randomly chooses a shard to add to its shard store. To increase the probability for an even distribution of the network ledger history, shards are randomly selected for acquisition, and the most recent shard is given no special consideration. -Once a shard is selected, the ledger acquire process begins by fetching the sequence of the last ledger in the shard and working backwards toward the first. The retrieval process begins with the server checking for the data locally. For data that is not available, the server requests data from its peer `rippled` servers. Those servers that have the data available for the requested period respond with their history. The requesting server combines those responses to create the shard. The shard is complete when it contains all the ledgers in a specific range. +Once a shard is selected, the ledger acquire process begins by fetching the sequence of the last ledger in the shard and working backwards toward the first. The retrieval process begins with the server checking for the data locally. For data that is not available, the server requests data from its peers. Those servers that have the data available for the requested period respond with their history. The requesting server combines those responses to create the shard. The shard is complete when it contains all the ledgers in a specific range. -If a `rippled` server runs out of space before completely acquiring a shard, it stops its retrieval process until it has space available to continue. After that point, the most recently completed shard may replace an older shard. If there is sufficient disk space, the `rippled` server acquires additional randomly selected shards to add to the shard store until reaching the maximum allocated disk space for shards (`max_size_gb`). +The server selects and downloads additional shards until it reaches the maximum number of shards it is configured to store. If a server runs out of space before completely acquiring a shard, it stops its retrieval process until it has space available to continue. ## XRP Ledger Network Data Integrity -The history of all ledgers is shared by servers agreeing to keep particular ranges of historical ledgers. This makes it possible for servers to confirm that they have all the data they agreed to maintain, and produce proof trees or ledger deltas. Since `rippled` servers that are configured with history sharding randomly select the shards that they store, the entire history of all closed ledgers is stored in a normal distribution curve, increasing the probability that the XRP Ledger Network evenly maintains the history. +The history of all ledgers is shared by servers agreeing to keep particular ranges of historical ledgers. This makes it possible for servers to confirm that they have all the data they agreed to maintain, and produce "proof trees" or "ledger deltas" which shows how each ledger in the blockchain's history was the result of applying transactions to the previous state. Since servers that are configured with history sharding randomly select the shards that they store, the entire history of all closed ledgers is stored in a normal distribution curve, increasing the probability that the XRP Ledger Network evenly maintains the history. + +History shards are recorded in a deterministic format, so that any two servers assembling the same shard produce binary-identical data regardless of what order they acquired the data and where they got it from. This makes it possible to compare checksums or cryptographic hashes of the shard data to verify the integrity of the data, and it is possible to share and import history shards through other formats. (For example, you could download shard data using Bittorrent or acquire physical media with the shard data pre-loaded on it, and verify that it matches the data that can be downloaded from the network.) [New in: rippled 1.8.0][] + ## See Also diff --git a/content/references/rippled-api/admin-rippled-methods/admin-rippled-methods.md b/content/references/rippled-api/admin-rippled-methods/admin-rippled-methods.md index 3b8a2d64e2..8abd572383 100644 --- a/content/references/rippled-api/admin-rippled-methods/admin-rippled-methods.md +++ b/content/references/rippled-api/admin-rippled-methods/admin-rippled-methods.md @@ -30,6 +30,7 @@ Use these methods to manage log levels and other data, such as ledgers. * **[`ledger_request`](ledger_request.html)** - Query a peer server for a specific ledger version. * **[`log_level`](log_level.html)** - Get or modify log verbosity. * **[`logrotate`](logrotate.html)** - Reopen the log file. +* **[`node_to_shard`](node_to_shard.html)** - Copy data from the ledger store to the shard store. ## [Server Control Methods](server-control-methods.html) @@ -41,7 +42,7 @@ Use these methods to manage the `rippled` server. ## [Signing Methods](signing-methods.html) -Use these methods to sign transactions. +Use these methods to sign transactions. * **[`sign`](sign.html)** - Cryptographically sign a transaction. * **[`sign_for`](sign_for.html)** - Contribute to a multi-signature. diff --git a/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md b/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md index b89880b3c4..ee22a9fe74 100644 --- a/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md +++ b/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md @@ -118,19 +118,20 @@ The response follows the [standard format][], with a successful result containin | `Field` | Type | Description | |:------------------|:-------|:------------------------------------------------| | `complete_shards` | String | _(May be omitted)_ The range of [history shards](history-sharding.html) that are available on the local server. This may be an empty string, or a disjointed range. For example, `1-2,5,7-9` indicates that shards 1, 2, 5, 7, 8, and 9 are available. Omitted if this server does not have history sharding enabled. | -| `peers` | Array | List of **Peer Shard Objects** (see below) describing which history shards each peer has available. | +| `peers` | Array | _(May be omitted)_ List of **Peer Shard Objects** (see below) describing which history shards each peer has available. The response omits this field if no peers within the number of hops specified by `limit` have any shards. | #### Peer Shard Objects Each member of the `peers` array of the response is an object that describes one server in the peer-to-peer network. The list only includes peers that have at least one complete [history shard](history-sharding.html) available. Each object in the array has the following fields: - | `Field` | Type | Description | |:----------|:-------|:--------------------------------------------------------| -| `complete_shards` | String | The range of history shards this peer has available. This may be disjointed. For example, `1-2,5,7-9` indicates that shards 1, 2, 5, 7, 8, and 9 are available. | -| `ip` | String | _(May be omitted)_ The IP address of the peer this object describes. This may be an IPv4 or IPv6 address. Omitted if this is a [private peer](peer-protocol.html#private-peers). | +| `complete_shards` | String | The range of complete history shards this peer has available. This may be disjointed. For example, `1-2,5,7-9` indicates that shards 1, 2, 5, 7, 8, and 9 are available. | +| `incomplete_shards` | String | _(May be omitted)_ A comma-separated list of history shards this peer has partially downloaded, and percent completion for each. For example, `1:50,2:25` indicates that shard 1 is 50% downloaded and shard 2 is 25% downloaded. [New in: rippled 1.8.0][] | | `public_key` | String | _(Omitted unless the request specified `"public_key": true`)_ The public key this peer uses for peer-to-peer communications, in the XRP Ledger's [base58 format](base58-encodings.html). | +The `ip` field is no longer provided. [Removed in: rippled 1.8.0][] + ### Possible Errors diff --git a/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/node_to_shard.md b/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/node_to_shard.md new file mode 100644 index 0000000000..6f3beb5dd7 --- /dev/null +++ b/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/node_to_shard.md @@ -0,0 +1,122 @@ +--- +html: node_to_shard.html +parent: logging-and-data-management-methods.html +blurb: Copy data from the ledger store into the shard store. +labels: + - Data Retention +--- +# node_to_shard +[[Source]](https://github.com/undertome/rippled/blob/develop/src/ripple/rpc/handlers/NodeToShard.cpp "Source") + +The `{{currentpage.name}}` method manages copying data from the ledger store to the [shard store](history-sharding.html). It can start, stop, or check the status of copying the data. + +_The `{{currentpage.name}}` method is an [admin method](admin-rippled-methods.html) that cannot be run by unprivileged users._ + + +### Request Format + +An example of the request format: + + + +*WebSocket* + +```json +{ + "command": "{{currentpage.name}}", + "action": "start" +} +``` + +*JSON-RPC* + +```json +{ + "method": "{{currentpage.name}}", + "params": [{ + "action": "start" + }] +} +``` + +*Commandline* + +```sh +#Syntax: {{currentpage.name}} start|stop|status +rippled {{currentpage.name}} start +``` + + + +The request includes the following parameters: + +| `Field` | Type | Description | +|:---------|:-------|:---------------------------------------------------------| +| `action` | String | Either `start`, `stop` or `status` depending on what action to take. | + + +### Response Format + +An example of a successful response: + + + +*WebSocket* + +```json +{ + "result": { + "message": "Database import initiated..." + }, + "status": "success", + "type": "response" +} +``` + +*JSON-RPC* + +```json +{ + "result" : { + "message" : "Database import initiated...", + "status" : "success" + } +} + +``` + +*Commandline* + +```json +Loading: "/etc/rippled.cfg" +Connecting to 127.0.0.1:5005 + +{ + "result" : { + "message" : "Database import initiated...", + "status" : "success" + } +} + +``` + + + +The response follows the [standard format][], with a successful result containing the following fields: + +| `Field` | Type | Description | +|:----------|:-------|:--------------------------------------------------------| +| `message` | String | A human-readable message indicating the action taken in response to the command. | + + +### Possible Errors + +- Any of the [universal error types][]. +- `internal` - If you attempt an invalid operation like checking the status of a copy when one isn't running. +- `notEnabled` - If the server is not configured to store [history shards](history-sharding.html). +- `invalidParams` - One or more fields are specified incorrectly, or one or more required fields are missing. + + +{% include '_snippets/rippled-api-links.md' %} +{% include '_snippets/tx-type-links.md' %} +{% include '_snippets/rippled_versions.md' %} diff --git a/dactyl-config.yml b/dactyl-config.yml index 0ce32208ee..d3d4b2aaf6 100644 --- a/dactyl-config.yml +++ b/dactyl-config.yml @@ -2980,6 +2980,11 @@ pages: targets: - ja + - md: references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/node_to_shard.md + targets: + - en + - ja + #TODO: translate title and blurb - name: Server Control Methods html: server-control-methods.html From c307652e19379b66c4a6b725daa52d560218e130 Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Tue, 23 Nov 2021 11:27:48 -0800 Subject: [PATCH 2/2] Update node_to_shard source code URL --- .../logging-and-data-management-methods/node_to_shard.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/node_to_shard.md b/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/node_to_shard.md index 6f3beb5dd7..1b2bb10d2c 100644 --- a/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/node_to_shard.md +++ b/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/node_to_shard.md @@ -6,7 +6,7 @@ labels: - Data Retention --- # node_to_shard -[[Source]](https://github.com/undertome/rippled/blob/develop/src/ripple/rpc/handlers/NodeToShard.cpp "Source") +[[Source]](https://github.com/ripple/rippled/blob/develop/src/ripple/rpc/handlers/NodeToShard.cpp "Source") The `{{currentpage.name}}` method manages copying data from the ledger store to the [shard store](history-sharding.html). It can start, stop, or check the status of copying the data.