From 8e64cd42198baf9a8c634f72bd45156faa20e218 Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Thu, 24 Jan 2019 13:37:45 -0800 Subject: [PATCH 1/5] crawl_shards: start drafting method ref --- .../crawl_shards.md | 134 ++++++++++++++++++ 1 file changed, 134 insertions(+) create mode 100644 content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md diff --git a/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md b/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md new file mode 100644 index 0000000000..9b35c0aae3 --- /dev/null +++ b/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md @@ -0,0 +1,134 @@ +# crawl_shards +[[Source]
](https://github.com/ripple/rippled/blob/master/src/ripple/rpc/handlers/CrawlShards.cpp "Source") + +Requests information from peer servers about which [shards of historical ledger data](history-sharding.html) they have available. [New in: rippled 1.2.0][] + +_The `crawl_shards` method is an [admin method](admin-rippled-methods.html) that cannot be run by unprivileged users._ + +### Request Format + +An example of the request format: + + + +*WebSocket* + +```json +{ + "command": "crawl_shards", + "pubkey": true, + "limit": 2 +} +``` + +*JSON-RPC* + +```json +{ + "method": "crawl_shards", + "params": [ + { + "pubkey": true, + "limit": 2 + } + ] +} +``` + + + + +The request includes the following fields: + +| `Field` | Type | Description | +|:---------|:--------|:--------------------------------------------------------| +| `pubkey` | Boolean | _(Optional)_ If `true`, the response includes the node public keys (for peer-to-peer communications) of servers that were crawled. The default is `false`. | +| `limit` | Number | _(Optional)_ How many hops deep to search. The default is 0, which searches direct peers only. With a limit of `1`, searches peers' peers also. (The number of peers searched potentially grows exponentially as `limit` increases.) The maximum value is `3`. | + + +### Response Format + +An example of a successful response: + + + +*WebSocket* + +```json +{ + "result": { + "complete_shards": "1-2,5,8-9,584,1973,2358", + "peers": [ + { + "complete_shards": "1-2,8,47,371,464,554,653,857,1076,1402,1555,1708,1813,1867", + "public_key": "n9LxFZiySnfDSvfh23N94UxsFkCjWyrchTeKHcYE6tJJQL5iejb2" + }, + { + "complete_shards": "8-9,584", + "ip": "192.168.1.132", + "public_key": "n9MN5xwYqbrj64rtfZAXQy7Y3sNxXZJeLt7Lj61a9DYEZ4SE2tQQ" + } + ] + }, + "status": "success", + "type": "response" +} +``` + + +*JSON-RPC* + +```json +200 OK + +{ + "result": { + "complete_shards": "1-2,5,8-9,584,1973,2358", + "peers": [ + { + "complete_shards": "1-2,8,47,371,464,554,653,857,1076,1402,1555,1708,1813,1867", + "public_key": "n9LxFZiySnfDSvfh23N94UxsFkCjWyrchTeKHcYE6tJJQL5iejb2" + }, + { + "complete_shards": "8-9,584", + "ip": "192.168.1.132", + "public_key": "n9MN5xwYqbrj64rtfZAXQy7Y3sNxXZJeLt7Lj61a9DYEZ4SE2tQQ" + } + ], + "status": "success" + } +} +``` + + + + +The response follows the [standard format][], with a successful result containing the following fields: + +| `Field` | Type | Description | +|:------------------|:-------|:------------------------------------------------| +| `complete_shards` | String | The range of history shards that are available across all servers that were searched. This may be disjointed. For example, `1-2,5,7-9` indicates that shards 1, 2, 5, 7, 8, and 9 are available. | +| `peers` | Array | List of **Peer Shard Objects** (see below) describing which history shards each peer has available. Only includes servers that have [history sharding](history-sharding.html) enabled. | + +#### Peer Shard Objects + +Each member of the `peers` array of the response is an object that describes one server in the peer-to-peer network. These objects contain the following fields: + + +| `Field` | Type | Description | +|:----------|:-------|:--------------------------------------------------------| +| `complete_shards` | String | The range of history shards this peer has available. This may be disjointed. For example, `1-2,5,7-9` indicates that shards 1, 2, 5, 7, 8, and 9 are available. | +| `ip` | String | _(May be omitted)_ The IP address of the peer this object describes. This may be an IPv4 or IPv6 address. Omitted if the server is a private peer. | +| `public_key` | String | _(Omitted unless the request specified `"pubkey": true`)_ The public key this peer uses for peer-to-peer communications, in the XRP Ledger's [base58 format](base58-encodings.html). | + + +### Possible Errors + +- Any of the [universal error types][]. +- `invalidParams` - One or more required fields were omitted from the request, or a provided field was specified as the wrong data type. + + + +{% include '_snippets/rippled-api-links.md' %} +{% include '_snippets/tx-type-links.md' %} +{% include '_snippets/rippled_versions.md' %} From 5275cd2b1f46fd4d3889cdfc8bc5778400f8f09c Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Thu, 24 Jan 2019 14:55:29 -0800 Subject: [PATCH 2/5] crawl_shards: add to config, see also, etc. --- content/_snippets/rippled-api-links.md | 2 ++ .../ledger-history/history-sharding.md | 3 ++- .../crawl_shards.md | 6 +++--- dactyl-config.yml | 11 +++++++++++ 4 files changed, 18 insertions(+), 4 deletions(-) diff --git a/content/_snippets/rippled-api-links.md b/content/_snippets/rippled-api-links.md index 00821c0164..7d3b1c3106 100644 --- a/content/_snippets/rippled-api-links.md +++ b/content/_snippets/rippled-api-links.md @@ -46,7 +46,9 @@ "channel_verify", "connect", "consensus_info", + "crawl_shards", "deposit_authorized", + "download_shard", "feature", "fee", "fetch_info", diff --git a/content/concepts/the-rippled-server/ledger-history/history-sharding.md b/content/concepts/the-rippled-server/ledger-history/history-sharding.md index 7483a13fab..07d8130801 100644 --- a/content/concepts/the-rippled-server/ledger-history/history-sharding.md +++ b/content/concepts/the-rippled-server/ledger-history/history-sharding.md @@ -27,7 +27,8 @@ The history of all ledgers is shared by servers agreeing to keep particular rang ## See Also - [Configure History Sharding](configure-history-sharding.html) - +- [download_shard method][] +- [crawl_shards method][] {% include '_snippets/rippled-api-links.md' %} diff --git a/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md b/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md index 9b35c0aae3..615703a6f1 100644 --- a/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md +++ b/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md @@ -108,17 +108,17 @@ The response follows the [standard format][], with a successful result containin | `Field` | Type | Description | |:------------------|:-------|:------------------------------------------------| | `complete_shards` | String | The range of history shards that are available across all servers that were searched. This may be disjointed. For example, `1-2,5,7-9` indicates that shards 1, 2, 5, 7, 8, and 9 are available. | -| `peers` | Array | List of **Peer Shard Objects** (see below) describing which history shards each peer has available. Only includes servers that have [history sharding](history-sharding.html) enabled. | +| `peers` | Array | List of **Peer Shard Objects** (see below) describing which history shards each peer has available. | #### Peer Shard Objects -Each member of the `peers` array of the response is an object that describes one server in the peer-to-peer network. These objects contain the following fields: +Each member of the `peers` array of the response is an object that describes one server in the peer-to-peer network. The list only includes peers that have at least one complete [history shard](history-sharding.html) available. Each object in the array contain the following fields: | `Field` | Type | Description | |:----------|:-------|:--------------------------------------------------------| | `complete_shards` | String | The range of history shards this peer has available. This may be disjointed. For example, `1-2,5,7-9` indicates that shards 1, 2, 5, 7, 8, and 9 are available. | -| `ip` | String | _(May be omitted)_ The IP address of the peer this object describes. This may be an IPv4 or IPv6 address. Omitted if the server is a private peer. | +| `ip` | String | _(May be omitted)_ The IP address of the peer this object describes. This may be an IPv4 or IPv6 address. Omitted if this is a [private peer](peer-protocol.html#private-peers). | | `public_key` | String | _(Omitted unless the request specified `"pubkey": true`)_ The public key this peer uses for peer-to-peer communications, in the XRP Ledger's [base58 format](base58-encodings.html). | diff --git a/dactyl-config.yml b/dactyl-config.yml index 665aa029f3..4a31889284 100644 --- a/dactyl-config.yml +++ b/dactyl-config.yml @@ -1905,6 +1905,17 @@ pages: targets: - local + - md: references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md + html: crawl_shards.html + blurb: Request information about which history shards peers have. + funnel: Docs + doc_type: References + supercategory: rippled API + category: Admin rippled Methods + subcategory: Logging and Data Management Methods + targets: + - local + - md: references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/download_shard.md html: download_shard.html blurb: Download a specific shard of ledger history. From 0ff9f909571b2c71e302bb1837cd3e9cce3e9d70 Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Thu, 24 Jan 2019 16:59:40 -0800 Subject: [PATCH 3/5] crawl_shards: no CLI syntax --- .../logging-and-data-management-methods/crawl_shards.md | 1 + 1 file changed, 1 insertion(+) diff --git a/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md b/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md index 615703a6f1..22a97aa7d7 100644 --- a/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md +++ b/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md @@ -37,6 +37,7 @@ An example of the request format: +**Note:** There is no command-line syntax for this method. Use the [json method][] to access this from the command line. The request includes the following fields: From 591ee65cfbe3e1b6e952aeff39d37e519e0d7e6c Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Tue, 29 Jan 2019 17:27:26 -0800 Subject: [PATCH 4/5] crawl_shards: updates per reviews --- .../logging-and-data-management-methods/crawl_shards.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md b/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md index 22a97aa7d7..99b27309ef 100644 --- a/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md +++ b/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md @@ -108,12 +108,12 @@ The response follows the [standard format][], with a successful result containin | `Field` | Type | Description | |:------------------|:-------|:------------------------------------------------| -| `complete_shards` | String | The range of history shards that are available across all servers that were searched. This may be disjointed. For example, `1-2,5,7-9` indicates that shards 1, 2, 5, 7, 8, and 9 are available. | +| `complete_shards` | String | _(May be omitted)_ The range of [history shards](history-sharding.html) that are available on the local server. This may be disjointed. For example, `1-2,5,7-9` indicates that shards 1, 2, 5, 7, 8, and 9 are available. Omitted if this server does not have any history sharding enabled or does not have any shards available. | | `peers` | Array | List of **Peer Shard Objects** (see below) describing which history shards each peer has available. | #### Peer Shard Objects -Each member of the `peers` array of the response is an object that describes one server in the peer-to-peer network. The list only includes peers that have at least one complete [history shard](history-sharding.html) available. Each object in the array contain the following fields: +Each member of the `peers` array of the response is an object that describes one server in the peer-to-peer network. The list only includes peers that have at least one complete [history shard](history-sharding.html) available. Each object in the array has the following fields: | `Field` | Type | Description | From 5f8df53dd6bea074c24bd56a6674803c649b58d7 Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Wed, 30 Jan 2019 16:16:05 -0800 Subject: [PATCH 5/5] crawl_shards: slight polishing after testing the method --- .../crawl_shards.md | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md b/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md index 99b27309ef..10f610a3c0 100644 --- a/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md +++ b/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/crawl_shards.md @@ -17,7 +17,7 @@ An example of the request format: { "command": "crawl_shards", "pubkey": true, - "limit": 2 + "limit": 0 } ``` @@ -29,7 +29,7 @@ An example of the request format: "params": [ { "pubkey": true, - "limit": 2 + "limit": 0 } ] } @@ -44,7 +44,9 @@ The request includes the following fields: | `Field` | Type | Description | |:---------|:--------|:--------------------------------------------------------| | `pubkey` | Boolean | _(Optional)_ If `true`, the response includes the node public keys (for peer-to-peer communications) of servers that were crawled. The default is `false`. | -| `limit` | Number | _(Optional)_ How many hops deep to search. The default is 0, which searches direct peers only. With a limit of `1`, searches peers' peers also. (The number of peers searched potentially grows exponentially as `limit` increases.) The maximum value is `3`. | +| `limit` | Number | _(Optional)_ How many hops deep to search. The default is 0, which searches direct peers only. With a limit of `1`, searches peers' peers also. The maximum value is `3`. | + +**Caution:** The number of peers potentially searched grows exponentially as `limit` increases. With a limit of 2 or 3, it can take several seconds for the server to respond to the API request. ### Response Format @@ -108,7 +110,7 @@ The response follows the [standard format][], with a successful result containin | `Field` | Type | Description | |:------------------|:-------|:------------------------------------------------| -| `complete_shards` | String | _(May be omitted)_ The range of [history shards](history-sharding.html) that are available on the local server. This may be disjointed. For example, `1-2,5,7-9` indicates that shards 1, 2, 5, 7, 8, and 9 are available. Omitted if this server does not have any history sharding enabled or does not have any shards available. | +| `complete_shards` | String | _(May be omitted)_ The range of [history shards](history-sharding.html) that are available on the local server. This may be an empty string, or a disjointed range. For example, `1-2,5,7-9` indicates that shards 1, 2, 5, 7, 8, and 9 are available. Omitted if this server does not have history sharding enabled. | | `peers` | Array | List of **Peer Shard Objects** (see below) describing which history shards each peer has available. | #### Peer Shard Objects