mirror of
https://github.com/XRPLF/xrpl-dev-portal.git
synced 2025-12-02 09:35:53 +00:00
Migrate content syntax via script
The changes in this commit were auto-generated by running tool/migrate.sh Following this commit, the Dactyl build no longer works but the Redocly build (mostly) should.
This commit is contained in:
mDuo13
@@ -8,7 +8,7 @@ labels:
|
||||
---
|
||||
# Configure Advisory Deletion
|
||||
|
||||
The default config file sets [`rippled`](xrpl-servers.html) to automatically delete outdated [history](ledger-history.html) of XRP Ledger state and transactions as new ledger versions become available. If your server uses most of its hardware resources during peak hours, you can configure the server to delete ledgers only when prompted by a command scheduled to run during off-peak hours, so that online deletion is less likely to impact [server performance](capacity-planning.html).
|
||||
The default config file sets [`rippled`](../../../concepts/networks-and-servers/index.md) to automatically delete outdated [history](../../../concepts/networks-and-servers/ledger-history.md) of XRP Ledger state and transactions as new ledger versions become available. If your server uses most of its hardware resources during peak hours, you can configure the server to delete ledgers only when prompted by a command scheduled to run during off-peak hours, so that online deletion is less likely to impact [server performance](../../installation/capacity-planning.md).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@@ -16,7 +16,7 @@ This tutorial assumes your server meets the following prerequisites:
|
||||
|
||||
- You are on a supported operating system: Ubuntu Linux, Red Hat Enterprise Linux (RHEL), or CentOS.
|
||||
|
||||
- The `rippled` server is already [installed](install-rippled.html) and [online deletion](online-deletion.html) is enabled.
|
||||
- The `rippled` server is already [installed](../../installation/index.md) and [online deletion](online-deletion.md) is enabled.
|
||||
|
||||
The default config file enables online deletion after 2000 ledger versions.
|
||||
|
||||
@@ -26,11 +26,13 @@ This tutorial assumes your server meets the following prerequisites:
|
||||
|
||||
On RHEL or CentOS, you can install the `cronie` package:
|
||||
|
||||
$ sudo yum install cronie
|
||||
```
|
||||
$ sudo yum install cronie
|
||||
```
|
||||
|
||||
- Your server has enough disk space to store your chosen amount of history in its ledger store.
|
||||
|
||||
See [Capacity Planning](capacity-planning.html) for details of how much storage is required for different configurations. With advisory deletion enabled, the maximum history a server may accumulate before deletion is equal to the number of ledger versions configured in the `online_delete` setting **plus** the amount of time between online deletion prompts.
|
||||
See [Capacity Planning](../../installation/capacity-planning.md) for details of how much storage is required for different configurations. With advisory deletion enabled, the maximum history a server may accumulate before deletion is equal to the number of ledger versions configured in the `online_delete` setting **plus** the amount of time between online deletion prompts.
|
||||
|
||||
- You know which hours are least busy for your server.
|
||||
|
||||
@@ -40,30 +42,36 @@ To configure advisory deletion with a daily schedule, perform the following step
|
||||
|
||||
1. Enable `advisory_delete` in the `[node_db]` stanza of your `rippled`'s config file.
|
||||
|
||||
[node_db]
|
||||
# Other settings unchanged ...
|
||||
online_delete=300000
|
||||
advisory_delete=1
|
||||
```
|
||||
[node_db]
|
||||
# Other settings unchanged ...
|
||||
online_delete=300000
|
||||
advisory_delete=1
|
||||
```
|
||||
|
||||
- Set `advisory_delete` to `1` to run online deletion only when prompted. (Set it to `0` to run online deletion automatically as new ledger versions become available.)
|
||||
- Set `online_delete` to the minimum number of ledger versions to keep after running online deletion. The server accumulates more history than this until online deletion runs.
|
||||
|
||||
{% include '_snippets/conf-file-location.md' %}<!--_ -->
|
||||
{% partial file="/_snippets/conf-file-location.md" /%}
|
||||
|
||||
2. Test running the [can_delete method][] to prompt the server to run online deletion.
|
||||
|
||||
You can use the [`rippled` commandline interface](get-started-using-http-websocket-apis.html#commandline) to run this command. For example:
|
||||
You can use the [`rippled` commandline interface](../../../tutorials/get-started/get-started-using-http-websocket-apis.md#commandline) to run this command. For example:
|
||||
|
||||
$ rippled --conf=/etc/opt/ripple/rippled.cfg can_delete now
|
||||
```
|
||||
$ rippled --conf=/etc/opt/ripple/rippled.cfg can_delete now
|
||||
```
|
||||
|
||||
The response indicates the maximum ledger index that the server may delete from its ledger store. For example, the following message indicates that ledger versions up to and including ledger index 43633667 can be deleted:
|
||||
|
||||
{
|
||||
"result": {
|
||||
"can_delete": 43633667,
|
||||
"status": "success"
|
||||
}
|
||||
}
|
||||
```
|
||||
{
|
||||
"result": {
|
||||
"can_delete": 43633667,
|
||||
"status": "success"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The server only deletes those ledger versions if the number of _newer_ validated ledger versions it has is equal to or greater than the `online_delete` setting.
|
||||
|
||||
@@ -71,11 +79,15 @@ To configure advisory deletion with a daily schedule, perform the following step
|
||||
|
||||
Edit your `cron` configuration:
|
||||
|
||||
$ crontab -e
|
||||
```
|
||||
$ crontab -e
|
||||
```
|
||||
|
||||
The following example sets the server to run deletion at 1:05 AM server time daily:
|
||||
|
||||
5 1 * * * rippled --conf /etc/opt/ripple/rippled.cfg can_delete now
|
||||
```
|
||||
5 1 * * * rippled --conf /etc/opt/ripple/rippled.cfg can_delete now
|
||||
```
|
||||
|
||||
Be sure that you schedule the command to run based on your server's configured time zone.
|
||||
|
||||
@@ -83,7 +95,9 @@ To configure advisory deletion with a daily schedule, perform the following step
|
||||
|
||||
4. Start (or restart) the `rippled` service.
|
||||
|
||||
$ sudo systemctl restart rippled
|
||||
```
|
||||
$ sudo systemctl restart rippled
|
||||
```
|
||||
|
||||
5. Periodically check your server's `complete_ledgers` range using the [server_info method][] to confirm that ledgers are being deleted as scheduled.
|
||||
|
||||
@@ -98,24 +112,21 @@ If online deletion does not seem to be running after configuring it, try the fol
|
||||
- Check that the user who configured the `cron` job has permissions to run the `rippled` server as a commandline client.
|
||||
- Check the syntax of your `cron` job and the time when it is supposed to run.
|
||||
- Check that the `rippled` executable is available at the path specified in your `cron` configuration. If necessary, specify the absolute path to the executable, such as `/opt/ripple/bin/rippled`.
|
||||
- Check your `rippled` logs for messages that begin with `SHAMapStore::WRN`. This can indicate that [online deletion is being interrupted](online-deletion.html#interrupting-online-deletion) because your server fell out of sync with the network.
|
||||
- Check your `rippled` logs for messages that begin with `SHAMapStore::WRN`. This can indicate that [online deletion is being interrupted](online-deletion.md#interrupting-online-deletion) because your server fell out of sync with the network.
|
||||
|
||||
## See Also
|
||||
|
||||
- **Concepts:**
|
||||
- [Ledger History](ledger-history.html)
|
||||
- [Online Deletion](online-deletion.html)
|
||||
- [Ledger History](../../../concepts/networks-and-servers/ledger-history.md)
|
||||
- [Online Deletion](online-deletion.md)
|
||||
- **Tutorials:**
|
||||
- [Configure Online Deletion](configure-online-deletion.html)
|
||||
- [Diagnosing Problems with rippled](diagnosing-problems.html)
|
||||
- [Understanding Log Messages](understanding-log-messages.html)
|
||||
- [Configure Online Deletion](configure-online-deletion.md)
|
||||
- [Diagnosing Problems with rippled](../../troubleshooting/diagnosing-problems.md)
|
||||
- [Understanding Log Messages](../../troubleshooting/understanding-log-messages.md)
|
||||
- **References:**
|
||||
- [server_info method][]
|
||||
- [can_delete method][]
|
||||
- [logrotate method][]
|
||||
- [Ledger Data Formats](ledger-data-formats.html)
|
||||
- [Ledger Data Formats](../../../references/protocol/ledger-data/index.md)
|
||||
|
||||
<!--{# common link defs #}-->
|
||||
{% include '_snippets/rippled-api-links.md' %}
|
||||
{% include '_snippets/tx-type-links.md' %}
|
||||
{% include '_snippets/rippled_versions.md' %}
|
||||
{% raw-partial file="/_snippets/common-links.md" /%}
|
||||
|
||||
@@ -20,7 +20,7 @@ Acquiring full history from the peer-to-peer network takes a long time (several
|
||||
|
||||
You do not need a full history server to participate in the network, validate transactions, or know the current state of the network. Full history is only useful for knowing the outcome of transactions that occurred in the past, or the state of the ledger at a given time in the past. To get such information, you must rely on other servers having the history you need.
|
||||
|
||||
If you want to contribute to storing the history of the XRP Ledger network without storing the full history, you can [configure history sharding](configure-history-sharding.html) to store randomly-selected chunks of ledger history instead.
|
||||
If you want to contribute to storing the history of the XRP Ledger network without storing the full history, you can [configure history sharding](configure-history-sharding.md) to store randomly-selected chunks of ledger history instead.
|
||||
|
||||
## Configuration Steps
|
||||
|
||||
@@ -28,32 +28,40 @@ To configure your server to acquire and store full history, complete the followi
|
||||
|
||||
1. Stop the `rippled` server if it is running.
|
||||
|
||||
$ sudo systemctl stop rippled
|
||||
```
|
||||
$ sudo systemctl stop rippled
|
||||
```
|
||||
|
||||
0. Remove (or comment out) the `online_delete` and `advisory_delete` settings from the `[node_db]` stanza of your server's config file, and change the type to `NuDB` if you haven't already:
|
||||
|
||||
[node_db]
|
||||
type=NuDB
|
||||
path=/var/lib/rippled/db/nudb
|
||||
#online_delete=300000
|
||||
#advisory_delete=0
|
||||
```
|
||||
[node_db]
|
||||
type=NuDB
|
||||
path=/var/lib/rippled/db/nudb
|
||||
#online_delete=300000
|
||||
#advisory_delete=0
|
||||
```
|
||||
|
||||
On a full-history server, you should use NuDB for the ledger store, because RocksDB requires too much RAM when the database is that large. For more information, see [Capacity Planning](capacity-planning.html). You can remove the following performance-related configuration options from the default `[node_db]` stanza, because they only apply to RocksDB: `open_files`, `filter_bits`, `cache_mb`, `file_size_mb`, and `file_size_mult.`
|
||||
On a full-history server, you should use NuDB for the ledger store, because RocksDB requires too much RAM when the database is that large. For more information, see [Capacity Planning](../../installation/capacity-planning.md). You can remove the following performance-related configuration options from the default `[node_db]` stanza, because they only apply to RocksDB: `open_files`, `filter_bits`, `cache_mb`, `file_size_mb`, and `file_size_mult.`
|
||||
|
||||
**Caution:** If you have any history already downloaded with RocksDB, you must either delete that data or change the paths to the databases in the config file when you switch to NuDB. You must change both the `path` field of the `[node_db]` stanza **and** the `[database_path]` (SQLite database) setting. Otherwise, the server may [fail to start](server-wont-start.html#state-db-error).
|
||||
**Caution:** If you have any history already downloaded with RocksDB, you must either delete that data or change the paths to the databases in the config file when you switch to NuDB. You must change both the `path` field of the `[node_db]` stanza **and** the `[database_path]` (SQLite database) setting. Otherwise, the server may [fail to start](../../troubleshooting/server-wont-start.md#state-db-error).
|
||||
|
||||
{% include '_snippets/conf-file-location.md' %}<!--_ -->
|
||||
{% partial file="/_snippets/conf-file-location.md" /%}
|
||||
|
||||
0. Set the `[ledger_history]` stanza of your server's config file to `full`:
|
||||
|
||||
[ledger_history]
|
||||
full
|
||||
```
|
||||
[ledger_history]
|
||||
full
|
||||
```
|
||||
|
||||
0. Set the `[ips_fixed]` stanza of your server's config file to explicitly peer with at least one server that has full history available.
|
||||
|
||||
[ips_fixed]
|
||||
169.55.164.20 51235
|
||||
50.22.123.215 51235
|
||||
```
|
||||
[ips_fixed]
|
||||
169.55.164.20 51235
|
||||
50.22.123.215 51235
|
||||
```
|
||||
|
||||
Your server can only download historical data from the peer-to-peer network if one its direct peers has the data available. The easiest way to ensure you can download full history is to peer with a server that already has full history.
|
||||
|
||||
@@ -61,29 +69,37 @@ To configure your server to acquire and store full history, complete the followi
|
||||
|
||||
0. If you have a database dump from another full-history server to use as a basis, set the `[import_db]` stanza of your server's config file to point to the data to be imported. (Otherwise, skip this step.)
|
||||
|
||||
[import_db]
|
||||
type=NuDB
|
||||
path=/tmp/full_history_dump/
|
||||
```
|
||||
[import_db]
|
||||
type=NuDB
|
||||
path=/tmp/full_history_dump/
|
||||
```
|
||||
|
||||
0. Remove your server's existing database files, if you have any from previously running `rippled`.
|
||||
|
||||
After disabling online deletion, the server ignores any data that was downloaded while online deletion was enabled, so you may as well clear up the disk space. For example:
|
||||
|
||||
rm -r /var/lib/rippled/db/*
|
||||
```
|
||||
rm -r /var/lib/rippled/db/*
|
||||
```
|
||||
|
||||
**Warning:** Be sure that you have not put any files you want to keep in the folder before you delete it. It is generally safe to delete all of a `rippled` server's database files, but you should only do this if the configured database folder is not used for anything other than `rippled`'s databases.
|
||||
|
||||
0. Start the `rippled` server, importing the database dump if you have one available:
|
||||
|
||||
If you have a database dump to load configured in `[import_db]`, start the server explicitly and include the `--import` [commandline option](commandline-usage.html#daemon-mode-options):
|
||||
If you have a database dump to load configured in `[import_db]`, start the server explicitly and include the `--import` [commandline option](../../commandline-usage.md#daemon-mode-options):
|
||||
|
||||
$ /opt/ripple/bin/rippled --conf /etc/opt/ripple/rippled.cfg --import
|
||||
```
|
||||
$ /opt/ripple/bin/rippled --conf /etc/opt/ripple/rippled.cfg --import
|
||||
```
|
||||
|
||||
Importing a large database dump may take several minutes or even hours. During this time, the server is not fully started and synced with the network. Watch the server logs to see the status of the import.
|
||||
|
||||
If you are not importing a database dump, start the server normally:
|
||||
|
||||
$ sudo systemctl start rippled
|
||||
```
|
||||
$ sudo systemctl start rippled
|
||||
```
|
||||
|
||||
0. If you added an `[import_db]` stanza to your server's config file, remove it after the import completes.
|
||||
|
||||
@@ -93,25 +109,22 @@ To configure your server to acquire and store full history, complete the followi
|
||||
|
||||
The range of available ledgers reported in the `complete_ledgers` field should increase over time.
|
||||
|
||||
The earliest available ledger version in the production XRP Ledger's history is ledger index **32570**. The first two weeks or so of ledger history was lost due to a bug in the server at the time. [Test nets and other chains](parallel-networks.html) generally have history going back to ledger index **1**.
|
||||
The earliest available ledger version in the production XRP Ledger's history is ledger index **32570**. The first two weeks or so of ledger history was lost due to a bug in the server at the time. [Test nets and other chains](../../../concepts/networks-and-servers/parallel-networks.md) generally have history going back to ledger index **1**.
|
||||
|
||||
## See Also
|
||||
|
||||
- **Concepts:**
|
||||
- [Ledger History](ledger-history.html)
|
||||
- [rippled Server Modes](rippled-server-modes.html)
|
||||
- [Ledger History](../../../concepts/networks-and-servers/ledger-history.md)
|
||||
- [rippled Server Modes](../../../concepts/networks-and-servers/rippled-server-modes.md)
|
||||
- **Tutorials:**
|
||||
- [Capacity Planning](capacity-planning.html), particularly [Disk Space](capacity-planning.html#disk-space)
|
||||
- [Configure Online Deletion](configure-online-deletion.html)
|
||||
- [Diagnosing Problems with rippled](diagnosing-problems.html)
|
||||
- [Understanding Log Messages](understanding-log-messages.html)
|
||||
- [Capacity Planning](../../installation/capacity-planning.md), particularly [Disk Space](../../installation/capacity-planning.md#disk-space)
|
||||
- [Configure Online Deletion](configure-online-deletion.md)
|
||||
- [Diagnosing Problems with rippled](../../troubleshooting/diagnosing-problems.md)
|
||||
- [Understanding Log Messages](../../troubleshooting/understanding-log-messages.md)
|
||||
- **References:**
|
||||
- [server_info method][]
|
||||
- [can_delete method][]
|
||||
- [Ledger Data Formats](ledger-data-formats.html)
|
||||
- [rippled Commandline Usage Reference](commandline-usage.html)
|
||||
- [Ledger Data Formats](../../../references/protocol/ledger-data/index.md)
|
||||
- [rippled Commandline Usage Reference](../../commandline-usage.md)
|
||||
|
||||
<!--{# common link defs #}-->
|
||||
{% include '_snippets/rippled-api-links.md' %}
|
||||
{% include '_snippets/tx-type-links.md' %}
|
||||
{% include '_snippets/rippled_versions.md' %}
|
||||
{% raw-partial file="/_snippets/common-links.md" /%}
|
||||
|
||||
@@ -8,7 +8,7 @@ labels:
|
||||
---
|
||||
# Configure History Sharding
|
||||
|
||||
[History Sharding](history-sharding.html) lets servers contribute to preserving historical XRP Ledger data without each server needing to store the full history. By default, `rippled` servers do not store history shards.
|
||||
[History Sharding](history-sharding.md) lets servers contribute to preserving historical XRP Ledger data without each server needing to store the full history. By default, `rippled` servers do not store history shards.
|
||||
|
||||
**Tip:** While both validator and tracking (or stock) `rippled` servers can be configured to store history shards, Ripple recommends _not_ configuring validator `rippled` servers to store shards, to reduce overhead on those servers. If you run a validator and want to contribute to storing XRP Ledger history, Ripple recommends you run a separate `rippled` server with history sharding enabled.
|
||||
|
||||
@@ -25,7 +25,7 @@ Before you configure your `rippled` server to store history shards, you must dec
|
||||
- The history shard store and the ledger store _MUST_ be stored at different file paths. You can configure the ledger store and history store to be on different disks or partitions if desired.
|
||||
- It is possible but redundant to hold full ledger history in both the ledger store and the history shard store.
|
||||
- The time to acquire a shard, number of file handles needed by the `rippled` server, and memory cache usage is directly affected by the size of the shard.
|
||||
- You can specify additional paths to store older history shards by providing a `[historical_shard_paths]` stanza. These paths may be on different, slower disks because they hold data that is used less often. The most recent two shards (the ones with the largest ledger indexes) are always stored in the path specified in the `[shard_db]` stanza. [New in: rippled 1.7.0][]
|
||||
- You can specify additional paths to store older history shards by providing a `[historical_shard_paths]` stanza. These paths may be on different, slower disks because they hold data that is used less often. The most recent two shards (the ones with the largest ledger indexes) are always stored in the path specified in the `[shard_db]` stanza. {% badge href="https://github.com/XRPLF/rippled/releases/tag/1.7.0" %}New in: rippled 1.7.0{% /badge %}
|
||||
|
||||
## 2. Edit rippled.cfg
|
||||
|
||||
@@ -33,7 +33,7 @@ Before you configure your `rippled` server to store history shards, you must dec
|
||||
|
||||
Edit your `rippled.cfg` file to add a `[shard_db]` stanza and optionally a `[historical_shard_paths]` stanza.
|
||||
|
||||
{% include '_snippets/conf-file-location.md' %}<!--_ -->
|
||||
{% partial file="/_snippets/conf-file-location.md" /%}
|
||||
|
||||
The following snippet shows an example of a `[shard_db]` stanza:
|
||||
|
||||
@@ -48,9 +48,9 @@ max_historical_shards=12
|
||||
/mnt/disk2
|
||||
```
|
||||
|
||||
The `type` field of `[shard_db]` can be omitted. If present, it _MUST_ be `NuDB`. [New in: rippled 1.3.1][]
|
||||
The `type` field of `[shard_db]` can be omitted. If present, it _MUST_ be `NuDB`. {% badge href="https://github.com/XRPLF/rippled/releases/tag/1.3.1" %}New in: rippled 1.3.1{% /badge %}
|
||||
|
||||
**Caution:** If `rippled` detects the wrong type of data in the shard store path, it may [fail to start](server-wont-start.html). You should use a new folder for the shard store. If you previously used a RocksDB shard store (`rippled` 1.2.x and lower), use a different path or delete the RocksDB shard data.
|
||||
**Caution:** If `rippled` detects the wrong type of data in the shard store path, it may [fail to start](../../troubleshooting/server-wont-start.md). You should use a new folder for the shard store. If you previously used a RocksDB shard store (`rippled` 1.2.x and lower), use a different path or delete the RocksDB shard data.
|
||||
|
||||
For more information, reference the `[shard_db]` example in the [rippled.cfg configuration example](https://github.com/XRPLF/rippled/blob/master/cfg/rippled-example.cfg).
|
||||
|
||||
@@ -68,24 +68,21 @@ This folder should contain a numbered folder for each shard your server has. At
|
||||
|
||||
You can instruct your server to download and import a shard from an archive file using the [download_shard method][].
|
||||
|
||||
To list the shards your server and its peers have available, you can use the [crawl_shards method][] or the [Peer Crawler](peer-crawler.html).
|
||||
To list the shards your server and its peers have available, you can use the [crawl_shards method][] or the [Peer Crawler](../../../references/http-websocket-apis/peer-port-methods/peer-crawler.md).
|
||||
|
||||
|
||||
## See Also
|
||||
|
||||
- **Concepts:**
|
||||
- [Ledger History](ledger-history.html)
|
||||
- [Online Deletion](online-deletion.html)
|
||||
- [Ledger History](../../../concepts/networks-and-servers/ledger-history.md)
|
||||
- [Online Deletion](online-deletion.md)
|
||||
- **Tutorials:**
|
||||
- [Configure Online Deletion](configure-online-deletion.html)
|
||||
- [Configure the Peer Crawler](configure-the-peer-crawler.html)
|
||||
- [Capacity Planning](capacity-planning.html)
|
||||
- [Configure Online Deletion](configure-online-deletion.md)
|
||||
- [Configure the Peer Crawler](../peering/configure-the-peer-crawler.md)
|
||||
- [Capacity Planning](../../installation/capacity-planning.md)
|
||||
- **References:**
|
||||
- [download_shard method][]
|
||||
- [crawl_shards method][]
|
||||
- [Ledger Data Formats](ledger-data-formats.html)
|
||||
- [Ledger Data Formats](../../../references/protocol/ledger-data/index.md)
|
||||
|
||||
<!--{# common link defs #}-->
|
||||
{% include '_snippets/rippled-api-links.md' %}
|
||||
{% include '_snippets/tx-type-links.md' %}
|
||||
{% include '_snippets/rippled_versions.md' %}
|
||||
{% raw-partial file="/_snippets/common-links.md" /%}
|
||||
|
||||
@@ -8,7 +8,7 @@ labels:
|
||||
---
|
||||
# Configure Online Deletion
|
||||
|
||||
In its default configuration, [the `rippled` server](xrpl-servers.html) [deletes history](online-deletion.html) older than the most recent 2000 [ledger versions](ledgers.html), keeping approximately 15 minutes of [ledger history](ledger-history.html) (based on the current rate between ledgers). This page describes how to configure the amount of history your `rippled` server stores before deleting.
|
||||
In its default configuration, [the `rippled` server](../../../concepts/networks-and-servers/index.md) [deletes history](online-deletion.md) older than the most recent 2000 [ledger versions](../../../concepts/ledgers/index.md), keeping approximately 15 minutes of [ledger history](../../../concepts/networks-and-servers/ledger-history.md) (based on the current rate between ledgers). This page describes how to configure the amount of history your `rippled` server stores before deleting.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
@@ -16,11 +16,11 @@ This tutorial assumes your server meets the following prerequisites:
|
||||
|
||||
- You are on a supported operating system: Ubuntu Linux, Red Hat Enterprise Linux (RHEL), or CentOS.
|
||||
|
||||
- The `rippled` server is already [installed](install-rippled.html) and [online deletion](online-deletion.html) is enabled.
|
||||
- The `rippled` server is already [installed](../../installation/index.md) and [online deletion](online-deletion.md) is enabled.
|
||||
|
||||
If you followed the installation instructions for a recommended platform, online deletion is enabled by default.
|
||||
|
||||
- Your server has [enough disk space](capacity-planning.html#disk-space) to store your chosen amount of history in its ledger store.
|
||||
- Your server has [enough disk space](../../installation/capacity-planning.md#disk-space) to store your chosen amount of history in its ledger store.
|
||||
|
||||
|
||||
## Configuration Steps
|
||||
@@ -29,24 +29,28 @@ To change the amount of history your server stores, perform the following steps:
|
||||
|
||||
1. Decide how many ledger versions' worth of history to store.
|
||||
|
||||
New ledger versions are usually validated 3 to 4 seconds apart, so the number of ledger versions corresponds roughly to the amount of time you want to store. See [Capacity Planning](capacity-planning.html) for details of how much storage is required for different configurations.
|
||||
New ledger versions are usually validated 3 to 4 seconds apart, so the number of ledger versions corresponds roughly to the amount of time you want to store. See [Capacity Planning](../../installation/capacity-planning.md) for details of how much storage is required for different configurations.
|
||||
|
||||
Online deletion is based on how many ledger versions to keep _after_ deleting history, so you should have enough disk space to store twice as many ledgers as you set it to keep.
|
||||
|
||||
0. In your `rippled`'s config file, edit the `online_delete` field of the `[node_db]` stanza.
|
||||
|
||||
[node_db]
|
||||
# Other settings unchanged ...
|
||||
online_delete=300000
|
||||
advisory_delete=0
|
||||
```
|
||||
[node_db]
|
||||
# Other settings unchanged ...
|
||||
online_delete=300000
|
||||
advisory_delete=0
|
||||
```
|
||||
|
||||
Set `online_delete` to the minimum number of ledger versions to keep after running online deletion. With automatic deletion (the default), the server typically runs deletion when it has accumulated about twice this many ledger versions.
|
||||
|
||||
{% include '_snippets/conf-file-location.md' %}<!--_ -->
|
||||
{% partial file="/_snippets/conf-file-location.md" /%}
|
||||
|
||||
0. Start (or restart) the `rippled` service.
|
||||
|
||||
$ sudo systemctl restart rippled
|
||||
```
|
||||
$ sudo systemctl restart rippled
|
||||
```
|
||||
|
||||
0. Wait for your server to sync to the network.
|
||||
|
||||
@@ -58,33 +62,28 @@ To change the amount of history your server stores, perform the following steps:
|
||||
|
||||
After online deletion runs, the `complete_ledgers` range reflects that older ledgers are no longer available. As your server accumulates history, the total number of ledgers available should slowly increase to twice the `online_delete` value you configured, then decrease when online deletion runs.
|
||||
|
||||
0. Monitor your `rippled` logs for messages that begin with `SHAMapStore::WRN`. This can indicate that [online deletion is being interrupted](online-deletion.html#interrupting-online-deletion) because your server fell out of sync with the network.
|
||||
0. Monitor your `rippled` logs for messages that begin with `SHAMapStore::WRN`. This can indicate that [online deletion is being interrupted](online-deletion.md#interrupting-online-deletion) because your server fell out of sync with the network.
|
||||
|
||||
If this happens regularly, your server may not have sufficient specifications to keep up with the ledger while running online deletion. Check that other services on the same hardware (such as scheduled backups or security scans) aren't competing with the `rippled` server for resources. You may want to try any of the following:
|
||||
|
||||
- Increase your system specs. See [System Requirements](system-requirements.html) for recommendations.
|
||||
- Increase your system specs. See [System Requirements](../../installation/system-requirements.md) for recommendations.
|
||||
- Change your configuration to store less history. (Step 2 of this tutorial)
|
||||
- Change your server's [`node_size` parameter](capacity-planning.html).
|
||||
- Use [NuDB instead of RocksDB](capacity-planning.html) for the ledger store.
|
||||
- [Schedule online deletion using Advisory Deletion](configure-advisory-deletion.html).
|
||||
- Change your server's [`node_size` parameter](../../installation/capacity-planning.md).
|
||||
- Use [NuDB instead of RocksDB](../../installation/capacity-planning.md) for the ledger store.
|
||||
- [Schedule online deletion using Advisory Deletion](configure-advisory-deletion.md).
|
||||
|
||||
|
||||
## See Also
|
||||
|
||||
- **Concepts:**
|
||||
- [Ledger History](ledger-history.html)
|
||||
- [Online Deletion](online-deletion.html)
|
||||
- [Ledger History](../../../concepts/networks-and-servers/ledger-history.md)
|
||||
- [Online Deletion](online-deletion.md)
|
||||
- **Tutorials:**
|
||||
- [Configure Advisory Deletion](configure-advisory-deletion.html)
|
||||
- [Configure History Sharding](configure-history-sharding.html)
|
||||
- [Capacity Planning](capacity-planning.html)
|
||||
- [Configure Advisory Deletion](configure-advisory-deletion.md)
|
||||
- [Configure History Sharding](configure-history-sharding.md)
|
||||
- [Capacity Planning](../../installation/capacity-planning.md)
|
||||
- **References:**
|
||||
- [server_info method][]
|
||||
- [Ledger Data Formats](ledger-data-formats.html)
|
||||
- [Ledger Data Formats](../../../references/protocol/ledger-data/index.md)
|
||||
|
||||
|
||||
|
||||
<!--{# common link defs #}-->
|
||||
{% include '_snippets/rippled-api-links.md' %}
|
||||
{% include '_snippets/tx-type-links.md' %}
|
||||
{% include '_snippets/rippled_versions.md' %}
|
||||
{% raw-partial file="/_snippets/common-links.md" /%}
|
||||
|
||||
@@ -8,13 +8,13 @@ labels:
|
||||
---
|
||||
# History Sharding
|
||||
|
||||
[Introduced in: rippled 0.90.0][]
|
||||
{% badge href="https://github.com/XRPLF/rippled/releases/tag/0.90.0" %}Introduced in: rippled 0.90.0{% /badge %}
|
||||
|
||||
As XRP Ledger servers run, they naturally produce a database containing data about the ledgers they built or acquired during network runtime. Each server stores that ledger data in its _ledger store_, but [online deletion](online-deletion.html) removes old ledgers' data automatically over time. History sharding provides a separate storage system for older ledger history so that the network can divide up the work of recording the entire (multiple terabyte) history of the XRP Ledger.
|
||||
As XRP Ledger servers run, they naturally produce a database containing data about the ledgers they built or acquired during network runtime. Each server stores that ledger data in its _ledger store_, but [online deletion](online-deletion.md) removes old ledgers' data automatically over time. History sharding provides a separate storage system for older ledger history so that the network can divide up the work of recording the entire (multiple terabyte) history of the XRP Ledger.
|
||||
|
||||
Historical sharding distributes the transaction history of the XRP Ledger into segments, called shards, across servers in the XRP Ledger network. A shard is a range of ledgers. A server uses mostly the same format for ledgers in both the ledger store and the shard store, but the two stores are separate.
|
||||
|
||||
[](img/xrp-ledger-network-ledger-store-and-shard-store.png)
|
||||
[](/img/xrp-ledger-network-ledger-store-and-shard-store.png)
|
||||
|
||||
<!-- Diagram source: https://docs.google.com/presentation/d/1mg2jZQwgfLCIhOU8Mr5aOiYpIgbIgk3ymBoDb2hh7_s/edit#slide=id.g417450e8da_0_316 -->
|
||||
|
||||
@@ -30,25 +30,21 @@ The server selects and downloads additional shards until it reaches the maximum
|
||||
|
||||
The history of all ledgers is shared by servers agreeing to keep particular ranges of historical ledgers. This makes it possible for servers to confirm that they have all the data they agreed to maintain, and produce "proof trees" or "ledger deltas" which shows how each ledger in the blockchain's history was the result of applying transactions to the previous state. Since servers that are configured with history sharding randomly select the shards that they store, the entire history of all closed ledgers is stored in a normal distribution curve, increasing the probability that the XRP Ledger Network evenly maintains the history.
|
||||
|
||||
History shards are recorded in a deterministic format, so that any two servers assembling the same shard produce the exact same binary data no matter what order they acquired the data and where they got it from. This makes it possible to compare checksums or cryptographic hashes of the shard data to verify the integrity of the data, and it is possible to share and import history shards through other formats. (For example, you could download shard data using Bittorrent or acquire physical media with the shard data pre-loaded on it, and verify that it matches the data that can be downloaded from the network.) [New in: rippled 1.8.1][]
|
||||
History shards are recorded in a deterministic format, so that any two servers assembling the same shard produce the exact same binary data no matter what order they acquired the data and where they got it from. This makes it possible to compare checksums or cryptographic hashes of the shard data to verify the integrity of the data, and it is possible to share and import history shards through other formats. (For example, you could download shard data using Bittorrent or acquire physical media with the shard data pre-loaded on it, and verify that it matches the data that can be downloaded from the network.) {% badge href="https://github.com/XRPLF/rippled/releases/tag/1.8.1" %}New in: rippled 1.8.1{% /badge %}
|
||||
|
||||
|
||||
## See Also
|
||||
|
||||
- **Concepts:**
|
||||
- [Ledgers](ledgers.html)
|
||||
- [Consensus](consensus.html)
|
||||
- [Ledgers](../../../concepts/ledgers/index.md)
|
||||
- [Consensus](../../../concepts/consensus-protocol/index.md)
|
||||
- **Tutorials:**
|
||||
- [Capacity Planning](capacity-planning.html)
|
||||
- [Configure `rippled`](configure-rippled.html)
|
||||
- [Configure History Sharding](configure-history-sharding.html)
|
||||
- [Capacity Planning](../../installation/capacity-planning.md)
|
||||
- [Configure `rippled`](../index.md)
|
||||
- [Configure History Sharding](configure-history-sharding.md)
|
||||
- **References:**
|
||||
- [crawl_shards method][]
|
||||
- [download_shard method][]
|
||||
- [Peer Crawler](peer-crawler.html)
|
||||
- [Peer Crawler](../../../references/http-websocket-apis/peer-port-methods/peer-crawler.md)
|
||||
|
||||
|
||||
<!--{# common link defs #}-->
|
||||
{% include '_snippets/rippled-api-links.md' %}
|
||||
{% include '_snippets/tx-type-links.md' %}
|
||||
{% include '_snippets/rippled_versions.md' %}
|
||||
{% raw-partial file="/_snippets/common-links.md" /%}
|
||||
|
||||
@@ -1,9 +1,13 @@
|
||||
---
|
||||
html: data-retention.html
|
||||
parent: configure-rippled.html
|
||||
template: pagetype-category.html.jinja
|
||||
metadata:
|
||||
indexPage: true
|
||||
blurb: Control how much data your server should keep and when it should remove old data.
|
||||
---
|
||||
# Data Retention
|
||||
|
||||
Control how much data your server should keep and when it should remove old data, including old versions of the ledger state and past transactions.
|
||||
Control how much data your server should keep and when it should remove old data, including old versions of the ledger state and past transactions.
|
||||
|
||||
|
||||
{% child-pages /%}
|
||||
|
||||
@@ -9,18 +9,18 @@ labels:
|
||||
# Online Deletion
|
||||
[[Source]](https://github.com/XRPLF/rippled/blob/master/src/ripple/app/misc/SHAMapStoreImp.cpp "Source")
|
||||
|
||||
The online deletion feature lets the `rippled` server delete the server's local copy of old ledger versions to keep disk usage from rapidly growing over time. The default config file sets online deletion to run automatically, but online deletion can also be configured to run only when prompted. [New in: rippled 0.27.0][]
|
||||
The online deletion feature lets the `rippled` server delete the server's local copy of old ledger versions to keep disk usage from rapidly growing over time. The default config file sets online deletion to run automatically, but online deletion can also be configured to run only when prompted. {% badge href="https://github.com/XRPLF/rippled/releases/tag/0.27.0" %}New in: rippled 0.27.0{% /badge %}
|
||||
|
||||
The server always keeps the complete _current_ state of the ledger, with all the balances and settings it contains. The deleted data includes older transactions and versions of the ledger state that are older than the stored history.
|
||||
|
||||
The default config file sets the `rippled` server to keep the most recent 2000 ledger versions and automatically delete older data.
|
||||
|
||||
**Tip:** Even with online deletion, the amount of disk space required to store the same time span's worth of ledger data increases over time, because the size of individual ledger versions tends to grow over time. This growth is very slow in comparison to the accumulation of data that occurs without deleting old ledgers. For more information on disk space needs, see [Capacity Planning](capacity-planning.html).
|
||||
**Tip:** Even with online deletion, the amount of disk space required to store the same time span's worth of ledger data increases over time, because the size of individual ledger versions tends to grow over time. This growth is very slow in comparison to the accumulation of data that occurs without deleting old ledgers. For more information on disk space needs, see [Capacity Planning](../../installation/capacity-planning.md).
|
||||
|
||||
|
||||
## Background
|
||||
|
||||
The `rippled` server stores [ledger history](ledger-history.html) in its _ledger store_. This data accumulates over time.
|
||||
The `rippled` server stores [ledger history](../../../concepts/networks-and-servers/ledger-history.md) in its _ledger store_. This data accumulates over time.
|
||||
|
||||
Inside the ledger store, ledger data is "de-duplicated". In other words, data that doesn't change from version to version is only stored once. The records themselves in the ledger store do not indicate which ledger version(s) contain them; part of the work of online deletion is identifying which records are only used by outdated ledger versions. This process is time consuming and affects the disk I/O and application cache, so the server cannot delete old data every time it closes a new ledger.
|
||||
|
||||
@@ -29,7 +29,7 @@ Inside the ledger store, ledger data is "de-duplicated". In other words, data th
|
||||
|
||||
The online deletion settings configure how many ledger versions the `rippled` server should keep available in the ledger store at a time. However, the specified number is a guideline, not a hard rule:
|
||||
|
||||
- The server never deletes data more recent than the configured number of ledger versions, but it may have less than that amount available if it has not been running for long enough or if it lost sync with the network at any time. (The server attempts to backfill at least some history; see [fetching history](ledger-history.html#fetching-history) for details.)
|
||||
- The server never deletes data more recent than the configured number of ledger versions, but it may have less than that amount available if it has not been running for long enough or if it lost sync with the network at any time. (The server attempts to backfill at least some history; see [fetching history](../../../concepts/networks-and-servers/ledger-history.md#fetching-history) for details.)
|
||||
- The server may store up to slightly over twice the configured number of ledger versions if online deletion is set to run automatically. (Each time it runs, it reduces the number of stored ledger versions to approximately the configured number.)
|
||||
|
||||
If online deletion is delayed because the server is busy, ledger versions can continue to accumulate. When functioning normally, online deletion begins when the server has twice the configured number of ledger versions, but it may not complete until after several more ledger versions have accumulated.
|
||||
@@ -55,7 +55,7 @@ The server only counts validated ledger versions when deciding how far back it c
|
||||
|
||||
### Interrupting Online Deletion
|
||||
|
||||
Online deletion automatically stops if the [server state](rippled-server-states.html) becomes less than `full`. If this happens, the server writes a log message with the prefix `SHAMapStore::WRN`. The server attempts to start online deletion again after the next validated ledger version after becoming fully synced.
|
||||
Online deletion automatically stops if the [server state](../../../references/http-websocket-apis/api-conventions/rippled-server-states.md) becomes less than `full`. If this happens, the server writes a log message with the prefix `SHAMapStore::WRN`. The server attempts to start online deletion again after the next validated ledger version after becoming fully synced.
|
||||
|
||||
If you stop the server or it crashes while online deletion is running, online deletion resumes after the server is restarted and the server becomes fully synced.
|
||||
|
||||
@@ -68,7 +68,7 @@ The following settings relate to online deletion:
|
||||
|
||||
- **`online_delete`** - Specify how many validated ledger versions to keep. The server periodically deletes any ledger versions that are older than this number. If not specified, no ledgers are deleted.
|
||||
|
||||
The default config file specifies 2000 for this value. This cannot be less than 256, because some events like [Fee Voting](fee-voting.html) and the [Amendment Process](amendments.html#amendment-process) update only every 256 ledgers.
|
||||
The default config file specifies 2000 for this value. This cannot be less than 256, because some events like [Fee Voting](../../../concepts/consensus-protocol/fee-voting.md) and the [Amendment Process](../../../concepts/networks-and-servers/amendments.md#amendment-process) update only every 256 ledgers.
|
||||
|
||||
**Caution:** If you run `rippled` with `online_delete` disabled, then later enable `online_delete` and restart the server, the server disregards but does not delete existing ledger history that your server already downloaded while `online_delete` was disabled. To save disk space, delete your existing history before re-starting the server after changing the `online_delete` setting.
|
||||
|
||||
@@ -78,7 +78,7 @@ The following settings relate to online deletion:
|
||||
|
||||
The following diagram shows the relationship between `online_delete` and `ledger_history` settings:
|
||||
|
||||
{{ include_svg("img/online_delete-vs-ledger_history.svg", "Ledgers older than `online_delete` are automatically deleted. Ledgers newer than `ledger_history` are backfilled. Ledgers in between are kept if available but not backfilled") }}
|
||||
[{% inline-svg file="/img/online_delete-vs-ledger_history.svg" /%}](/img/online_delete-vs-ledger_history.svg "Ledgers older than `online_delete` are automatically deleted. Ledgers newer than `ledger_history` are backfilled. Ledgers in between are kept if available but not backfilled")
|
||||
|
||||
- **`advisory_delete`** - If enabled, online deletion is not scheduled automatically. Instead, an administrator must manually trigger online deletion. Use the value `0` for disabled or `1` for enabled.
|
||||
|
||||
@@ -92,9 +92,9 @@ The following settings relate to online deletion:
|
||||
|
||||
The following diagram shows how `fetch_depth` works:
|
||||
|
||||
{{ include_svg("img/fetch_depth.svg", "Ledger versions older than fetch_depth are not served to peers") }}
|
||||
[{% inline-svg file="/img/fetch_depth.svg" /%}](/img/fetch_depth.svg "Ledger versions older than fetch_depth are not served to peers")
|
||||
|
||||
For estimates of how much disk space is required to store different amounts of history, see [Capacity Planning](capacity-planning.html#disk-space).
|
||||
For estimates of how much disk space is required to store different amounts of history, see [Capacity Planning](../../installation/capacity-planning.md#disk-space).
|
||||
|
||||
### Advisory Deletion
|
||||
|
||||
@@ -118,15 +118,15 @@ When it comes time for online deletion, the server first walks through the oldes
|
||||
## See Also
|
||||
|
||||
- **Concepts:**
|
||||
- [Ledgers](ledgers.html)
|
||||
- [Consensus](consensus.html)
|
||||
- [Ledgers](../../../concepts/ledgers/index.md)
|
||||
- [Consensus](../../../concepts/consensus-protocol/index.md)
|
||||
- **Tutorials:**
|
||||
- [Capacity Planning](capacity-planning.html)
|
||||
- [Configure `rippled`](configure-rippled.html)
|
||||
- [Configure Online Deletion](configure-online-deletion.html)
|
||||
- [Configure Advisory Deletion](configure-advisory-deletion.html)
|
||||
- [Configure History Sharding](configure-history-sharding.html)
|
||||
- [Configure Full History](configure-full-history.html)
|
||||
- [Capacity Planning](../../installation/capacity-planning.md)
|
||||
- [Configure `rippled`](../index.md)
|
||||
- [Configure Online Deletion](configure-online-deletion.md)
|
||||
- [Configure Advisory Deletion](configure-advisory-deletion.md)
|
||||
- [Configure History Sharding](configure-history-sharding.md)
|
||||
- [Configure Full History](configure-full-history.md)
|
||||
- **References:**
|
||||
- [ledger method][]
|
||||
- [server_info method][]
|
||||
@@ -134,8 +134,4 @@ When it comes time for online deletion, the server first walks through the oldes
|
||||
- [can_delete method][]
|
||||
- [ledger_cleaner method][]
|
||||
|
||||
|
||||
<!--{# common link defs #}-->
|
||||
{% include '_snippets/rippled-api-links.md' %}
|
||||
{% include '_snippets/tx-type-links.md' %}
|
||||
{% include '_snippets/rippled_versions.md' %}
|
||||
{% raw-partial file="/_snippets/common-links.md" /%}
|
||||
|
||||
Reference in New Issue
Block a user