Merge branch 'v1.6.0' into master

This commit is contained in:
mDuo13
2020-08-19 09:06:56 -07:00
21 changed files with 482 additions and 51 deletions

View File

@@ -37,6 +37,7 @@
"1.3.1",
"1.4.0",
"1.5.0",
"1.6.0",
] %}
{% for v in rippled_versions %}

View File

@@ -92,6 +92,8 @@ The amendments that a `rippled` server is configured to vote for or against have
If your server is amendment blocked, you must [upgrade to a new version](install-rippled.html) to sync with the network.
It is also possible to be amendment blocked because you connected your server to a [parallel network](parallel-networks.html) that has different amendments enabled. For example, the XRP Ledger Devnet typically has upcoming and experimental amendments enabled. If you are using the latest production release, your server is likely to be amendment blocked when connecting to Devnet. You could resolve this issue by upgrading to an unstable pre-release or nightly build, or you could [connect to a different network such as Testnet](connect-your-rippled-to-the-xrp-test-net.html) instead.
#### How to Tell If Your `rippled` Server Is Amendment Blocked

View File

@@ -18,17 +18,21 @@ The [server_info method][] reports how many ledger versions your server has avai
## Fetching History
When it starts, a `rippled` server's first priority is to get a complete copy of the latest validated ledger. From there, it keeps up with advances in the ledger progress. If configured to do so, the server also backfills ledger history up to a configured amount, which must be equal to or less than the cutoff beyond which online deletion is configured to delete.
When an XRP Ledger server starts, its first priority is to get a complete copy of the latest validated ledger. From there, it keeps up with advances in the ledger progress. The server fills in any gaps in its ledger history that occur after syncing, and can backfill history from before it became synced. (Gaps in ledger history can occur if a server temporarily becomes too busy to keep up with the network, loses its network connection, or suffers other temporary issues.) When downloading ledger history, the server requests the missing data from its [peer servers](peer-protocol.html), and verifies the data's integrity using cryptographic [hashes][Hash].
The server can backfill history from before it became synced, as well as filling in any gaps in the history it has collected after syncing. (Gaps in ledger history can occur if a server temporarily becomes too busy to keep up with the network, loses its network connection, or suffers other temporary issues.) To backfill history, the server requests data from its peer `rippled` servers. The amount the server tries to backfill is defined by the `[ledger_history]` setting.
Backfilling history is one of the server's lowest priorities, so it may take a long time to fill missing history, especially if the server is busy or has less than sufficient hardware and network specs. For recommendations on hardware specs, see [Capacity Planning](capacity-planning.html). Backfilling history also requires that at least one of the server's direct peers has the history in question. For more information on managing your server's peer-to-peer connections, see [Configure Peering](configure-peering.html).
The XRP Ledger identifies data (on several different levels) by a unique hash of its contents. The XRP Ledger's state data contains a short summary of the ledger's history, in the form of the [LedgerHashes object type](ledgerhashes.html). Servers use the LedgerHashes objects to know which ledger versions to fetch, and to confirm that the ledger data they receive is correct and complete.
Backfilling history is one of the server's lowest priorities, so it may take a long time to fill missing history, especially if the server is busy or has less than sufficient hardware and network specs. For recommendations on hardware specs, see [Capacity Planning](capacity-planning.html). Backfilling history also requires that at least one of the server's direct peers has the history in question. <!--{# TODO: link some info for managing your peer connections when that exists #}-->
### With Advisory Deletion
<a id="with-advisory-deletion"></a><!-- old anchor to this area -->
### Backfilling
[Updated in: rippled 1.6.0][]
The amount of history a server attempts to download depends on its configuration. The server automatically tries to fill gaps by downloading history up to **the oldest ledger it already has available**. You can use the `[ledger_history]` setting to make the server backfill history beyond that point. However, the server never downloads ledgers that would be scheduled for [deletion](online-deletion.html).
The `[ledger_history]` setting defines a minimum number of ledgers to accumulate from before the current validated ledger. Use the special value `full` to download the [full history](#full-history) of the network. If you specify a number of ledgers, it must be equal to or more than the `online_deletion` setting; you cannot use `[ledger_history]` to make the server download _less_ history. To reduce the amount of history a server stores, change the [online deletion](online-deletion.html) settings instead.
If [online deletion](online-deletion.html) and advisory deletion are both enabled, the server automatically backfills data up to the oldest ledger it has not been allowed to delete yet. This can fetch data beyond the number of ledger versions configured in the `[ledger_history]` and `online_delete` settings. The [can_delete method][] tells the server what ledger versions it is allowed to delete.
## Full History

View File

@@ -28,18 +28,18 @@ To participate in the XRP Ledger, `rippled` servers connect to arbitrary peers u
Ideally, the server should be able to send _and_ receive connections on the peer port. You should [forward the port used for the peer protocol through your firewall](forward-ports-for-peering.html) to the `rippled` server.
The [default `rippled` config file](https://github.com/ripple/rippled/blob/master/cfg/rippled-example.cfg) listens for incoming peer protocol connections on port 51235 on all network interfaces. You can change the port used by editing the appropriate stanza in your `rippled.cfg` file.
IANA [has assigned port **2459**](https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml?search=2459) for the XRP Ledger peer protocol, but for compatibility with legacy systems, the [default `rippled` config file](https://github.com/ripple/rippled/blob/master/cfg/rippled-example.cfg) listens for incoming peer protocol connections on **port 51235** on all network interfaces. If you run a server, you can configure which port(s) your server listens on using the `rippled.cfg` file.
Example:
```
[port_peer]
port = 51235
port = 2459
ip = 0.0.0.0
protocol = peer
```
The peer protocol port also serves the [special Peer Crawler API method](peer-crawler.html).
The peer protocol port also serves [special peer port methods](peer-port-methods.html).
## Node Key Pair

View File

@@ -1,7 +1,7 @@
# download_shard
[[Source]](https://github.com/ripple/rippled/blob/master/src/ripple/rpc/handlers/DownloadShard.cpp "Source")
Instructs the server to download a specific [shard of historical ledger data](history-sharding.html) from an external source. Your `rippled` server must be [configured to store history shards](configure-history-sharding.html). [New in: rippled 1.1.0][]
Instructs the server to download a specific [shard of historical ledger data](history-sharding.html) from an external source. Your `rippled` server must be [configured to store history shards](configure-history-sharding.html). [Updated in: rippled 1.6.0][]
_The `download_shard` method is an [admin method](admin-rippled-methods.html) that cannot be run by unprivileged users._
@@ -45,15 +45,23 @@ An example of the request format:
}
```
*Commandline*
```sh
# Syntax: download_shard [[<index> <url>]]
rippled download_shard 1 https://example.com/1.tar.lz4 2 https://example.com/2.tar.lz4 5 https://example.com/5.tar.lz4
```
<!-- MULTICODE_BLOCK_END -->
The request includes the following fields:
The request includes the following field:
| `Field` | Type | Description |
|:-----------|:--------|:------------------------------------------------------|
| `shards` | Array | List of Shard Descriptor objects (see below) describing shards to download and where to download them from. |
| `validate` | Boolean | _(Optional)_ If `false`, skip validating the downloaded data. The default is `true`, which checks that the shard in the archive contains all the data objects for the shard and the shard is part of the ledger history of the current validated ledger. |
The `validate` field is deprecated and may be removed in a future version. (The server always checks the integrity of shards when it imports them.) [Updated in: rippled 1.6.0][]
Each **Shard Descriptor object** in the `shards` array has the following fields:
@@ -94,6 +102,19 @@ An example of a successful response:
}
```
*Commandline*
```json
Loading: "/etc/rippled.cfg"
Connecting to 127.0.0.1:5005
{
"result": {
"message": "downloading shards 1-2,5",
"status": "success"
}
}
```
<!-- MULTICODE_BLOCK_END -->

View File

@@ -49,7 +49,7 @@ rippled connect 192.170.145.88 51235
| `Field` | 型 | 説明 |
|:--------|:-------|:----------------------------------------------------------|
| `ip` | 文字列 | 接続するサーバーのIPアドレス。 |
| `port` | 数値 | _省略可_ 接続時に使用するポート番号。デフォルトでは6561です。 |
| `port` | 数値 | _省略可_ 接続時に使用するポート番号。デフォルトでは**2459**です。 [新規: rippled 1.6.0][] |
### 応答フォーマット

View File

@@ -49,7 +49,7 @@ The request includes the following parameters:
| `Field` | Type | Description |
|:--------|:-------|:----------------------------------------------------------|
| `ip` | String | IP address of the server to connect to |
| `port` | Number | _(Optional)_ Port number to use when connecting. Defaults to 6561. |
| `port` | Number | _(Optional)_ Port number to use when connecting. The default is **2459**. [Updated in: rippled 1.6.0][] |
### Response Format

View File

@@ -0,0 +1,109 @@
# Health Check
[[Source]](https://github.com/ripple/rippled/blob/de0c52738785de8bf837f9124da65c7905e7bb5a/src/ripple/overlay/impl/OverlayImpl.cpp#L1084-L1168 "Source")
The Health Check is a special [peer port method](peer-port-methods.html) for reporting on the health of an individual `rippled` server. This method is intended for use in automated monitoring to recognize outages and prompt automated or manual interventions such as restarting the server. [New in: rippled 1.6.0][]
This method checks several metrics to see if they are in ranges generally considered healthy. If all metrics are in normal ranges, this method reports that the server is healthy. If any metric is outside normal ranges, this method reports that the server is unhealthy and reports the metric(s) that are unhealthy. Since some metrics may rapidly fluctuate into and out of unhealthy ranges, you should not raise alerts unless the health check fails multiple times in a row.
**Note:** Since the health check is a [peer port method](peer-port-methods.html), it is not available when testing the server in [stand-alone mode](rippled-server-modes.html#reasons-to-run-a-rippled-server-in-stand-alone-mode).
## Request Format
To request the Health Check information, make the following HTTP request:
- **Protocol:** https
- **HTTP Method:** GET
- **Host:** (any `rippled` server, by hostname or IP address)
- **Port:** (the port number where the `rippled` server uses the Peer Protocol, typically 51235)
- **Path:** `/health`
- **Security:** Most `rippled` servers use a self-signed certificate to respond to the request. By default, most tools (including web browsers) flag or block such responses for being untrusted. You must ignore the certificate checking (for example, if using cURL, add the `--insecure` flag) to display a response from those servers.
<!-- TODO: link a tutorial for how to run rippled with a non-self-signed TLS cert -->
## Example Response
<!-- MULTICODE_BLOCK_START -->
*Healthy*
```json
HTTP/1.1 200 OK
Server: rippled-1.6.0-b8
Content-Type: application/json
Connection: close
Transfer-Encoding: chunked
{
"info": {}
}
```
*Warning*
```json
HTTP/1.1 503 Service Unavailable
Server: rippled-1.6.0
Content-Type: application/json
Connection: close
Transfer-Encoding: chunked
{
"info": {
"server_state": "connected",
"validated_ledger": -1
}
}
```
*Critical*
```json
HTTP/1.1 500 Internal Server Error
Server: rippled-1.6.0
Content-Type: application/json
Connection: close
Transfer-Encoding: chunked
{
"info": {
"peers": 0,
"server_state": "disconnected",
"validated_ledger":-1
}
}
```
<!-- MULTICODE_BLOCK_END -->
## Response Format
The response's HTTP status code indicates the health of the server:
| Status Code | Health Status | Description |
|:------------------------------|:--------------|:-----------------------------|
| **200 OK** | Healthy | All health metrics are within acceptable ranges. |
| **503 Service Unavailable** | Warning | One or more metric is in the warning range. Manual intervention may or may not be necessary. |
| **500 Internal Server Error** | Critical | One or more metric is in the critical range. There is a serious problem that probably needs manual intervention to fix. |
The response body is a JSON object with a single `info` object at the top level. The `info` object contains values for each metric that is in a warning or critical range. The response omits metrics that are in a healthy range, so a fully healthy server has an empty object.
The `info` object may contain the following fields:
| `Field` | Value | Description |
|:--------------------|:--------|:---------------------------------------------|
| `amendment_blocked` | Boolean | _(May be omitted)_ If `true`, the server is [amendment blocked](amendments.html#amendment-blocked) and must be upgraded to remain synced with the network; this state is critical. If the server is not amendment blocked, this field is omitted. |
| `load_factor` | Number | _(May be omitted)_ A measure of the overall load the server is under. This reflects I/O, CPU, and memory limitations. This is a warning if the load factor is over 100, or critical if the load factor is 1000 or higher. |
| `peers` | Number | _(May be omitted)_ The number of [peer servers](peer-protocol.html) this server is connected to. This is a warning if connected to 7 or fewer peers, and critical if connected to zero peers. |
| `server_state` | String | _(May be omitted)_ The current [server state](rippled-server-states.html). This is a warning if the server is in the `tracking`, `syncing`, or `connected` states. This is critical if the server is in the `disconnected` state. |
| `validated_ledger` | Number | _(May be omitted)_ The number of seconds since the last time a ledger was validated by [consensus](intro-to-consensus.html). If there is no validated ledger available ([as during the initial sync period when starting the server](server-doesnt-sync.html#normal-syncing-behavior)), this is the value `-1` and is considered a warning. This metric is also a warning if the last validated ledger was at least 7 seconds ago, or critical if the last validated ledger was at least 20 seconds ago. |
## See Also
For guidance interpreting the results of the health check, see [Health Check Interventions](health-check-interventions.html).
<!--{# common link defs #}-->
{% include '_snippets/rippled-api-links.md' %}
{% include '_snippets/tx-type-links.md' %}
{% include '_snippets/rippled_versions.md' %}

View File

@@ -1,6 +1,6 @@
# Peer Crawler
The Peer Crawler is a special API endpoint for reporting on the health and topology of the peer-to-peer network. This API method is available by default on a non-privileged basis through the [Peer Protocol](peer-protocol.html) port, which is also used for `rippled` servers' peer-to-peer communications about consensus, ledger history, and other necessary information.
The Peer Crawler is a special [peer port method](peer-port-methods.html) for reporting on the health and topology of the peer-to-peer network. This API method is available by default on a non-privileged basis through the [Peer Protocol](peer-protocol.html) port, which is also used for `rippled` servers' peer-to-peer communications about consensus, ledger history, and other necessary information.
The information reported by the peer crawler is effectively public, and can be used to report on the overall XRP Ledger network, its health, and topology.

View File

@@ -1,8 +1,8 @@
# Connect Your rippled to an XRP Ledger Altnet
# Connect Your rippled to a Parallel Network
Ripple hosts [alternative test and development networks](parallel-networks.html) for developers to test their apps on the latest non-production version of the XRP Ledger (Testnet) or to test and experiment with features on the latest beta version (Devnet). **The funds used on these networks are not real funds and are intended for testing only.** You can connect your [`rippled` server](the-rippled-server.html) to either the Testnet or Devnet.
**Note:** The XRP Testnet and Devnet ledger and balances are reset on a regular basis.
**Caution:** The Devnet frequently has new and experimental [amendments](amendments.html) enabled, so the latest production release version is likely to be amendment blocked when connecting to Devnet. You should use a pre-release or nightly build when connecting to Devnet.
## Steps

View File

@@ -33,6 +33,8 @@ If you run multiple [`rippled` servers](the-rippled-server.html) in the same dat
This defines specific peer servers to which this server should always attempt to maintain a direct peer-to-peer connection.
**Note:** If you omit the port number, the server uses port 2459, the IANA-assigned port for the [XRP Ledger protocol](peer-protocol.html). [New in: rippled 1.6.0][]
2. In the `[node_seed]` section, set the server's node seed to one of the `validation_seed` values you generated using the [validation_create method][] in step 2. Each server must use a unique node seed. For example:
[node_seed]

View File

@@ -40,9 +40,11 @@ To set up a specific server as a private peer, complete the following steps:
If your server connects using **proxies**, the IP addresses and ports should match the configurations of the `rippled` servers you are using as proxies. For each of those servers, the port number should match the `protocol = peer` port in that server's config file (usually 51235). For example, your configuration might look like this:
[ips_fixed]
192.168.0.1 51235
192.168.0.2 51235
[ips_fixed]
192.168.0.1 51235
192.168.0.2 51235
**Note:** If you omit the port number, the server uses port 2459, the IANA-assigned port for the [XRP Ledger protocol](peer-protocol.html). [New in: rippled 1.6.0][]
4. If using proxies, cluster them with your private peer and each other.

View File

@@ -0,0 +1,52 @@
---
html: enable-link-compression.html
funnel: Docs
doc_type: Tutorials
category: Manage the rippled Server
subcategory: Configure Peering
blurb: Save bandwidth by compressing peer-to-peer communications.
---
# Enable Link Compression
The `rippled` server can save bandwidth by compressing its [peer-to-peer communications](peer-protocol.html), at a cost of greater CPU usage. If you enable link compression, the server automatically compresses communications with peer servers that also have link compression enabled. [New in: rippled 1.6.0][]
## Steps
To enable link compression on your server, complete the following steps:
### 1. Edit your `rippled` server's config file.
```sh
$ vim /etc/opt/ripple/rippled.cfg
```
{% include '_snippets/conf-file-location.md' %}<!--_ -->
### 2. In the config file, add or uncomment the `[compression]` stanza.
To enable compression:
```text
[compression]
true
```
Use `false` to disable compression (the default).
### 3. Restart the `rippled` server
```sh
$ sudo systemctl restart rippled.service
```
After the restart, your server automatically uses link compression with other peers that also have link compression enabled.
## See Also
- [Capacity Planning](capacity-planning.html)
- [Peer Protocol](peer-protocol.html)
<!--{# common link defs #}-->
{% include '_snippets/rippled-api-links.md' %}
{% include '_snippets/tx-type-links.md' %}
{% include '_snippets/rippled_versions.md' %}

View File

@@ -8,7 +8,7 @@ Use these steps to manually connect your server to a specific [peer](peer-protoc
## Prerequisites
- You must know the IP address of the peer you want to connect to.
- You must know what port the peer you want to connect to uses for the XRP Ledger [peer protocol](peer-protocol.html). By default, this is port 51235.
- You must know what port the peer you want to connect to uses for the XRP Ledger [peer protocol](peer-protocol.html). The default config file uses port 51235.
- You must have a network connection from your server to the peer. For example, the peer server must [forward the appropriate port through its firewall](forward-ports-for-peering.html).
- The peer server must have available peer slots. If the peer is already at its maximum number of peers, you can ask the peer server's operator to add a [peer reservation](use-a-peer-reservation.html) for your server.

View File

@@ -53,17 +53,21 @@ For development purposes, run `rippled` as a non-admin user, not using `sudo`.
$ git clone git@github.com:ripple/rippled.git
$ cd rippled
0. By default, cloning puts you on the `develop` branch. Use this branch if you are doing development work and want the latest set of untested features.
0. Switch to the appropriate branch for the software version you want:
If you want the latest stable release, checkout the `master` branch.
For the latest stable release, use the `master` branch.
$ git checkout master
If you want to test out the latest release candidate, checkout the `release` branch:
For the latest release candidate, use the `release` branch:
$ git checkout release
Or, you can checkout one of the tagged releases listed on [GitHub](https://github.com/ripple/rippled/releases).
For the latest in-progress version, use the `develop` branch:
$ git checkout develop
Or, you can checkout one of the tagged releases listed on [GitHub](https://github.com/ripple/rippled/releases).
0. In the `rippled` directory you cloned, create your build directory and access it. For example:

View File

@@ -1,3 +1,11 @@
---
html: capacity-planning.html
funnel: Docs
doc_type: Tutorials
category: Manage the rippled Server
subcategory: Installation
blurb: Plan system specs and tune configuration for rippled in production environments.
---
# Capacity Planning
This section describes configuration, network, and hardware recommendations that you can use to tune and optimize the performance of your `rippled` server. Being aware of these considerations can help you ensure that your `rippled` server is ready to handle XRP Ledger network capacity today and in the near future.
@@ -177,16 +185,21 @@ Memory requirements are mainly a function of the `node_size` configuration setti
#### Network
Any enterprise or carrier-class data center should have substantial network bandwidth to support running `rippled` servers.
Any enterprise or carrier-class data center should have substantial network bandwidth to support running `rippled` servers. The actual bandwidth necessary varies significantly based on the current transaction volume in the network. Server behavior (such as backfilling [ledger history](ledger-history.html)) also affects network use.
Here are examples of observed network bandwidth use for common `rippled` tasks:
During exceptionally high periods of transaction volume, some operators have reported that their `rippled` servers have completely saturated a 100 MBit/s network link. Therefore, a gigabit network interface is required for reliable performance.
Here are examples of observed uncompressed network bandwidth use for common `rippled` tasks:
| Task | Transmit/Receive |
|:------------------------------------------------|:---------------------------|
| Process current transaction volumes | 2 Mbps transmit, 2 Mbps receive |
| Process average transaction volumes | 2 Mbps transmit, 2 Mbps receive |
| Process peak transaction volumes | >100 Mbps transmit |
| Serve historical ledger and transaction reports | 100 Mbps transmit |
| Start up `rippled` | 20 Mbps receive |
You can save bandwidth by [enabling compression on peer-to-peer communications](enable-link-compression.html), at a cost of higher CPU. Many hardware configurations have spare CPU capacity during normal use, so this can be an economical option if your network bandwidth is limited.
## See Also

View File

@@ -14,6 +14,16 @@ Before you install `rippled`, you must meet the [System Requirements](system-req
1. Install the Ripple RPM repository:
Choose the appropriate RPM repository for the stability of releases you want:
- `stable` for the latest production release (`master` branch)
- `unstable` for pre-release builds (`release` branch)
- `nightly` for experimental/development builds (`develop` branch)
<!-- MULTICODE_BLOCK_START -->
*Stable*
$ cat << REPOFILE | sudo tee /etc/yum.repos.d/ripple.repo
[ripple-stable]
name=XRP Ledger Packages
@@ -24,6 +34,33 @@ Before you install `rippled`, you must meet the [System Requirements](system-req
repo_gpgcheck=1
REPOFILE
*Pre-release*
$ cat << REPOFILE | sudo tee /etc/yum.repos.d/ripple.repo
[ripple-unstable]
name=XRP Ledger Packages
baseurl=https://repos.ripple.com/repos/rippled-rpm/unstable/
enabled=1
gpgcheck=0
gpgkey=https://repos.ripple.com/repos/rippled-rpm/unstable/repodata/repomd.xml.key
repo_gpgcheck=1
REPOFILE
*Development*
$ cat << REPOFILE | sudo tee /etc/yum.repos.d/ripple.repo
[ripple-nightly]
name=XRP Ledger Packages
baseurl=https://repos.ripple.com/repos/rippled-rpm/nightly/
enabled=1
gpgcheck=0
gpgkey=https://repos.ripple.com/repos/rippled-rpm/nightly/repodata/repomd.xml.key
repo_gpgcheck=1
REPOFILE
<!-- MULTICODE_BLOCK_START -->
2. Fetch the latest repo updates:
$ sudo yum -y update

View File

@@ -0,0 +1,110 @@
# Health Check Interventions
The [Health Check method](health-check.html) can be used by automated monitoring to recognize when a `rippled` server is not healthy and prompt interventions such as restarting the server or alerting a human administrator.
Infrastructure monitoring, and reliability engineering more generally, is an advanced discipline that involves using multiple sources of data to make decisions in context. This document provides some suggestions for how to use the health check most effectively, but these recommendations are only meant as guidelines as part of a larger strategy.
## Momentary Failures
Some [metrics][] in the health check can rapidly fluctuate into unhealthy ranges and then recover automatically shortly afterward. It is unnecessary and undesirable to raise alerts every single time the health check reports an unhealthy status. An automated monitoring system should call the health check method frequently, but only escalate to a higher level of intervention based on the severity and frequency of the problem.
For example, if you check the health of the server once per second, you might raise an alert if the server reports "warning" status three times in a row, or four times in a five-second span. You might also raise an alert if the server reports "critical" status twice in a five-second span.
**Tip:** The server normally reports a "critical" status for the first few seconds after startup, switches to a "warning" status after it establishes a connection to the network, and finally reports a "healthy" status when it has fully synced to the network. After a restart, you should give a server 515 minutes to sync before taking additional interventions.
## Special Cases
Certain server configurations may always report a `warning` status even when operating normally. If your server qualifies as a special case, you must configure your automated monitoring to recognize the difference between the normal status and an actual problem. This probably involves parsing the JSON response body for the health check method and comparing the values there with expected normal ranges.
Some examples of special cases that may occur include:
- A [private peer](peer-protocol.html#private-peers) typically has a very small number of peer-to-peer connections to known servers only, but the health check reports a warning on the `peers` metric if the server is connected to 7 or fewer peers. You should know the exact number of peers your server is configured to have and check for that value.
- On a [parallel or test network](parallel-networks.html) where new transactions are not being sent continuously, the network waits up to 20 seconds for new transactions before attempting to validate a new ledger version, but the health check reports a warning on the `validated_ledger` metric if the latest validated ledger is 7 or more seconds old. If you are running `rippled` on a non-production network, you may want to ignore `warning` messages for this metric unless you know that there should be transactions being regularly sent. You may still want to alert on the `critical` level of 20 seconds, because the XRP Ledger protocol is designed to validate new ledger versions at least once every 20 seconds even if there are no new transactions to process.
## Suggested Interventions
When a health check fails, and it's not just a [momentary failure](#momentary-failures), the action to take to recover from the outage varies based on the cause. You may be able to configure your infrastructure to fix some types of failures automatically. Other failures require the intervention of a human administrator who can investigate and take the necessary steps to resolve more complex or critical failures; depending on the structure of your organization, you may have different levels of human administrator so that less skilled, lower level administrators can fix certain issues independently, but need to escalate to higher level administrators to fix larger or more complex issues. How and when you respond is likely to depend on your unique situation, but the metrics reported in the health check result can be a factor in these decisions.
The following sections suggest some common interventions you may want to attempt and the health check statuses most likely to prompt those interventions. Automated systems and human administrators may selectively escalate through these and other interventions:
- [Redirect traffic](#redirect-traffic) away from the affected server
- [Restart](#restart) the server software or hardware
- [Upgrade](#upgrade) the `rippled` software
- [Investigate network](#investigate-network) in case the problem originates elsewhere
- [Replace hardware](#replace-hardware)
### Redirect Traffic
A common reliability technique is to run a pool of redundant servers through one or more load-balancing proxies. You can do this with `rippled` servers, but should not do this with [validators](rippled-server-modes.html). In some cases, the load balancers can monitor the health of servers in their pools and direct traffic only to the servers that are currently reporting themselves as healthy. This allows servers to recover from being temporarily overloaded and automatically rejoin the pool of active servers.
Redirecting traffic away from a server that is unhealthy is an appropriate response, especially for servers that report a `health` status of `warning`. Servers in the `critical` range may need more significant interventions.
### Restart
The most straightforward intervention is to restart the server. This can resolve temporary issues with several types of failures, including any of the following [metrics][]:
- `load_factor`
- `peers`
- `server_state`
- `validated_ledger`
To restart only the `rippled` service, use `systemctl`:
```
$ sudo systemctl restart rippled.service
```
A stronger intervention is to restart the entire machine.
**Caution:** After a server starts, it typically needs up to 15 minutes to sync to the network. During this time, the health check is likely to report a critical or warning status. You should be sure your automated systems give servers enough time to sync before restarting them again.
### Upgrade
If the server reports `"amendment_blocked": true` in the health check, this indicates that the XRP Ledger has enabled a [protocol amendment](amendments.html) that your server does not understand. As a precaution against misinterpreting the revised rules of the network in a way that causes you to lose money, such servers become "amendment blocked" instead of operating normally.
To resolve being amendment blocked, [update your server](install-rippled.html) to a newer software version that understands the amendment.
Also, software bugs can cause a server to get [stuck not syncing](server-doesnt-sync.html). In this case, the `server_state` metric is likely to be in a warning or critical state. If you are not using the latest stable release, you should upgrade to get the latest fixes for any known issues that could cause this.
### Investigate Network
An unreliable or insufficient network connection can cause a server to report outages. Warning or critical values in the following [metrics][] can indicate network problems:
- `peers`
- `server_state`
- `validated_ledger`
In this case, the necessary interventions may involve changes to other systems, such as:
- Adjusting firewall rules to allow necessary traffic to reach a server, or to block harmful traffic from outside
- Restarting or replacing network interfaces, switches, routers, or cabling
- Contacting other network service providers to resolve an issue on their end
### Replace Hardware
If the outage is caused by a hardware failure or by higher load than the hardware is capable of handling, you may need to replace some components or even the entire server.
The amount of load on a server in the XRP Ledger depends in part on transaction volume in the network, which varies organically. Load also depends on your usage pattern. See [Capacity Planning](capacity-planning.html) for how to plan the appropriate hardware and settings for your situation.
Warning or critical values for the following [metrics][] may indicate insufficient hardware:
- `load_factor`
- `server_state`
- `validated_ledger`
<!--{# common link defs #}-->
[metrics]: health-check.html#response-format
{% include '_snippets/rippled-api-links.md' %}
{% include '_snippets/tx-type-links.md' %}
{% include '_snippets/rippled_versions.md' %}

View File

@@ -4,6 +4,28 @@ The following sections describe some of the most common types of log messages th
This is an important step in [Diagnosing Problems](diagnosing-problems.html) with `rippled`.
## Log Message Structure
The following shows the format of the log file:
```text
2020-Jul-08 20:10:17.372178946 UTC Peer:WRN [236] onReadMessage from n9J2CP7hZypxDJ27ZSxoy4VjbaSgsCNaRRJtJkNJM5KMdGaLdRy7 at 197.27.127.136:53046: stream truncated
2020-Jul-08 20:11:13.582438263 UTC PeerFinder:ERR Logic testing 52.196.126.86:13308 with error, Connection timed out
2020-Jul-08 20:11:57.728448343 UTC Peer:WRN [242] onReadMessage from n9J2CP7hZypxDJ27ZSxoy4VjbaSgsCNaRRJtJkNJM5KMdGaLdRy7 at 197.27.127.136:53366: stream truncated
2020-Jul-08 20:12:12.075081020 UTC LoadMonitor:WRN Job: sweep run: 1172ms wait: 0ms
```
Each line represents one log entry, with the following parts in order, separated by spaces:
1. The date the log entry was written, such as `2020-Jul-08`.
2. The time the log entry was written, such as `20:12:12.075081020`.
3. The time zone indicator `UTC`. (Log dates are always in UTC.) [New in: rippled 1.5.0][]
4. The log partition and severity, such as `LoadMonitor:WRN`.
5. The log message, such as `Job: sweep run: 1172ms wait: 0ms`.
For simplicity, the examples in this page omit the date, time, and time zone indicator.
## Crashes
Messages in the log that mention runtime errors can indicate that the server crashed. These messages usually start with a message such as one of the following examples:
@@ -32,7 +54,7 @@ If none of the above apply, please report the issue to Ripple as a security-sens
Log messages such as the following indicate that a server received validations for different ledger indexes out of order.
```text
2018-Aug-28 22:55:58.316094260 Validations:WRN Val for 2137ACEFC0D137EFA1D84C2524A39032802E4B74F93C130A289CD87C9C565011 trusted/full from nHUeUNSn3zce2xQZWNghQvd9WRH6FWEnCBKYVJu2vAizMxnXegfJ signing key n9KcRZYHLU9rhGVwB9e4wEMYsxXvUfgFxtmX25pc1QPNgweqzQf5 already validated sequence at or past 12133663 src=1
Validations:WRN Val for 2137ACEFC0D137EFA1D84C2524A39032802E4B74F93C130A289CD87C9C565011 trusted/full from nHUeUNSn3zce2xQZWNghQvd9WRH6FWEnCBKYVJu2vAizMxnXegfJ signing key n9KcRZYHLU9rhGVwB9e4wEMYsxXvUfgFxtmX25pc1QPNgweqzQf5 already validated sequence at or past 12133663 src=1
```
Occasional messages of this type do not usually indicate a problem. If this type of message occurs often with the same sending validator, it could indicate a problem, including any of the following (roughly in order of most to least likely):
@@ -47,7 +69,7 @@ Occasional messages of this type do not usually indicate a problem. If this type
The following log message indicates that [StatsD export](configure-statsd.html) failed:
```text
2020-Apr-07 16:59:36.299867439 UTC Collector:ERR async_send failed: Connection refused
Collector:ERR async_send failed: Connection refused
```
This could mean:
@@ -60,12 +82,23 @@ Check the `[insight]` stanza in your `rippled`'s config file and confirm that yo
This error has no other impact on the `rippled` server, which should continue to work as normal except for the sending of StatsD metrics.
## Check for upgrade
The following message indicates that the server has detected that it is running an older software version than at least 60% of its trusted validators:
```text
LedgerMaster:ERR Check for upgrade: A majority of trusted validators are running a newer version.
```
This is not strictly a problem, but an old server version is likely to become [amendment blocked](amendments.html#amendment-blocked). You should [update `rippled`](install-rippled.html) to the latest stable version. (If you are connected to [devnet](parallel-networks.html), update to the latest nightly version instead.)
## Connection reset by peer
The following log message indicates that a peer `rippled` server closed a connection:
```text
2018-Aug-28 22:55:41.738765510 Peer:WRN [012] onReadMessage: Connection reset by peer
Peer:WRN [012] onReadMessage: Connection reset by peer
```
Losing connections from time to time is normal for any peer-to-peer network. **Occasional messages of this kind do not indicate a problem.**
@@ -81,7 +114,7 @@ A large number of these messages around the same time may indicate a problem, su
The following log message indicates that a client to the server's public API has been dropped as a result of [rate limiting](rate-limiting.html):
```text
2020-Feb-24 23:05:35.566312806 Resource:WRN Consumer entry 169.55.164.21 dropped with balance 15970 at or above drop threshold 15000
Resource:WRN Consumer entry 169.55.164.21 dropped with balance 15970 at or above drop threshold 15000
```
The entry contains the IP address of the client that exceeded its rate limit, and the client's "balance", which is a score estimating the rate at which the client has been using the API. The threshold for dropping a client is [hardcoded to a score of 15000](https://github.com/ripple/rippled/blob/06c371544acc3b488b9d9c057cee4e51f6bef7a2/src/ripple/resource/impl/Tuning.h#L34-L35).
@@ -128,10 +161,10 @@ Messages such as the following occur when a function takes a long time to run (o
The following similar message occurs when a job spends a long time waiting to run (again, over 11 seconds in this example):
```text
2018-Aug-28 22:56:36.180970431 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 11566ms
2018-Aug-28 22:56:36.181053831 LoadMonitor:WRN Job: AcquisitionDone run: 0ms wait: 11566ms
2018-Aug-28 22:56:36.181110594 LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 11566ms
2018-Aug-28 22:56:36.181169931 LoadMonitor:WRN Job: AcquisitionDone run: 0ms wait: 11566ms
LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 11566ms
LoadMonitor:WRN Job: AcquisitionDone run: 0ms wait: 11566ms
LoadMonitor:WRN Job: processLedgerData run: 0ms wait: 11566ms
LoadMonitor:WRN Job: AcquisitionDone run: 0ms wait: 11566ms
```
These two types of messages often occur together, when a long-running job causes other jobs to wait a long time for it to finish.
@@ -252,7 +285,7 @@ Aside from the bug, this error can also occur if `rippled` became unable to writ
Log messages such as the following occur when the server sees a validation message from a peer and it does not know the parent ledger version that server is building on. This can occur when the server is not in sync with the rest of the network:
```text
2018-Aug-28 22:56:22.256065549 Validations:WRN Unable to determine hash of ancestor seq=3 from ledger hash=00B1E512EF558F2FD9A0A6C263B3D922297F26A55AEB56A009341A22895B516E seq=12133675
Validations:WRN Unable to determine hash of ancestor seq=3 from ledger hash=00B1E512EF558F2FD9A0A6C263B3D922297F26A55AEB56A009341A22895B516E seq=12133675
```
{% include '_snippets/unsynced_warning_logs.md' %}
@@ -264,9 +297,9 @@ Log messages such as the following occur when the server sees a validation messa
Log messages such as the following occur when a server is not in sync with the rest of the network:
```text
2018-Aug-28 22:56:22.368460130 LedgerConsensus:WRN View of consensus changed during open status=open, mode=proposing
2018-Aug-28 22:56:22.368468202 LedgerConsensus:WRN 96A8DF9ECF5E9D087BAE9DDDE38C197D3C1C6FB842C7BB770F8929E56CC71661 to 00B1E512EF558F2FD9A0A6C263B3D922297F26A55AEB56A009341A22895B516E
2018-Aug-28 22:56:22.368499966 LedgerConsensus:WRN {"accepted":true,"account_hash":"89A821400087101F1BF2D2B912C6A9F2788CC715590E8FA5710F2D10BF5E3C03","close_flags":0,"close_time":588812130,"close_time_human":"2018-Aug-28 22:55:30.000000000","close_time_resolution":30,"closed":true,"hash":"96A8DF9ECF5E9D087BAE9DDDE38C197D3C1C6FB842C7BB770F8929E56CC71661","ledger_hash":"96A8DF9ECF5E9D087BAE9DDDE38C197D3C1C6FB842C7BB770F8929E56CC71661","ledger_index":"3","parent_close_time":588812070,"parent_hash":"5F5CB224644F080BC8E1CC10E126D62E9D7F9BE1C64AD0565881E99E3F64688A","seqNum":"3","totalCoins":"100000000000000000","total_coins":"100000000000000000","transaction_hash":"0000000000000000000000000000000000000000000000000000000000000000"}
LedgerConsensus:WRN View of consensus changed during open status=open, mode=proposing
LedgerConsensus:WRN 96A8DF9ECF5E9D087BAE9DDDE38C197D3C1C6FB842C7BB770F8929E56CC71661 to 00B1E512EF558F2FD9A0A6C263B3D922297F26A55AEB56A009341A22895B516E
LedgerConsensus:WRN {"accepted":true,"account_hash":"89A821400087101F1BF2D2B912C6A9F2788CC715590E8FA5710F2D10BF5E3C03","close_flags":0,"close_time":588812130,"close_time_human":"2018-Aug-28 22:55:30.000000000","close_time_resolution":30,"closed":true,"hash":"96A8DF9ECF5E9D087BAE9DDDE38C197D3C1C6FB842C7BB770F8929E56CC71661","ledger_hash":"96A8DF9ECF5E9D087BAE9DDDE38C197D3C1C6FB842C7BB770F8929E56CC71661","ledger_index":"3","parent_close_time":588812070,"parent_hash":"5F5CB224644F080BC8E1CC10E126D62E9D7F9BE1C64AD0565881E99E3F64688A","seqNum":"3","totalCoins":"100000000000000000","total_coins":"100000000000000000","transaction_hash":"0000000000000000000000000000000000000000000000000000000000000000"}
```
{% include '_snippets/unsynced_warning_logs.md' %}

View File

@@ -131,6 +131,10 @@ targets:
"transaction-metadata.html#affectednodes": "transaction-metadata.html"
# Fix link from untranslated peer-crawler.html:
"peer-protocol.html#private-peers": "peer-protocol.html#プライベートピア"
# Fix links from untranslated health-check.html:
"rippled-server-modes.html#reasons-to-run-a-rippled-server-in-stand-alone-mode": "rippled-server-modes.html#rippledサーバーをスタンドアロンモードで実行する理由"
"server-doesnt-sync.html#normal-syncing-behavior": "server-doesnt-sync.html#通常の同期動作"
- name: xrp-api-only
@@ -2246,12 +2250,6 @@ pages:
- ja
- md: tutorials/manage-the-rippled-server/installation/capacity-planning.md
html: capacity-planning.html
funnel: Docs
doc_type: Tutorials
category: Manage the rippled Server
subcategory: Installation
blurb: Plan system specs and tune configuration for rippled in production environments.
targets:
- en
@@ -2566,6 +2564,12 @@ pages:
targets:
- ja
# TODO: translate this page including the blurb in its frontmatter.
- md: tutorials/manage-the-rippled-server/configure-peering/enable-link-compression.md
targets:
- en
- ja
- md: tutorials/manage-the-rippled-server/configure-peering/forward-ports-for-peering.md
html: forward-ports-for-peering.html
funnel: Docs
@@ -2760,6 +2764,18 @@ pages:
targets:
- en
# TODO: translate page and blurb
- md: tutorials/manage-the-rippled-server/troubleshooting/health-check-interventions.md
html: health-check-interventions.html
funnel: Docs
doc_type: Tutorials
category: Manage the rippled Server
subcategory: Troubleshooting rippled
blurb: Use the rippled server's health check as part of automated infrastructure monitoring.
targets:
- en
- ja
- md: tutorials/manage-the-rippled-server/troubleshooting/diagnosing-problems.ja.md
html: diagnosing-problems.html
funnel: Docs
@@ -5870,35 +5886,60 @@ pages:
targets:
- ja
- md: references/rippled-api/peer-crawler.md
# TODO: translate title & blurb
- name: Peer Port Methods
html: peer-port-methods.html
funnel: Docs
doc_type: References
supercategory: rippled API
category: Peer Port Methods
template: template-landing-children.html
blurb: Special API method for sharing network topology and status metrics.
targets:
- en
- ja
# TODO: translate page & blurb
- md: references/rippled-api/peer-port-methods/health-check.md
html: health-check.html
funnel: Docs
doc_type: References
supercategory: rippled API
category: Peer Port Methods
blurb: Special API method for reporting server health.
targets:
- en
- ja
- md: references/rippled-api/peer-port-methods/peer-crawler.md
html: peer-crawler.html
funnel: Docs
doc_type: References
supercategory: rippled API
category: Peer Crawler
category: Peer Port Methods
blurb: Special API method for sharing network topology and status metrics.
targets:
- en
# TODO: translate page
- md: references/rippled-api/peer-crawler.md
- md: references/rippled-api/peer-port-methods/peer-crawler.md
html: peer-crawler.html
funnel: Docs
doc_type: References
supercategory: rippled API
category: Peer Crawler
category: Peer Port Methods
blurb: ネットワークトポロジーとステータスメトリックを共有するための特殊なAPIメソッドです。
untranslated_warning: true
targets:
- ja
# TODO: translate page & blurb
- md: references/rippled-api/validator-list.md
- md: references/rippled-api/peer-port-methods/validator-list.md
html: validator-list.html
funnel: Docs
doc_type: References
supercategory: rippled API
category: Validator List
category: Peer Port Methods
blurb: Special API method for sharing recommended validator lists.
targets:
- en