ARCHIVE/xrpl-dev-portal
1
0
Fork 0
mirror of https://github.com/XRPLF/xrpl-dev-portal.git synced 2025-11-29 16:15:48 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #448 from mDuo13/rippled_v110

Browse Source 
rippled v1.1.0 Changes [Waiting for official release]
This commit is contained in:
Rome Reginelli
2018-09-14 18:11:30 -07:00
committed by GitHub
parent 891def5003 4b30d04aa4
commit 4de64228d8
44 changed files with 615 additions and 96 deletions
Download Patch File Download Diff File Expand all files Collapse all files

86
content/tutorials/manage-the-rippled-server/build-run-rippled-ubuntu.md
View File

@@ -4,7 +4,7 @@
For an overview of `rippled`, see [Operating rippled Servers](install-rippled.html).
Use these instructions to build a `rippled` executable from source version 1.0.0 or higher on Ubuntu Linux 16.04 or higher. These instructions were tested on Ubuntu 16.04 LTS.
Use these instructions to build a `rippled` executable from source version 1.1.0 or higher on Ubuntu Linux 16.04 or higher. These instructions were tested on Ubuntu 16.04 LTS.
For information about building `rippled` for other platforms, see [Builds](https://github.com/ripple/rippled/tree/develop/Builds) in the `rippled` GitHub repository.
@@ -32,99 +32,85 @@ These instructions use Ubuntu's APT (Advanced Packaging Tool) to install the sof
sudo apt-get -y upgrade
3. Install Git.
3. Install dependencies.
sudo apt-get -y install git
sudo apt-get -y install git cmake pkg-config protobuf-compiler libprotobuf-dev libssl-dev wget
4. Install CMake.
4. Compile Boost.
sudo apt-get -y install cmake
Version 1.1.0 of `rippled` requires Boost version 1.67.0 exactly. Because Boost version 1.67.0 isn't available in the Ubuntu 16.04 software repositories, you must compile it yourself.
5. Install `pkg-config`.
If you have previously built Boost 1.67.0 for `rippled` and configured the `BOOST_ROOT` environment variable, you can skip these steps.
sudo apt-get -y install pkg-config
1. Download Boost 1.67.0.
6. Install Protocol Buffers.
wget https://dl.bintray.com/boostorg/release/1.67.0/source/boost_1_67_0.tar.gz
sudo apt-get -y install protobuf-compiler
sudo apt-get -y install libprotobuf-dev
2. Extract `boost_1_67_0.tar.gz`.
7. Install Secure Socket Layer (SSL) toolkit development files.
tar xvzf boost_1_67_0.tar.gz
sudo apt-get -y install libssl-dev
3. Change to the new `boost_1_67_0` directory.
8. Install `wget` to be able to download Boost in the next step.
cd boost_1_67_0
sudo apt-get -y install wget
9. Compile Boost.
The recommended Boost version is 1.64.0. Because Boost version 1.64.0 isn't available in the Ubuntu 16.04 repos, you must compile it yourself.
If you have previously built Boost 1.64.0 for `rippled` and configured the `BOOST_ROOT` environment variable, you can skip these steps.
1. Download Boost 1.64.0.
wget https://dl.bintray.com/boostorg/release/1.64.0/source/boost_1_64_0.tar.gz
2. Untar `boost_1_64_0.tar.gz`.
tar xvzf boost_1_64_0.tar.gz
3. Access the new `boost_1_64_0` directory:
cd boost_1_64_0
4. To prepare the Boost.Build system for use, run:
4. Prepare the Boost.Build system for use.
./bootstrap.sh
5. To invoke Boost.Build to build the separately-compiled Boost libraries, run the following command. Replace `<number of parallel jobs>` with the number of jobs to run in parallel. Choose this value based on the number of CPU cores you want to use for building. This may take about 10 minutes, depending on your hardware specs.
5. Build the separately-compiled Boost libraries. This may take about 10 minutes, depending on your hardware specs.
./b2 -j <number of parallel jobs>
./b2 -j 4
6. Set the environment variable `BOOST_ROOT` to point to the new `boost_1_64_0` directory. It's best to put this environment variable in your `.profile`, or equivalent, file for your shell so it's automatically set when you log in. Add the following line to the file:
**Tip:** This example uses 4 processes to build in parallel. The best number of processes to use depends on how many CPU cores your hardware has available. You can use `cat /proc/cpuinfo` to get information about your hardware's processor.
export BOOST_ROOT=/home/ubuntu/boost_1_64_0
6. Set the environment variable `BOOST_ROOT` to point to the new `boost_1_67_0` directory. It's best to put this environment variable in your `.profile`, or equivalent, file for your shell so it's automatically set when you log in. Add the following line to the file:
export BOOST_ROOT=/home/ubuntu/boost_1_67_0
7. Source your updated `.profile` file. For example:
source ~/.profile
10. From a working directory, get the `rippled` source code. The `master` branch has the latest released version.
5. From a working directory, get the `rippled` source code. The `master` branch has the latest released version.
cd ~
git clone https://github.com/ripple/rippled.git
cd rippled
git checkout master
11. Check the commit log to be sure you're compiling the version you intend to. The most recent commit should be signed by a well-known Ripple developer and should set the version number to the latest released version. For example:
6. Check the commit log to be sure you're compiling the version you intend to. The most recent commit should be signed by a well-known Ripple developer and should set the version number to the latest released version. For example:
$ git log
$ git log -1
commit f31ca2860fb5f045b618aa05d1e76c7e2e9494ec (HEAD, tag: 1.0.0, origin/release, origin/master, master)
Author: Nikolaos D. Bougalis <nikb@bougalis.net>
Date: Fri May 11 10:29:41 2018 -0700
commit 3e22a1e9e8f2de450eded6ca4c2db6411e329b2a
Author: Nik Bougalis <nikb@bougalis.net>
Date: Wed Sep 5 18:34:43 2018 -0700
Set version to 1.0.0
Set version to 1.1.0
12. If you previously built, or (more importantly) tried and failed to build `rippled`, you should delete the `my_build/` directory (or whatever you named it) to start clean before moving on to the next step. Otherwise, you may get unexpected behavior, like a `rippled` executable that crashes due to a segmentation fault (segfault).
7. If you previously built, or (more importantly) tried and failed to build `rippled`, you should delete the `my_build/` directory (or whatever you named it) to start clean before moving on to the next step. Otherwise, you may get unexpected behavior, like a `rippled` executable that crashes due to a segmentation fault (segfault).
If this is your first time building `rippled` 1.0.0 or higher, you won't have a `my_build/` directory and can move on to the next step.
13. Use CMake to build a `rippled` binary executable from source code. The result will be a `rippled` binary executable in the `my_build` directory.
8. Use CMake to build a `rippled` binary executable from source code. The result will be a `rippled` binary executable in the `my_build` directory.
1. Generate the build system. Builds should be performed in a directory that is separate from the source tree root. In this example, we'll use a `my_build` directory that is a subdirectory of `rippled`.
mkdir my_build
cd my_build
cmake -Dtarget=gcc.debug.unity ..
cmake ..
**Tip:** The default build includes debugging symbols, which can be useful for development but are inefficient in production. To build `rippled` for use on production servers, add the `-DCMAKE_BUILD_TYPE=Release` flag when running the `cmake` command.
2. Build the `rippled` binary executable. Replace `<number of parallel jobs>` with the number of jobs to run in parallel. Choose this value based on the number of CPU cores you want to use for building.
cmake --build . -- -j <number of parallel jobs>
cmake --build . -- -j 4
14. _(Optional)_ Run `rippled` unit tests. If there are no test failures, you can be fairly certain that your `rippled` executable compiled correctly.
**Tip:** This example uses 4 processes to build in parallel. The best number of processes to use depends on how many CPU cores your hardware has available. You can use `cat /proc/cpuinfo` to get information about your hardware's processor.
9. _(Optional)_ Run `rippled` unit tests. If there are no test failures, you can be fairly certain that your `rippled` executable compiled correctly.
./rippled -u

35
content/tutorials/manage-the-rippled-server/enable-public-signing.md Normal file
View File

@@ -0,0 +1,35 @@
# Enable Public Signing
[New in: rippled 1.1.0][] By default, the signing methods for `rippled` are limited to administrative connections. If you want to allow signing methods to be used as public API methods (like with versions of `rippled` before v1.1.0), you can enable it with a configuration change.
This enables the following methods to be used on "public" [JSON-RPC and WebSocket connections](get-started-with-the-rippled-api.html), if your server accepts them:
- [sign][sign method]
- [sign_for][sign_for method]
- [submit][submit method] (in "sign-and-submit" mode)
You **do not** need to enable public signing to use these methods from an admin connection.
**Caution:** Ripple does not recommend enabling public signing. Like the [wallet_propose method][], the signing commands do not perform any actions that would require administrative-level permissions, but restricting them to admin connections protects users from irresponsibly sending or receiving secret keys over unsecured communications, or to servers they do not control.
To enable public signing, perform the following steps:
1. Edit your `rippled`'s config file.
vim /etc/opt/ripple/rippled.cfg
The [recommended installation](install-rippled.html) uses the config file `/etc/opt/ripple/rippled.cfg` by default. Other places you can put a config file include `$HOME/.config/ripple/rippled.cfg` (where `$HOME` is the home directory of the user running `rippled`) or
2. Add the following stanza to your config file, and save the changes:
[signing_support]
true
3. Restart your `rippled` server:
systemctl restart rippled
<!--{# common link defs #}-->
{% include '_snippets/rippled-api-links.md' %}
{% include '_snippets/tx-type-links.md' %}
{% include '_snippets/rippled_versions.md' %}

173
content/tutorials/manage-the-rippled-server/fix-sqlite-tx-db-page-size-issue.md Normal file
View File

@@ -0,0 +1,173 @@
# Fix SQLite Transaction Database Page Size Issue
`rippled` servers with full transaction history (or a very large amount of transaction history) and a database that was initially created with a `rippled` version earlier than 0.40.0 (released January 2017) may encounter a problem with their SQLite database page size that stops the server from operating properly. Servers that store only recent transaction history (the default configuration) and servers whose database files were created with `rippled` version 0.40.0 and later are not likely to encounter this problem.
This document describes steps to detect and correct this problem if it occurs.
## Background
`rippled` servers store a copy of their transaction history in a SQLite database. Before version 0.40.0, `rippled` configured this database to have a capacity of roughly 2TB. For most uses, this is plenty. However, full transaction history back to ledger 32570 (the oldest ledger version available in the production XRP Ledger history) is likely to exceed this exceed the SQLite database capacity. `rippled` servers version 0.40.0 and later create their SQLite database files with a larger capacity, so they are unlikely to encounter this problem.
The capacity of the SQLite database is a result of the database's _page size_ parameter, which cannot be easily changed after the database is created. (For more information on SQLite's internals, see [the official SQLite documentation](https://www.sqlite.org/fileformat.html).) The database can reach its capacity even if there is still free space on the disk and filesystem where it is stored. As described in the [Fix](#fix) below, reconfiguring the page size to avoid this problem requires a somewhat time-consuming migration process.
**Tip:** Full history is not necessary to operate a `rippled` server for most use cases. Servers with full transaction history may be useful for long-term analysis and archive purposes or as a precaution against disasters. For a less resource-intense way to contribute to the storage of transaction history, see [History Sharding](history-sharding.html).
## Detection
If your server is vulnerable to this problem, you can detect it two ways:
- You can detect the problem [proactively](#proactive-detection) (before it causes problems) if your `rippled` server is [version 1.1.0][New in: rippled 1.1.0] or later.
- You can identify the problem [reactively](#reactive-detection) (when your server is crashing) on any `rippled` version.
In both cases, detection of the problem requires access to `rippled`'s server logs.
**Tip:** The location of the debug log depends on your `rippled` server's configuration file. The [default configuration](https://github.com/ripple/rippled/blob/master/cfg/rippled-example.cfg#L1139-L1142) writes the server's debug log to the file `/var/log/rippled/debug.log`.
### Proactive Detection
To detect the SQLite page size problem proactively, you must be running **[rippled 1.1.0][New in: rippled 1.1.0] or later**. The `rippled` server writes a message such as the following in its debug log periodically, at least once every 2 minutes. (The exact numeric values from the log entry and the path to your transaction database depend on your environment.)
```text
Transaction DB pathname: /opt/rippled/transaction.db; SQLite page size: 1024
bytes; Free pages: 247483646; Free space: 253423253504 bytes; Note that this
does not take into account available disk space.
```
The value `SQLite page size: 1024 bytes` indicates that your transaction database is configured with a smaller page size and does not have capacity for full transaction history. If the value is already 4096 bytes or higher, then your SQLite database should already have adequate capacity to store full transaction history and you do not need to perform the migration described in this document.
The `rippled` server halts if the `Free space` described in this log message becomes less than 524288000 bytes (500MB). If your free space is approaching that threshold, [fix the problem](#fix) to avoid an unexpected outage.
### Reactive Detection
If your server's SQLite database capacity has already been exceeded, the `rippled` service writes a log message indicating the problem and halts.
#### rippled 1.1.0 and Later
On `rippled` versions 1.1.0 and later, the server shuts down gracefully with a message such as the following in the server's debug log:
```text
Free SQLite space for transaction db is less than 512MB. To fix this, rippled
must be executed with the vacuum <sqlitetmpdir> parameter before restarting.
Note that this activity can take multiple days, depending on database size.
```
#### Earlier than rippled 1.1.0
On `rippled` versions before 1.1.0, the server crashes repeatedly with messages such as the following in the server's debug log:
```text
Terminating thread doJob: AcquisitionDone: unhandled
N4soci18sqlite3_soci_errorE 'sqlite3_statement_backend::loadOne: database or
disk is full while executing "INSERT INTO [...]
```
## Fix
You can fix this issue using `rippled` on supported Linux systems according to the steps described in this document. In the case of a full-history server with system specs approximately matching the [recommended hardware configuration](capacity-planning.html#recommendation-1), the process may take more than two full days.
### Prerequisites
- You must be running **[rippled version 1.1.0][New in: rippled 1.1.0] or later**.
- [Upgrade rippled](update-rippled.html) to the latest stable version before starting this process.
- You can check what version of `rippled` you have installed locally by running the following command:
rippled --version
- You must have enough free space to temporarily store a second copy of the transaction database, in a directory that is writable by the `rippled` user. This free space does not need to be in the same filesystem as the existing transaction database.
The transaction database is stored in the `transaction.db` file in the folder specified by your configuration's `[database_path]` setting. You can check the size of this file to see how much free space you need. For example:
ls -l /var/lib/rippled/db/transaction.db
### Migration Process
To migrate your transaction database to a larger page size, perform the following steps:
1. Check that you meet all the [prerequisites](#prerequisites).
2. Create a folder to store temporary files during the migration process.
mkdir /tmp/rippled_txdb_migration
3. Grant the `rippled` user ownership of the temporary folder so it can write files there. (This is not necessary if your temporary folder is somewhere the `rippled` user already has write access to.)
chown rippled /tmp/rippled_txdb_migration
4. Confirm that your temporary folder has enough free space to store a copy of the transaction database.
For example, compare the `Avail` output from the `df` command to the [size of your `transaction.db` file](#prerequisites).
df -h /tmp/rippled_txdb_migration
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 5.4T 2.6T 2.6T 50% /tmp
5. If `rippled` is still running, stop it:
sudo systemctl stop rippled
6. Open a `screen` session (or other similar tool) so that the process does not stop when you log out:
screen
7. Become the `rippled` user:
sudo su - rippled
8. Run `rippled` executable directly, providing the `--vacuum` command with the path to the temporary directory:
/opt/ripple/bin/rippled -q --vacuum /tmp/rippled_txdb_migration
The `rippled` executable immediately displays the following message:
VACUUM beginning. page_size: 1024
9. Wait for the process to complete. This can take more than two full days.
When the process is complete, the `rippled` executable displays the following message, then exits:
VACUUM finished. page_size: 4096
While you wait, you can detach your `screen` session by pressing **CTRL-A**, then **D**. Later, you can reattach your screen session with a command such as the following:
screen -x -r
When the process is over, exit the screen session:
exit
For more information on the `screen` command, see [the official Screen User's Manual](https://www.gnu.org/software/screen/manual/screen.html) or any of the other many resources available online.
10. Restart the `rippled` service.
sudo systemctl start rippled
11. Confirm that the `rippled` service started successfully.
You can use the [commandline interface](get-started-with-the-rippled-api.html#commandline) to check the server status (unless you have configured your server not to accept JSON-RPC requests). For example:
/opt/ripple/bin/rippled server_info
For a description of the expected response from this command, see the [server_info method][] documentation.
12. Watch the server's debug log to confirm that the `SQLite page size` is now 4096:
tail -F /var/log/rippled/debug.log
The [periodic log message](#proactive-detection) should also show significantly more free pages and free pages than it did before the migration.
13. Optionally, you may now remove the temporary folder you created for the migration process.
rm -r /tmp/rippled_txdb_migration
If you mounted additional storage to hold the temporary copy of the transaction database, you can unmount and remove it now.
<!--{# common link defs #}-->
{% include '_snippets/rippled-api-links.md' %}
{% include '_snippets/tx-type-links.md' %}
{% include '_snippets/rippled_versions.md' %}