New file structure for concepts & tutorials

content/concepts/the-rippled-server/history-sharding.md

@@ -0,0 +1,55 @@
+# History Sharding
+
+[Introduced in: rippled 0.90.0][New in: rippled 0.90.0]
+
+As servers operate, they naturally produce a database containing data about the ledgers they witnessed or acquired during network runtime. Each `rippled` server stores that ledger data in its ledger store, but the online delete logic rotates these databases when the number of stored ledgers exceeds configured space limitations.
+
+Historical sharding distributes the transaction history of the XRP Ledger into segments, called shards, across servers in the XRP Ledger network. A shard is a range of ledgers. A `rippled` server stores ledgers in both the ledger store and the shard store in the same way.
+
+Using the history sharding feature, individual `rippled` servers can contribute to storing historical data without needing to store the entire (multiple terabyte) history. A shard store does not replace a ledger store, but implements a reliable path towards distributed ledger history across the XRP Ledger Network.
+
+[![XRP Ledger Network: Ledger Store and Shard Store Diagram](img/xrp-ledger-network-ledger-store-and-shard-store.png)](img/xrp-ledger-network-ledger-store-and-shard-store.png)
+
+## Acquiring and Sharing History Shards
+
+`rippled` servers acquire and store history shards only if configured to do so. For those servers, acquiring shards begins after synchronizing with the network and backfilling ledger history to the configured number of recent ledgers. During this time of lower network activity, a `rippled` server set to maintain a `shard_db` randomly chooses a shard to add to its shard store. To increase the probability for an even distribution of the network ledger history, shards are randomly selected for acquisition, and the current shard is given no special consideration.
+
+Once a shard is selected, the ledger acquire process begins by fetching the sequence of the last ledger in the shard and working backwards toward the first. The retrieval process begins with the server checking for the data locally. For data that is not available, the server requests data from its peer `rippled` servers. Those servers that have the data available for the requested period respond with their history. The requesting server combines those responses to create the shard. The shard is complete when it contains all the ledgers in a specific range.
+
+If a `rippled` server runs out of space before completely acquiring a shard, it stops its retrieval process until it has space available to continue. After that point, the most recently completed shard may replace an older shard. If there is sufficient disk space, the `rippled` server acquires additional randomly selected shards to add to the shard store until reaching the maximum allocated disk space for shards (`max_size_gb`).
+
+## XRP Ledger Network Data Integrity
+
+The history of all ledgers is shared by servers agreeing to keep particular ranges of historical ledgers. This makes it possible for servers to confirm that they have all the data they agreed to maintain, and produce proof trees or ledger deltas. Since `rippled` servers that are configured with history sharding randomly select the shards that they store, the entire history of all closed ledgers is stored in a normal distribution curve, increasing the probability that the XRP Ledger Network evenly maintains the history.
+
+## Shard Store Configuration
+
+To configure your `rippled` to store shards of ledger history, add a `shard_db` section to your `rippled.cfg` file.
+
+### Shard Configuration Example
+The following snippet from an example `rippled.cfg` file shows the configuration fields for adding sharding to a `rippled` server:
+
+```
+[shard_db]
+type=NuDB
+path=/var/lib/rippled/db/shards/nudb
+max_size_gb=50
+```
+
+**Tip:** Ripple recommends using NuDB for the shard store (`type=NuDB`). NuDB uses fewer file handles per shard than RocksDB. RocksDB uses memory that scales with the size of data it stores, which may require excessive memory overhead.
+
+**Tip:** While both validator and tracking (or stock) `rippled` servers can be configured to use history shard stores, Ripple recommends adding history sharding only for non-validator `rippled` servers to reduce overhead for validators. If you run a validator and want to manage ledger history using sharding, run a separate `rippled` server with sharding enabled.
+
+For more information, reference the `[shard_db]` example in the [rippled.cfg configuration example](https://github.com/ripple/rippled/blob/master/doc/rippled-example.cfg).
+
+### Sizing the Shard Store
+Determining a suitable size for the shard store involves careful consideration. You should consider the following when deciding what size your shard store should be:
+
+- Although redundant, it is possible to hold full ledger history in the ledger store and the history shard store.
+- An effective configuration might limit the ledger store only to recent history.
+- The ledger store history size should at minimum be twice the ledgers per shard, due to the fact that the current shard may be chosen to be stored and it would be wasteful to reacquire that data.
+- The time to acquire, number of file handles, and memory cache usage is directly affected by sizing.
+- Each shard contains 2^14 ledgers (16384).
+- A shard occupies approximately 200 MB to 4 GB based on the age of the shard. Older shards are smaller because there was less activity in the XRP Ledger at the time.
+
+{% include '_snippets/rippled_versions.md' %}

content/concepts/the-rippled-server/stand-alone-mode.md

@@ -0,0 +1,109 @@
+# Stand-Alone Mode
+
+You can run `rippled` in stand-alone mode without a consensus of trusted servers. In stand-alone mode, `rippled` does not communicate with any other servers in the XRP Ledger peer-to-peer network, but you can do most of the same actions on your local server only. Stand-alone provides a method for testing `rippled` behavior without being tied to the live network. For example, you can [test the effects of Amendments](concept-amendments.html#testing-amendments) before those Amendments have gone into effect across the decentralized network.
+
+When you run `rippled` in stand-alone mode, you have to tell it what ledger version to start from:
+
+* Create a [new genesis ledger](#new-genesis-ledger) from scratch.
+* [Load an existing ledger version](#load-saved-ledger) from disk.
+
+**Caution:** In stand-alone mode, you must [manually advance the ledger](#advancing-ledgers-in-stand-alone-mode).
+
+
+## New Genesis Ledger
+
+In stand-alone mode, you can have `rippled` create a new genesis ledger. This provides a known state, with none of the ledger history from the production XRP Ledger. (This is very useful for unit tests, among other things.)
+
+* To start `rippled` in stand-alone mode with a new genesis ledger, use the [`-a`](https://wiki.ripple.com/Rippled#--standalone.2C_-a) and [`--start`](https://wiki.ripple.com/Rippled#--start) options:
+
+```
+rippled -a --start --conf=/path/to/rippled.cfg
+```
+
+In a genesis ledger, the [genesis address](concept-accounts.html#special-addresses) holds all 100 billion XRP. The keys of the genesis address are [hardcoded](https://github.com/ripple/rippled/blob/94ed5b3a53077d815ad0dd65d490c8d37a147361/src/ripple/app/ledger/Ledger.cpp#L184) as follows:
+
+**Address:** `rHb9CJAWyB4rj91VRWn96DkukG4bwdtyTh`
+
+**Secret:** `snoPBrXtMeMyMHUVTgbuqAfg1SUTb` ("masterpassphrase")
+
+### Settings in New Genesis Ledgers
+
+In a new genesis ledger, the hard-coded default [Reserve](concept-reserves.html) is **200 XRP** minimum for funding a new address, with an increment of **50 XRP** per object in the ledger. These values are higher than the current reserve requirements of the production network. (See also: [Fee Voting](concept-fee-voting.html))
+
+By default, a new genesis ledger has no [amendments](concept-amendments.html) enabled. If you start a new genesis ledger with `--start`, the genesis ledger contains an [EnableAmendment pseudo-transaction](reference-transaction-format.html#enableamendment) to turn on all amendments natively supported by the `rippled` server, except for amendments that you explicitly disable in the configuration file. The effects of those amendments are available starting from the very next ledger version. (Reminder: in stand-alone mode, you must [advance the ledger manually](#advancing-ledgers-in-stand-alone-mode).) [New in: rippled 0.50.0][]
+
+
+## Load Saved Ledger
+
+You can start with a ledger version that was saved to disk if your `rippled` server was previously synced with the XRP Ledger peer-to-peer network (either the production network or the [Test Net](tutorial-rippled-setup.html#parallel-networks)).
+
+### 1. Start `rippled` normally.
+
+To load an existing ledger, you must first retrieve that ledger from the network. Start `rippled` in online mode as normal:
+
+```
+rippled --conf=/path/to/rippled.cfg
+```
+
+### 2. Wait until `rippled` is synced.
+
+Use the [`server_info` command](reference-rippled.html#server-info) to check the state of your server relative to the network. Your server is synced when the `server_state` value shows any of the following values:
+
+* `full`
+* `proposing`
+* `validating`
+
+For more information, see [Possible Server States](reference-rippled.html#possible-server-states).
+
+### 3. (Optional) Retrieve specific ledger versions.
+
+If you only want the most recent ledger, you can skip this step.
+
+If you want to load a specific historical ledger version, use the [`ledger_request` command](reference-rippled.html#ledger-request) to make `rippled` fetch it. If `rippled` does not already have the ledger version, you may have to run the `ledger_request` command multiple times until it has finished retrieving the ledger.
+
+If you want to replay a specific historical ledger version, you must fetch both the ledger version to replay and the ledger version before it. (The previous ledger version sets up the initial state upon which you apply the changes described by the ledger version you replay.)
+
+### 4. Shut down `rippled`.
+
+Use the [`stop` command](reference-rippled.html#stop):
+
+```
+rippled stop --conf=/path/to/rippled.cfg
+```
+
+### 5. Start `rippled` in stand-alone mode.
+
+To load the most recent ledger version, you can use the [`-a`](https://wiki.ripple.com/Rippled#--standalone.2C_-a) and [`--load`](https://wiki.ripple.com/Rippled#--load) options when starting the server:
+
+```
+rippled -a --load --conf=/path/to/rippled.cfg
+```
+
+To instead load a specific historical ledger, use the [`--load`](https://wiki.ripple.com/Rippled#--load) parameter along with the `--ledger` parameter, providing the ledger index or identifying hash of the ledger version to load:
+
+```
+rippled -a --load --ledger 19860944 --conf=/path/to/rippled.cfg
+```
+
+### 6. Manually advance the ledger.
+
+When you load a ledger with `--ledger` in stand-alone mode, it goes to the current open ledger, so you must [manually advance the ledger](#advancing-ledgers-in-stand-alone-mode):
+
+```
+rippled ledger_accept --conf=/path/to/rippled.cfg
+```
+
+
+## Advancing Ledgers in Stand-Alone Mode
+
+In stand-alone mode, `rippled` does not communicate to other members of the peer-to-peer network or participate in a consensus process. Instead, you must manually advance the ledger index using the [`ledger_accept` command](reference-rippled.html#ledger-accept):
+
+```
+rippled ledger_accept --conf=/path/to/rippled.cfg
+```
+
+In stand-alone mode, `rippled` makes no distinction between a "closed" ledger version and a "validated" ledger version. (For more information about the difference, see [The XRP Ledger Consensus Process](concept-consensus.html).)
+
+Whenever `rippled` closes a ledger, it reorders the transactions according to a deterministic but hard-to-game algorithm. (This is an important part of consensus, since transactions may arrive at different parts of the network in different order.) When using `rippled` in stand-alone mode, you should manually advance the ledger before submitting a transaction that depends on the result of a transaction from a different address. Otherwise, the two transactions might be executed in reverse order when the ledger is closed. **Note:** You can safely submit multiple transactions from a single address to a single ledger, because `rippled` sorts transactions from the same address in ascending order by [`Sequence` number](reference-transaction-format.html#common-fields).
+
+{% include '_snippets/rippled_versions.md' %}

content/concepts/the-rippled-server/the-rippled-server.md

@@ -0,0 +1,63 @@
+# The rippled Server
+
+`rippled` is the core peer-to-peer server that manages the XRP Ledger. This section covers concepts that help you learn the "what" and "why" behind fundamental aspects of the `rippled` server.
+
+Other pages in this category:
+
+* **[History Sharding](history-sharding.html)**
+
+      Historical sharding distributes the transaction history of the XRP Ledger into segments, called shards, across servers in the XRP Ledger network. Understand the purpose of history sharding and when to use it.
+<!--{# TODO: Is this History Sharding or Historical Sharding? #}-->
+
+* **[Clustering](clustering.html)**
+
+      If you are running multiple `rippled` servers in a single datacenter, you can configure those servers into a cluster to maximize efficiency. Understand the purpose of clustering and when to use it.
+
+
+## Types of rippled Servers
+
+The `rippled` server software can run in several modes depending on its configuration, including:
+
+* Stock server - follows the network with a local copy of the ledger.
+* Validating server, or _validator_ for short - participates in consensus.
+* `rippled` server in stand-alone mode - for testing. Does not communicate to other `rippled` servers.
+
+You can also run the `rippled` executable as a client application for accessing [`rippled` APIs](reference-rippled.html) locally. (Two instances of the same binary can run side-by-side in this case; one as a server, and the other running briefly as a client and then terminating.)
+
+## Reasons to Run a Stock Server
+
+There are lots of reasons you might want to run your own `rippled` server, but most of them can be summarized as: you can trust your own server, you have control over its workload, and you're not at the mercy of others to decide when and how you can access it. Of course, you must practice good network security to protect your server from malicious hackers.
+
+You need to trust the `rippled` you use. If you connect to a malicious server, there are many ways that it can take advantage of you or cause you to lose money. For example:
+
+* A malicious server could report that you were paid when no such payment was made.
+* It could selectively show or hide payment paths and currency exchange offers to guarantee its own profit while not providing you the best deal.
+* If you sent it your address's secret key, it could make arbitrary transactions on your behalf, and even transfer or destroy all the money your address holds.
+
+Additionally, running your own server gives you admin control over it, which allows you to run important admin-only and load-intensive commands. If you use a shared server, you have to worry about other users of the same server competing with you for the server's computing power. Many of the commands in the WebSocket API can put a lot of strain on the server, so `rippled` has the option to scale back its responses when it needs to. If you share a server with others, you may not always get the best results possible.
+
+Finally, if you run a validating server, you can use a stock server as a proxy to the public network while keeping your validating server on a private subnet only accessible to the outside world through the stock server. This makes it more difficult to compromise the integrity of your validating server.
+
+
+## Reasons to Run a Validator
+
+The robustness of the XRP Ledger depends on an interconnected web of validators who each trust a few other validators _not to collude_. The more operators with different interests there are who run validators, the more certain each member of the network can be that it continues to run impartially. If you or your organization relies on the XRP Ledger, it is in your interest to contribute to the consensus process.
+
+Not all `rippled` servers need to be validators: trusting more servers from the same operator does not offer better protection against collusion. An organization might run validators in multiple regions for redundancy in case of natural disasters and other emergencies.
+
+If your organization runs a validating server, you may also run one or more stock servers, to balance the computing load of API access, or as a proxy between your validation server and the outside network.
+
+### Properties of a Good Validator
+
+There are several properties that define a good validator. The more of these properties your server embodies, the more reason others have to include your server in their list of trusted validators:
+
+* **Availability**. An ideal validator should always be running, submitting validation votes for every proposed ledger.
+    * Strive for 100% uptime.
+* **Agreement**. A validator's votes should match the outcome of the consensus process as often as possible. To do otherwise could indicate that the validator's software is outdated, buggy, or intentionally biased.
+    * Always run the latest `rippled` release without modifications.
+* **Timeliness**. A validator's votes should arrive quickly, and not after a consensus round has already finished.
+    * A fast internet connection helps with this.
+* **Identified**. It should be clear who runs the validator. Ideally, a list of trusted validators should include validators operated by different owners in multiple legal jurisdictions and geographic areas, to reduce the chance that any localized events could interfere with the validator's impartial operation.
+    * Setting up [Domain Verification](#domain-verification) is a good start.
+
+At present, Ripple (the company) cannot recommend any validators aside from those in the default validator list. However, we are collecting data on other validators and building tools to report on their performance. For metrics on validators, see [validators.ripple.com](https://validators.ripple.com).