mirror of
https://github.com/XRPLF/rippled.git
synced 2025-12-06 17:27:55 +00:00
262 lines
21 KiB
HTML
262 lines
21 KiB
HTML
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "https://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
|
|
<html xmlns="http://www.w3.org/1999/xhtml">
|
|
<head>
|
|
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
|
|
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
|
|
<meta name="generator" content="Doxygen 1.8.17"/>
|
|
<meta name="viewport" content="width=device-width, initial-scale=1"/>
|
|
<title>rippled: Database Documentation</title>
|
|
<link href="tabs.css" rel="stylesheet" type="text/css"/>
|
|
<script type="text/javascript" src="jquery.js"></script>
|
|
<script type="text/javascript" src="dynsections.js"></script>
|
|
<link href="search/search.css" rel="stylesheet" type="text/css"/>
|
|
<script type="text/javascript" src="search/searchdata.js"></script>
|
|
<script type="text/javascript" src="search/search.js"></script>
|
|
<link href="doxygen.css" rel="stylesheet" type="text/css" />
|
|
</head>
|
|
<body>
|
|
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
|
|
<div id="titlearea">
|
|
<table cellspacing="0" cellpadding="0">
|
|
<tbody>
|
|
<tr style="height: 56px;">
|
|
<td id="projectalign" style="padding-left: 0.5em;">
|
|
<div id="projectname">rippled
|
|
</div>
|
|
</td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
</div>
|
|
<!-- end header part -->
|
|
<!-- Generated by Doxygen 1.8.17 -->
|
|
<script type="text/javascript">
|
|
/* @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&dn=gpl-2.0.txt GPL-v2 */
|
|
var searchBox = new SearchBox("searchBox", "search",false,'Search');
|
|
/* @license-end */
|
|
</script>
|
|
<script type="text/javascript" src="menudata.js"></script>
|
|
<script type="text/javascript" src="menu.js"></script>
|
|
<script type="text/javascript">
|
|
/* @license magnet:?xt=urn:btih:cf05388f2679ee054f2beb29a391d25f4e673ac3&dn=gpl-2.0.txt GPL-v2 */
|
|
$(function() {
|
|
initMenu('',true,false,'search.php','Search');
|
|
$(document).ready(function() { init_search(); });
|
|
});
|
|
/* @license-end */</script>
|
|
<div id="main-nav"></div>
|
|
<!-- window showing the filter options -->
|
|
<div id="MSearchSelectWindow"
|
|
onmouseover="return searchBox.OnSearchSelectShow()"
|
|
onmouseout="return searchBox.OnSearchSelectHide()"
|
|
onkeydown="return searchBox.OnSearchSelectKey(event)">
|
|
</div>
|
|
|
|
<!-- iframe showing the search results (closed by default) -->
|
|
<div id="MSearchResultsWindow">
|
|
<iframe src="javascript:void(0)" frameborder="0"
|
|
name="MSearchResults" id="MSearchResults">
|
|
</iframe>
|
|
</div>
|
|
|
|
</div><!-- top -->
|
|
<div class="PageDoc"><div class="header">
|
|
<div class="headertitle">
|
|
<div class="title">Database Documentation </div> </div>
|
|
</div><!--header-->
|
|
<div class="contents">
|
|
<div class="textblock"><ul>
|
|
<li><a href="#nodestore">NodeStore</a></li>
|
|
<li><a href="#benchmarks">Benchmarks</a></li>
|
|
<li><a href="#downloaded-shard-validation">Downloaded Shard Validation</a></li>
|
|
<li><a href="#shard-storage-paths">Shard Storage Paths</a></li>
|
|
</ul>
|
|
<h1><a class="anchor" id="autotoc_md179"></a>
|
|
NodeStore</h1>
|
|
<h2><a class="anchor" id="autotoc_md180"></a>
|
|
Introduction</h2>
|
|
<p>A <code>NodeObject</code> is a simple object that the Ledger uses to store entries. It is comprised of a type, a hash and a blob. It can be uniquely identified by the hash, which is a 256 bit hash of the blob. The blob is a variable length block of serialized data. The type identifies what the blob contains. The fields are as follows:</p>
|
|
<ul>
|
|
<li><p class="startli"><code>mType</code></p>
|
|
<p class="startli">An enumeration that determines what the blob holds. There are four different types of objects stored.</p><ul>
|
|
<li><p class="startli"><b>ledger</b></p>
|
|
<p class="startli">A ledger header.</p>
|
|
</li>
|
|
<li><p class="startli"><b>transaction</b></p>
|
|
<p class="startli">A signed transaction.</p>
|
|
</li>
|
|
<li><p class="startli"><b>account node</b></p>
|
|
<p class="startli">A node in a ledger's account state tree.</p>
|
|
</li>
|
|
<li><p class="startli"><b>transaction node</b></p>
|
|
<p class="startli">A node in a ledger's transaction tree.</p>
|
|
</li>
|
|
</ul>
|
|
</li>
|
|
<li><p class="startli"><code>mHash</code></p>
|
|
<p class="startli">A 256-bit hash of the blob.</p>
|
|
</li>
|
|
<li><p class="startli"><code>mData</code></p>
|
|
<p class="startli">A blob containing the payload. Stored in the following format.</p>
|
|
</li>
|
|
</ul>
|
|
<table class="markdownTable">
|
|
<tr class="markdownTableHead">
|
|
<th class="markdownTableHeadLeft">Byte </th><th class="markdownTableHeadLeft"></th><th class="markdownTableHeadLeft"></th></tr>
|
|
<tr class="markdownTableRowOdd">
|
|
<td class="markdownTableBodyLeft">0...7 </td><td class="markdownTableBodyLeft">unused </td><td class="markdownTableBodyLeft"></td></tr>
|
|
<tr class="markdownTableRowEven">
|
|
<td class="markdownTableBodyLeft">8 </td><td class="markdownTableBodyLeft">type </td><td class="markdownTableBodyLeft">NodeObjectType enumeration </td></tr>
|
|
<tr class="markdownTableRowOdd">
|
|
<td class="markdownTableBodyLeft">9...end </td><td class="markdownTableBodyLeft">data </td><td class="markdownTableBodyLeft">body of the object data </td></tr>
|
|
</table>
|
|
<hr />
|
|
<p> The <code>NodeStore</code> provides an interface that stores, in a persistent database, a collection of NodeObjects that rippled uses as its primary representation of ledger entries. All ledger entries are stored as NodeObjects and as such, need to be persisted between launches. If a NodeObject is accessed and is not in memory, it will be retrieved from the database.</p>
|
|
<h2><a class="anchor" id="autotoc_md181"></a>
|
|
Backend</h2>
|
|
<p>The <code>NodeStore</code> implementation provides the <code>Backend</code> abstract interface, which lets different key/value databases to be chosen at run-time. This allows experimentation with different engines. Improvements in the performance of the NodeStore are a constant area of research. The database can be specified in the configuration file [node_db] section as follows.</p>
|
|
<p>One or more lines of key / value pairs</p>
|
|
<p>Example: </p><div class="fragment"><div class="line">type=RocksDB</div>
|
|
<div class="line">path=rocksdb</div>
|
|
<div class="line">compression=1</div>
|
|
</div><!-- fragment --><p>Choices for 'type' (not case-sensitive)</p>
|
|
<ul>
|
|
<li><p class="startli"><b>HyperLevelDB</b></p>
|
|
<p class="startli">An improved version of LevelDB (preferred).</p>
|
|
</li>
|
|
<li><p class="startli"><b>LevelDB</b></p>
|
|
<p class="startli">Google's LevelDB database (deprecated).</p>
|
|
</li>
|
|
<li><p class="startli"><b>none</b></p>
|
|
<p class="startli">Use no backend.</p>
|
|
</li>
|
|
<li><p class="startli"><b>RocksDB</b></p>
|
|
<p class="startli">Facebook's RocksDB database, builds on LevelDB.</p>
|
|
</li>
|
|
<li><p class="startli"><b>SQLite</b></p>
|
|
<p class="startli">Use SQLite.</p>
|
|
</li>
|
|
</ul>
|
|
<p>'path' speficies where the backend will store its data files.</p>
|
|
<p>Choices for 'compression'</p>
|
|
<ul>
|
|
<li><b>0</b> off</li>
|
|
<li><b>1</b> on (default)</li>
|
|
</ul>
|
|
<h1><a class="anchor" id="autotoc_md182"></a>
|
|
Benchmarks</h1>
|
|
<p>The <code>NodeStore.Timing</code> test is used to execute a set of read/write workloads to compare current available nodestore backends. It can be executed with:</p>
|
|
<div class="fragment"><div class="line">$rippled --unittest=NodeStoreTiming</div>
|
|
</div><!-- fragment --><p>It is also possible to use alternate DB config params by passing config strings as <code>--unittest-arg</code>.</p>
|
|
<h2><a class="anchor" id="autotoc_md183"></a>
|
|
Addendum</h2>
|
|
<p>The discussion below refers to a <code>RocksDBQuick</code> backend that has since been removed from the code as it was not working and not maintained. That backend primarily used one of the several rocks <code>Optimize*</code> methods to setup the majority of the DB options/params, whereas the primary RocksDB backend exposes many of the available config options directly. The code for RocksDBQuick can be found in versions of this repo 1.2 and earlier if you need to refer back to it. The conclusions below date from about 2014 and may need revisiting based on newer versions of RocksDB (TBD).</p>
|
|
<h2><a class="anchor" id="autotoc_md184"></a>
|
|
Discussion</h2>
|
|
<p>RocksDBQuickFactory is intended to provide a testbed for comparing potential rocksdb performance with the existing recommended configuration in rippled.cfg. Through various executions and profiling some conclusions are presented below.</p>
|
|
<ul>
|
|
<li>If the write ahead log is enabled, insert speed soon clogs up under load. The BatchWriter class intends to stop this from blocking the main threads by queuing up writes and running them in a separate thread. However, rocksdb already has separate threads dedicated to flushing the memtable to disk and the memtable is itself an in-memory queue. The result is two queues with a guarantee of durability in between. However if the memtable was used as the sole queue and the rocksdb::Flush() call was manually triggered at opportune moments, possibly just after ledger close, then that would provide similar, but more predictable guarantees. It would also remove an unneeded thread and unnecessary memory usage. An alternative point of view is that because there will always be many other rippled instances running there is no need for such guarantees. The nodes will always be available from another peer.</li>
|
|
<li>Lookup in a block was previously using binary search. With rippled's use case it is highly unlikely that two adjacent key/values will ever be requested one after the other. Therefore hash indexing of blocks makes much more sense. Rocksdb has a number of options for hash indexing both memtables and blocks and these need more testing to find the best choice.</li>
|
|
<li>The current Database implementation has two forms of caching, so the LRU cache of blocks at Factory level does not make any sense. However, if the hash indexing and potentially the new <a href="http://rocksdb.org/blog/1427/new-bloom-filter-format/">bloom filter</a> can provide faster lookup for non-existent keys, then potentially the caching could exist at Factory level.</li>
|
|
<li>Multiple runs of the benchmarks can yield surprisingly different results. This can perhaps be attributed to the asynchronous nature of rocksdb's compaction process. The benchmarks are artifical and create highly unlikely write load to create the dataset to measure different read access patterns. Therefore multiple runs of the benchmarks are required to get a feel for the effectiveness of the changes. This contrasts sharply with the keyvadb benchmarking were highly repeatable timings were discovered. Also realistically sized datasets are required to get a correct insight. The number of 2,000,000 key/values (actually 4,000,000 after the two insert benchmarks complete) is too low to get a full picture.</li>
|
|
<li>An interesting side effect of running the benchmarks in a profiler was that a clear pattern of what RocksDB does under the hood was observable. This led to the decision to trial hash indexing and also the discovery of the native CRC32 instruction not being used.</li>
|
|
<li>Important point to note that is if this factory is tested with an existing set of sst files none of the old sst files will benefit from indexing changes until they are compacted at a future point in time.</li>
|
|
</ul>
|
|
<h1><a class="anchor" id="autotoc_md185"></a>
|
|
Downloaded Shard Validation</h1>
|
|
<h2><a class="anchor" id="autotoc_md186"></a>
|
|
Overview</h2>
|
|
<p>In order to validate shards that have been downloaded from file servers (as opposed to shards acquired from peers), the application must confirm the validity of the downloaded shard's last ledger. So before initiating the download, we first confirm that we are able to retrieve the shard's last ledger hash. The following sections describe this confirmation process in greater detail.</p>
|
|
<h2><a class="anchor" id="autotoc_md187"></a>
|
|
Hash Verification</h2>
|
|
<h3><a class="anchor" id="autotoc_md188"></a>
|
|
Flag Ledger</h3>
|
|
<p>Since the number of ledgers contained in each shard is always a multiple of 256, a shard's last ledger is always a flag ledger. Conveniently, the skip list stored within a ledger will provide us with a series of flag ledger hashes, enabling the software to corroborate a shard's last ledger hash. We access the skip list by calling <code>LedgerMaster::walkHashBySeq</code> and providing the sequence of a shard's last ledger:</p>
|
|
<div class="fragment"><div class="line"> {C++}</div>
|
|
<div class="line">std::optional<uint256> expectedHash;</div>
|
|
<div class="line">expectedHash =</div>
|
|
<div class="line"> app_.getLedgerMaster().walkHashBySeq(lastLedgerSeq(shardIndex));</div>
|
|
</div><!-- fragment --><p>When a user requests a shard download, the <code>ShardArchiveHandler</code> will first use this function to retrieve the hash of the shard's last ledger. If the function returns a hash, downloading the shard can proceed. Once the download completes, the server can reliably retrieve this last ledger hash to complete validation of the shard.</p>
|
|
<h3><a class="anchor" id="autotoc_md189"></a>
|
|
Caveats</h3>
|
|
<h4><a class="anchor" id="autotoc_md190"></a>
|
|
Later Ledger</h4>
|
|
<p>The <code>walkHashBySeq</code> function will provide the hash of a flag ledger only if the application has stored a later ledger. When verifying the last ledger hash of a pending shard download, if there is no later ledger stored, the download will be deferred until a later ledger has been stored.</p>
|
|
<p>We use the presence (or absence) of a validated ledger with a sequence number later than the sequence of the shard's last ledger as a heuristic for determining whether or not we should have the shard's last ledger hash. A later ledger must be present in order to reliably retrieve the hash of the shard's last ledger. The hash will only be retrieved when a later ledger is present. Otherwise verification of the shard will be deferred.</p>
|
|
<h3><a class="anchor" id="autotoc_md191"></a>
|
|
Retries</h3>
|
|
<h4><a class="anchor" id="autotoc_md192"></a>
|
|
Retry Limit</h4>
|
|
<p>If the server must defer hash verification, the software will initiate a timer that upon expiration, will re-attempt verifying the last ledger hash. We place an upper limit on the number of attempts the server makes to achieve this verification. When the maximum number of attempts has been reached, the download request will fail, and the <code>ShardArchiveHandler</code> will proceed with any remaining downloads. An attempt counts toward the limit only when we are able to get a later validated ledger (implying a current view of the network), but are unable to retrieve the last ledger hash. Retries that occur because no validated ledger was available are not counted.</p>
|
|
<h1><a class="anchor" id="autotoc_md193"></a>
|
|
Shard Storage Paths</h1>
|
|
<h2><a class="anchor" id="autotoc_md194"></a>
|
|
Overview</h2>
|
|
<p>The shard database stores validated ledgers in logical groups called shards. As of June 2020, a shard stores 16384 ledgers by default. In order to allow users to store shards on multiple devices, the shard database can be configured with several file system paths. Each path provided should refer to a directory on a distinct filesystem, and no two paths should ever correspond to the same filesystem. Violating this restriction will cause the server to inaccurately estimate the amount of space available for storing shards. In the absence of a suitable platform agnostic solution, this requirement is enforced only on Linux. However, on other platforms we employ a heuristic that issues a warning if we suspect that this restriction is violated.</p>
|
|
<h2><a class="anchor" id="autotoc_md195"></a>
|
|
Configuration</h2>
|
|
<p>The <code>shard_db</code> and <code>historical_shard_paths</code> sections of the server's configuration file will be used to determine where the server stores shards. Minimally, the <code>shard_db</code> section must contain a single <code>path</code> key. If this is the only storage path provided, all shards will be stored at this location. If the configuration also lists one or more lines in the <code>historical_shard_paths</code> section, all older shards will be stored at these locations, and the <code>path</code> will be used only to store the current and previous shards. The goal is to allow users to provide an efficient SSD for storing recent shards, as these will be accessed more frequently, while using large mechanical drives for storing older shards that will be accessed less frequently.</p>
|
|
<p>Below is a sample configuration snippet that provides a path for main storage and several paths for historical storage:</p>
|
|
<div class="fragment"><div class="line"># This is the persistent datastore for shards. It is important for the health</div>
|
|
<div class="line"># of the network that server operators shard as much as practical.</div>
|
|
<div class="line"># NuDB requires SSD storage. Helpful information can be found on</div>
|
|
<div class="line"># https://xrpl.org/history-sharding.html</div>
|
|
<div class="line">[shard_db]</div>
|
|
<div class="line">type=NuDB</div>
|
|
<div class="line"> </div>
|
|
<div class="line"># A single path for storing</div>
|
|
<div class="line"># the current and previous</div>
|
|
<div class="line"># shards:</div>
|
|
<div class="line"># -------------------------</div>
|
|
<div class="line">path=/var/lib/rippled/db/shards/nudb</div>
|
|
<div class="line"> </div>
|
|
<div class="line"># Path where shards are stored</div>
|
|
<div class="line"># while being downloaded:</div>
|
|
<div class="line"># ----------------------------</div>
|
|
<div class="line">download_path=/var/lib/rippled/db/shards/</div>
|
|
<div class="line"> </div>
|
|
<div class="line"># The number of historical shards to store.</div>
|
|
<div class="line"># The default value is 0, which means that</div>
|
|
<div class="line"># the server won't store any historical</div>
|
|
<div class="line"># shards - only the current and previous</div>
|
|
<div class="line"># shards will be stored.</div>
|
|
<div class="line"># ------------------------------------</div>
|
|
<div class="line">max_historical_shards=100</div>
|
|
<div class="line"> </div>
|
|
<div class="line"># List of paths for storing older shards.</div>
|
|
<div class="line">[historical_shard_paths]</div>
|
|
<div class="line">/mnt/disk1</div>
|
|
<div class="line">/mnt/disk2</div>
|
|
<div class="line">/mnt/disk3</div>
|
|
</div><!-- fragment --><h2><a class="anchor" id="autotoc_md196"></a>
|
|
Shard Migration</h2>
|
|
<p>When a new shard (<em>current shard</em>) is confirmed by the network, the recent shards will shift. The <em>previous shard</em> will become a <em>historical shard</em>, the <em>current shard</em> will become the <em>previous shard</em>, and the new shard will become the <em>current shard</em>. These are just logical labels, and the shards themselves don't change to reflect being current, previous, or historical. However, if the server's configuration specifies one or more paths for historical storage, during this shift the formerly <em>previous shard</em> will be migrated to one of the historical paths. If multiple paths are provided, the server dynamically chooses one with sufficient space for storing the shard.</p>
|
|
<p><b>Note:</b> As of June 2020, the shard database does not store the partial shard currently being built by live network transactions, but this is planned to change. When this feature is implemented, the <em>current shard</em> will refer to this partial shard, and the <em>previous shard</em> will refer to the most recently validated shard.</p>
|
|
<h3><a class="anchor" id="autotoc_md197"></a>
|
|
Selecting a Historical Storage Path</h3>
|
|
<p>When storing historical shards, if multiple historical paths are provided, the path to use for each shard will be selected in a random fashion. By using all available storage devices, we create a uniform distribution of disk utilization for disks of equivalent size, (provided that the disks are used only to store shards). In theory, selecting devices in this manner will also increase our chances for concurrent access to stored shards, however as of June 2020 concurrent shard access is not implemented. Lastly, a storage path is included in the random distribution only if it has enough storage capacity to hold the next shard.</p>
|
|
<h2><a class="anchor" id="autotoc_md198"></a>
|
|
Shard Acquisition</h2>
|
|
<p>When the server is acquiring shard history, these acquired shards will be stored at a path designated for historical storage (<code>historical_storage_path</code>). If no such path is provided, acquired shards will be stored at the <code>path</code>.</p>
|
|
<h2><a class="anchor" id="autotoc_md199"></a>
|
|
Storage capacity</h2>
|
|
<h3><a class="anchor" id="autotoc_md200"></a>
|
|
Filesystem Capacity</h3>
|
|
<p>When the shard database updates its record of disk utilization, it trusts that the provided historical paths refer to distinct devices, or at least distinct filesystems. If this requirement is violated, the database will operate with an inaccurate view of how many shards it can store. Violation of this requirement won't necessarily impede database operations, but the database will fail to identify scenarios wherein storing the maximum number of historical shards (as per the 'historical_shard_count' parameter in the configuration file) would exceed the amount of storage space available.</p>
|
|
<h3><a class="anchor" id="autotoc_md201"></a>
|
|
Shard Migration</h3>
|
|
<p>During a "recent shard shift", if the server has already reached the configured limit of stored historical shards, instead of moving the formerly <em>previous shard</em> to a historical drive (or keeping it at the 'path') the shard will be dropped and removed from the filesystem.</p>
|
|
<h3><a class="anchor" id="autotoc_md202"></a>
|
|
Shard Acquisition</h3>
|
|
<p>Once the configured limit of stored historical shards has been reached, shard acquisition halts, and no additional shards will be acquired. </p>
|
|
</div></div><!-- contents -->
|
|
</div><!-- PageDoc -->
|
|
<!-- start footer part -->
|
|
<hr class="footer"/><address class="footer"><small>
|
|
Generated by  <a href="http://www.doxygen.org/index.html">
|
|
<img class="footer" src="doxygen.png" alt="doxygen"/>
|
|
</a> 1.8.17
|
|
</small></address>
|
|
</body>
|
|
</html>
|