From 45e7b2aca0333514abcf12baf0d6b6736e7b1346 Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Tue, 4 Dec 2018 19:34:07 -0800 Subject: [PATCH 01/14] Online deletion: concept 1st draft w/ diagrams --- content/_img-sources/fetch_depth.uxf | 87 ++++++++++++ .../online_delete-vs-ledger_history.uxf | 107 ++++++++++++++ .../the-rippled-server/online-deletion.md | 131 ++++++++++++++++++ dactyl-config.yml | 9 ++ img/fetch_depth.png | Bin 0 -> 10064 bytes img/online_delete-vs-ledger_history.png | Bin 0 -> 8121 bytes 6 files changed, 334 insertions(+) create mode 100644 content/_img-sources/fetch_depth.uxf create mode 100644 content/_img-sources/online_delete-vs-ledger_history.uxf create mode 100644 content/concepts/the-rippled-server/online-deletion.md create mode 100644 img/fetch_depth.png create mode 100644 img/online_delete-vs-ledger_history.png diff --git a/content/_img-sources/fetch_depth.uxf b/content/_img-sources/fetch_depth.uxf new file mode 100644 index 0000000000..97480a876a --- /dev/null +++ b/content/_img-sources/fetch_depth.uxf @@ -0,0 +1,87 @@ + + + 10 + + UMLClass + + 30 + 70 + 210 + 40 + + Ledgers not stored +bg=#e1e4e8 +fg=#23292f +lt=.. + + + + Relation + + 20 + 110 + 740 + 50 + + lt=->> + +Ledger versions +m1=oldest +m2=newest + 10.0;20.0;720.0;20.0 + + + UMLClass + + 240 + 70 + 240 + 40 + + Ledgers available locally but not served to peers +style=wordwrap +bg=#999da2 +fg=#f5f7f9 +transparency=0 + + + + UMLClass + + 480 + 70 + 260 + 40 + + Ledgers served to peers +when requested +style=wordwrap +bg=#1db4ff +transparency=0 +fg=#f5f7f9 + + + + Text + + 270 + 30 + 170 + 30 + + fetch_depth setting +style=wordwrap + + + + Relation + + 410 + 30 + 90 + 60 + + lt=<<<- + 70.0;40.0;70.0;10.0;10.0;10.0 + + diff --git a/content/_img-sources/online_delete-vs-ledger_history.uxf b/content/_img-sources/online_delete-vs-ledger_history.uxf new file mode 100644 index 0000000000..051c9efe45 --- /dev/null +++ b/content/_img-sources/online_delete-vs-ledger_history.uxf @@ -0,0 +1,107 @@ + + + 10 + + UMLClass + + 30 + 70 + 330 + 40 + + Delete automatically +bg=#e1e4e8 +fg=#23292f +lt=.. + + + + Relation + + 20 + 110 + 740 + 50 + + lt=->> + +Ledger versions +m1=oldest +m2=newest + 10.0;20.0;720.0;20.0 + + + UMLClass + + 360 + 70 + 170 + 40 + + Keep if available +bg=#999da2 +fg=#f5f7f9 +transparency=0 + + + + UMLClass + + 530 + 70 + 210 + 40 + + Backfill if possible +bg=#1db4ff +transparency=0 +fg=#f5f7f9 + + + + Relation + + 520 + 30 + 70 + 60 + + lt=<<<- + 10.0;40.0;10.0;10.0;50.0;10.0 + + + Text + + 570 + 30 + 210 + 30 + + ledger_history setting +style=wordwrap + + + + Text + + 150 + 30 + 170 + 30 + + online_delete setting +style=wordwrap + + + + Relation + + 300 + 30 + 80 + 60 + + lt=<<<- + 60.0;40.0;60.0;10.0;10.0;10.0 + + diff --git a/content/concepts/the-rippled-server/online-deletion.md b/content/concepts/the-rippled-server/online-deletion.md new file mode 100644 index 0000000000..41afbdcbd8 --- /dev/null +++ b/content/concepts/the-rippled-server/online-deletion.md @@ -0,0 +1,131 @@ +# Online Deletion +[[Source]
](https://github.com/ripple/rippled/blob/master/src/ripple/app/misc/SHAMapStoreImp.cpp "Source") + +The online deletion feature lets the `rippled` server delete the server's local copy of old ledger versions to keep disk usage from rapidly growing over time. Online deletion happens automatically by default, but can be configured to run only when prompted. [New in: rippled 0.27.0][] + +The server always keeps the complete _current_ state of the ledger, with all the balances and settings it contains. The deleted data includes older transactions and versions of the ledger state that are older than the stored history. + +The default config file sets the `rippled` server to keep the most recent 2000 ledger versions and automatically delete older data. + +**Tip:** Even with online deletion, the amount of disk space required to store the same time span's worth of ledger data increases over time, because the size of individual ledger versions tends to grow over time. This growth is very slow in comparison to the accumulation of data that occurs without deleting old ledgers. For more information on disk space needs, see [Capacity Planning](capacity-planning.html). + + +## Background + +The `rippled` server stores data from recent ledger versions in its _ledger store_. This data accumulates as the XRP Ledger network creates new ledger versions and validates them through the [consensus process](intro-to-consensus.html). + +The XRP Ledger stores the complete state of the current ledger, with all balances, settings, and other data, in each ledger version. Unlike many cryptocurrency systems, an XRP Ledger server does not require full history to sync with the network, know the full state, and contribute to making forward progress. + +Inside the ledger store, ledger data is "de-duplicated", with data that doesn't change from version to version only stored once. The records themselves in the ledger store do not indicate which ledger version(s) contain them; part of the work of online deletion is identifying which records are only used by outdated ledger versions. This process is time consuming and affects the disk I/O and application cache, so it is not feasible to delete old data on every ledger close. + +### Fetching History + +When it starts, a `rippled` server's first priority is to get a complete copy of the latest validated ledger. From there, it keeps up with advances in the ledger progress. If configured to do so, the server also backfills ledger history up to a configured amount, which must be equal or less than the cutoff beyond which online deletion is configured to delete. + +The server can backfill history from before it became synced, as well as filling in any gaps in the history it has collected after syncing. (Gaps in ledger history can occur if a server temporarily becomes too busy to keep up with the network, loses its network connection, or suffers other temporary issues.) To backfill history, the server requests data from its peer `rippled` servers. The amount the server tries to backfill is defined by the `[ledger_history]` setting. + +The XRP Ledger identifies data (on several different levels) by a unique hash of its contents. The XRP Ledger's state data contains a short summary of the ledger's history, in the form of the [LedgerHashes object type](ledgerhashes.html). Servers use the LedgerHashes objects to know which ledger versions to fetch, and to confirm that the ledger data they receive is correct and complete. + +Backfilling history is one of the server's lowest priorities, so it may take a long time to fill missing history, especially if the server is busy or has less than sufficient hardware and network specs. For recommendations on hardware specs, see [Capacity Planning](capacity-planning.html). Backfilling history also requires that at least one of the server's direct peers has the history in question. + +### Full History + +Some servers in the XRP Ledger network are configured as "full-history" servers. These servers, which require significantly more disk space than other tracking servers, collect all available XRP Ledger history and **do not use online deletion**. + + + +### History Sharding + +An alternative to storing the full history of the XRP Ledger on a single expensive machine is to configure many servers to each store a portion of ledger history. The [History Sharding](history-sharding.html) feature makes this possible, storing ranges of ledger history in a separate storage area called the _shard store_. When a peer server asks for specific data (as described in [fetching history](#fetching-history) above), a server can answer using data from either its ledger store or shard store. + +Online deletion **does not** delete from the shard store. However, if you configure online deletion to keep at least 32768 ledger versions in your server's ledger store, your server can copy full shards from the ledger store to the shard store before automatically deleting them from the ledger store. + +For more information, see [Configure History Sharding](configure-history-sharding.html). + + +## Online Deletion Behavior + +The online deletion settings configure how many ledger versions the `rippled` server should keep available in the ledger store at a time. However, the specified number is a guideline, not a hard rule: + +- The server may have less than the configured number of ledger versions if it has not been running for long enough or if it lost sync with the network at any time. (The server attempts to backfill at least some history; see [fetching history](#fetching-history) above for details.) +- The server may store up to twice the configured number of ledger versions if online deletion is set to run automatically. (Each time it runs, it reduces the number of stored ledger versions to the configured number.) +- If advisory delete is enabled, the server stores all the ledger versions that it has acquired and built until someone calls the [can_delete method][]. + + For example, if you call `can_delete` with a value of `now` once per day and an `online_delete` value of 2000, the server typically stores 2000 ledger versions at minimum and just over a full day's worth of ledger versions at maximum. + +With online deletion enabled and running automatically (that is, with advisory delete disabled), the total amount of ledger data stored should remain at minimum equal to the number of ledger versions the server is configured to keep, with the maximum being roughly twice that many. + +When online deletion runs, it does not reduce the size of the database files on disk; it only makes space within those files available to be reused for new data. To actually reduce the disk space allocated to the database files, you must restart `rippled`. + + +## Configuration + +The following settings relate to online deletion: + +- **`online_delete`** - Specify a number of validated ledger versions to keep. The server periodically deletes any ledger versions that are older than this number. If not specified, no ledgers are deleted. + + The default config file specifies 2000 for this value. This cannot be less than 256, because some events like [Fee Voting](fee-voting.html) and the [Amendment Process](amendments.html#amendment-process) update only every 256 ledgers. + + **Caution:** If you run `rippled` with `online_delete` disabled, then later enable `online_delete` and restart the server, the server disregards but does not delete existing ledger history that your server already downloaded while `online_delete` was disabled. To save disk space, delete your existing history before re-starting the server with `online_delete` back on. + +- **`[ledger_history]`** - Specify a number of validated ledgers, equal to or less than `online_delete`. If the server does not have at least this many validated ledger versions, it attempts to backfill them by fetching the data from peers. + + The default for this setting is 256 ledgers. + + The following diagram shows the relationship between `online_delete` and `ledger_history` settings: + + ![Ledgers older than `online_delete` are automatically deleted. Ledgers newer than `ledger_history` are backfilled. Ledgers in between are kept if available but not backfilled](img/online_delete-vs-ledger_history.png) + +- **`advisory_delete`** - If enabled, online deletion is not scheduled automatically. Instead, an administrator must manually trigger online deletion. Use the value `0` for disabled or `1` for enabled. + + This setting is disabled by default. + +- **`[fetch_depth]`** - Specify a number of ledger versions. The server does not accept fetch requests from peers for historical data that is older than the specified number of ledger versions. Specify the value `full` to serve any available data to peers. + + The default for `fetch_depth` is `full` (serve all available data). + + The `fetch_depth` setting cannot be higher than `online_delete` if both are specified. If `fetch_depth` is set higher, the server treats it as equal to `online_delete` instead. + + The following diagram shows how fetch_depth works: + + ![Ledger versions older than fetch_depth are not served to peers](img/fetch_depth.png) + + +### Relation to Real Time + +Online deletion settings must be configured in terms of the number of ledger versions, not in terms of real time. You can estimate the amount of time represented by a number of ledger versions using the rate of ledger progress. Ledger versions typically close 3-4 seconds apart, with the overall rate of ledger closures ranging from about 20,000 to 27,000 per day throughout the years 2017-2018. There are hard limits of at least 2 seconds and no more than 20 seconds between ledgers as long as consensus is operating properly. + +The following table summarizes some + +| Real Time Amount | Approximate Number of Ledgers | +|:-----------------|:------------------------------| +| 1 day | 25,000 | +| 14 days | 350,000 | +| 30 days | 750,000 | +| 90 days | 2,250,000 | +| 1 year | 10,000,000 | +| 2 years | 20,000,000 | + +These numbers are estimates and depend on several factors: + +- The rate of agreement among trusted validators. More disagreement slows the rate of new ledger creation. Disagreement can occur because of random chance, network latency, or validators running different software versions. +- The volume of transactions in the network. A high rate of transactions can increase the work needed to close a new ledger version, reducing the number of separate ledgers closed. An extremely low rate of transactions can also cause ledgers to close further apart (because there is nothing to be done). +- Changes and optimizations to the XRP Ledger's consensus algorithm can affect the speed of consensus and the rate of disagreement either positively or negatively. + +### Advisory Delete + +By default, online deletion happens automatically and periodically. If the advisory delete setting is enabled, online deletion only happens when an administrator triggers it using the [can_delete method][]. (You can trigger this command with a `cron` job to run automatic deletion based on actual time instead of the number of ledger versions closed.) + +If your server is heavily used, the extra load from online deletion can cause your server to fall behind and temporarily de-sync from the consensus network. If this is the case, you can use advisory delete and schedule online deletion to happen only during off-peak times. + +You can also use advisory delete for other reasons, such as if you want to manually confirm that the transaction data is backed up to a separate server before deleting it. + +The `can_delete` API method can enable advisory delete (with the value `never`) or disable advisory delete (with the value `always`). These setting changes persist until you restart the `rippled` server, at which point the settings in the config file override them. + + + + + +{% include '_snippets/rippled-api-links.md' %} +{% include '_snippets/tx-type-links.md' %} +{% include '_snippets/rippled_versions.md' %} diff --git a/dactyl-config.yml b/dactyl-config.yml index a67e7f9e90..cdf9f06032 100644 --- a/dactyl-config.yml +++ b/dactyl-config.yml @@ -545,6 +545,15 @@ pages: targets: - local + - md: concepts/the-rippled-server/online-deletion.md + html: online-deletion.html + funnel: Docs + doc_type: Concepts + category: The rippled Server + blurb: Online deletion purges outdated transaction and state history. + targets: + - local + - md: concepts/the-rippled-server/peer-protocol.md html: peer-protocol.html funnel: Docs diff --git a/img/fetch_depth.png b/img/fetch_depth.png new file mode 100644 index 0000000000000000000000000000000000000000..f760d6d91e35aaf5ed0d538ddeb4838fecba84c3 GIT binary patch literal 10064 zcmbVyWmr_**Eblqaid5p5()!I3(_qi9FQ()Q5uG30EbXQN{K^B4G2S*Fm!s;-93cF zh%kh7!+Y@lU)TFQpWf$r=fgfTd-mCTt-aP>_1ht8D)M)3Q{N^aAh-i4$Z8M}TqT0O z=iR&lwGrP;qzMREr~ujLS{_sDDYrbx<)+>f8reVhHoN|uisV(RQIXu%HDWDtl2fBS z-HAZ@fA*v&gR64>W+(~`#s@mj$zPEzA|NEad5_dNG9Wo1GN5~Ue2Fg_f5Chw#loV7 z_mN`wfpjEws#*2tQA(y<}HF|x*aB};2L7d!x38Qa-1N>vVUW) z@j}ju5Jbm8+{13A(e3m+>*o`0=d7G&P-0_eU+zg@Of@L8a>BwZISs4S3|xA?@F)|t zw6=C8n)n=b*YPpFxN)Nf&#aRC%GkK*i3|XQ6!?y6 z>&`ef-yvgud7tM^05f30G>CYN7dx1Cp%rnh+Mlw`ewm=5u0CHDSwz)_#TuBExZ%fU zYTZ-$ZR08u!{pyCbwnXf3Pl=OIt%X0$;rKK4~Jtb3XNv8scMgJEmISK*VHZ!cZXFk zw&NRrnOOFG8O+x$PD^`a-FS9-+9EwxWP~rnEcp{lMDF_xXDL&LF$oDB#_Adm2Bzs@ zER^E8di(q1y{5`-D?PT}$Pq%@4x=Qcq>PJ=Wja#w&kONOhn4=Vr>w;+D5wjf@2B%x zhyz9BS!ro$+W9(jJ|~51?83sHt0TGd^Yc`$d?U?pi`Gbr-J06!>J>K!D=I5UxqvG< zo1m74#!!}W62#lX6zh)2$KiC3h92qgILwN*6Up5~`5arH;lsEagb-`5F!fYaRF!s< zL(R=HC;B=%W7RH}B-h;lGb;CKryvv`a^G&U>m_K&`tr@2biO4PRa6;%!FkK7iUA03 zjeI*%=eYyzqV?b1-Q5m#TQ49AB4O`qo&ct%rb8$rb!Ci2Z=wL4&7{^1|MM4N5HWom zdSc>LHppk*MShka?DxxWayBC)gNfdW+`BVcRf07=jVokrVN};-RB&r&Ct544bwTTL zd2@5zVuYkm^Buve>T38*jrPd8U-(CMaOo9MdkrzY1PyeiAO+)6)UdM3DYN9^;ZYgq zd-uK~OW>$Y9sJ~yX%vMHx@I}J=T`gjLtU&K=$t!50t+Ru<{$ycI zrFZzk%8ka0b3eZ;>v-g5{rFdzAWnmdrRC+IfPnif_9l_>3KdgJxV1IA#TK|6up5H) z^z@u+fp5XG16@m=e)#Z#yYbXnR#p}O01pq3_HbHrcSuFn8ld&3g6PEGb)h0tno6!GLO)<^9#se9xVqMzsj<#m|C6!^7n`=>3CPqyv7ea8I7V>-=bWrpC>7 zvK06I!`06b1XYg!HZCqYL3=}N1zgl^mCxtYW3m57uOdTjq%z8TU$3Ohwy65WuJnaz zqtRm{j!$+HzUQYB-p6G&ql)|d@F~9ageyh#cWR3vIjO4;e&Kn$TO+wY@wUxW_Rnq; zR6XTd?0(@RE-QlEU(0n5c?=M|hY?CG#==ZY<{*+S%*@bPSvA$wT8$8z(NH6)h zlN=CmEt~jF9PYCo7#bEf=e5(@mm+H7y*KKAgZS?4+csbRb2|Q5;7jxS=dAocA3|3N z3AiM0-MXb~=p;u*Wd*TqW5Xs-bZLjJIHsXx%BeN_QR0#(YtOyz>!={7H)3taMC~He+7q!3nV4DLK3)3D-t7f z4dR!8PvfsU)Jt06RHXww6;=Bd;ia>nW`zjh)Gzz?481W^tBMj!@W1+Juno#<8_0Of z?=V}ZnXOV?T}|`glD{NFUT5gTu!aIaC%>uiSDnXA{N4(gWb zpm%w#_WQH3_O-!|3{ypLdT!W`Oo8;dVHpeGo!-**p<|hS0O&Uvx z6nOIr@TThXqe$2lfZ--A1O@YZUG@3T-%{YJ+RNnKaBMC8>xX;01je%{^p~o)^687K zZuwq(k6G6gJ}p{e6kbZudL1eh>KV1na=*G`6g27|*vo)iT6&Q%*e&R#Nc6`?Ky{p7^$z77y6qOY_s0G0 zL(OiTqQSUqON+*{V&97+Zg*mu-QP>O7jy>^g1Z6|oX>2BL^Ur)F1XrZm}!{t;fYe| zRK7zEfa?ED$F77}+ol!RuEw6+_4*OT(tr|`6cwB+vMUXVF|3UeGaz@J{*X<+HMjyUo0QZAoo{B>u?nk?*io8CU7fNn(fLxyzcs8VM6ck-}i-Ch^TY z^`*mW>y59@=RQZ=GOqQ}Y>q=G)#YW$#kJ#Jxr8ZZlor^VQcP`cVDsac;>^ion3wh9 z>A11Zd2JN$Hgfc1m&710Z_TCJf)?;jo!S6O~* z|EOGgI`{0H@s6*W!bImVE4|kteaVgbBtk0?I)&+IlxYxwGa&E{_|;PN$wpW*UP zvH5c;J5=29{BIY@BvAY;LVVNe;JfKr2V9~XtMvoWa+A=V+ho0nsrRtI07>VuEA{=c zHL+PX!;=#Eh|wudvkJ{xW%2~<=)%I?*ZxuH_i4?n)(NPjijZTwsa4K(oC>|99vw9a z)18ZB`bsW?6(=DveMDhWWLe8Q4FwPMe^iCP)gi9n6{88xM9H)y)%Sft_0-HWA-*>$ z%Q~1QXnhhA3U*=JH)XDwwZB@&Xlq@q$!31f*6Wspu%~b#lB}*%A`MEL>T9@;7-Yhe z*z@d|fpo?PlhvrCNWYUU z;!{nT?BXzAZU_2o&`js~Gr-1997t`Ak+;wbS}!Kw^ZfZbvuwIJU}EaWX2hgEPX3JT zq7Jhfbm*KLym9+)QX8;ARPs{sg}FIcDV(ZDvHRNRy<8kA@6d*0b=}$9jZQf5Ysu?S zik|&9m0{TW{1#nfA~A#ak{dasZeQ*he!bL*RTSD`p*vz(KWMW!=BzK2lj>_VwsI&J z2B6Zp>+9gP6454XSI|Y{Ok#No8qI5fVt-FV{hX1rk$}i>*jr=8{*H~&gT6gU`$0;J zW&I6mFOYQKL#vB`R&vDri&=4oCqHFIAWpGFUKw5fVFnbYFW|@ zi!Y3e(;mNT4L)<(F0;^>dQCzuZv77!J2YzxSX8elZ*QTa-wja{tLFBx9o*rHGvVdt zIG=d8IW+umxM>VC8lzsSfwQUeq+YPKn(zrUH__ZB03?tE`eKv#D%7si<2U6@PC^rid{Z0~V$$E`$^>EVqwx z3);i;lGBIc{rkBZJq@%qP}(lnt-V{tn3I)OYYG*6q?CBOl^y!rQ*ui_BfZ#hKZtgD z;7z?cYHO|bRHOzfv4zSKwM%;gp`ZLTCJm$f(-&+2P^>l7jOrL1c$q)ECB(Y)bvN-M zcqIx~`nOy?*SfkKNSy_{9vBe9w|lB78A4ZifS=7T_M+um3i%fNR+9-M4BGhwm;8aWmnf6mszl;vv zMFbxd)q5VvF;I>kC4Ukd4`3JIiH2by@w7E0D;v}^S$Bq(xAl%{BiMkAhXUHfTJVu* zGw$|i6>Gi1nQtq8;r_#l@QGhcfqEqtDHg9S$RfxoWI-6+9;Q6^sI}tB6E4mIuHS<+ zc>mGvO&-?(J5g`MNGDGxe)@fkp8Y!)iGx?wPG20_6H1rF@a|hUUs0{chy|g$zg(}k z8m}Q9^GHlEzYx!_+>_9(NUv~aV=Ib7-`JlLpef6?Xl}h_UhB)|!Ulwq+Qdn^3{=b) z&SquSl*v;7HeZ_IAO86|6cXPSXSS(bPa57i#crR-K^H<`4D$CPdBf4h%@BHj293b{b&n7n5g=C|^(FtwIDd zK=%{UeAcX6O3gIn)aROtH51_C#|L`VL!rgabKe|S@~$iCP?E>m&8_6S=Pc>nc0W4g zG{LE>?P4Sfg-tGQBZ@Y`S1P1j)_c#wH?G#7EVXY&dG$mJU zeMKauwYaIpAaH|+JQ~P>t`K~mV zE0jcM)+}9a-+OVuIzg}gc;;w^`0w?gd0smi6)KTwi@2l&*n_u+7h;lUi~WW%Fu1H; zuZXJnN=Q`yYTV8cJK0>Sz;Nu_^v#i~5D}MJ2BQ{2yQj_YseAqB{?GTjxoxU?IRtfm zl-yRcM+SZ?EP}DN)-t_8lhf8&}P z0M^rlTakk!E-m9rSue(y}nyG4b z8qbP7ar$s$%xv|W9z}KY(U$cB!+vFKT!AC;O_2iCX-e+SPK+IP1vBx~>4bqzTbz5= zsFWgQZxSg5c@zpUa?&_dECzAL42XCX zTzsv(Fu%A-(5b8ndY$iW67SIp_x7rDm%8l4bC{eRYWDB0x@Q*%+xCq+r??q`ycm~k z-_yyI1BqXU?A|1amgnxf>)tm+B(8k_k3^^F5npe7lTZY=+^b2C`Bv+1T#@5PKaWN5 za4=XLu@M(OV;vGO);klu_$4$fPaxY4NVKUx%!Wid` zW2YiTUniaX>rudX%YK1vu3Mj~LgjV#nv3K-zMpLg0_rY40WkEF2 z9cTiCWqY5C|MSfkTEQ|)6tQ&O<$aPmvzC^bW7(zY$ML^eIL6FF_9F;hrD#RNj_$Wd zB>((5k?cOBvxo3#ohPzIe1-S+zqR$bwb^xB!`@Jetvw# z^-=G>5^|A|pdpM~64%5?0Q2o%l_x+`#)O(9W^|CbT%(X1_Ha*HAx{mWLvCZ~DSt1J z%+4T4q7qT&$&Jz@Q9>x!JTA!#3{PF{I&*Q}FFX9M0$u(uo17aR9$#SKF1B?;{DhKA zFTrcOXKN{X*urNDik4SRw#nkfxi)LDeM#_jT>XL6k5X;CPJuBMDV;Sd2jVRLw8dfH z*9@ug{%gb?;26G8e1m=VbIvaeCU35&YjdF+^EZCMp)U#s%Qd}QY1E|9-*Nu>xZV8@ z-OxceGL@FU?LzG-r5~*NJ``3-(2%4gM90UQM7nO5EoZ?MfRxbHi^Qg@g#WlzN8Zr} z->u~&y7%n6rt)DmQE-f-8vUPK*s87>e(eK{whgCa;nc<5uv?4jfW1z5JLOpRTkdh@V_7-Pxw%Dx73HwvzUe zE@il}p7IzBS$d6BvCt~mB1yD}z>O$wl!aT!uT1Ye+<}bIh(33#b1hpwF*Uer(3T^G zt$piI{p-FIQALnk~PxpRI7sL_#+Eobs-mZx2XOL6vZHbc4gV!J@p@($T6#q`y#KLuz0? zrtIZpAWXaQWpn9pyQ**J$Ev9PHloS+!!9TpvYFM;RYTW1_i@|^83=x|ZqCt7W89X$ z0r2K$tRzY4RM|RQ5q8|Dcd*FRh#4#D3iba8J8N^wcaP1ia`o8TyqLRNd@m`aPvT^X z?1B&P+gHzmPOIAOZLT=5Cc>Jq>OWmce&R)`Nta-b^pfhQt*CTK?&J1mLQ7ZDV+z+` zx!=&P&K`m1_V?sabM&GorBRZbWxkWaPzZx#uI>Ie1&W))X&%j0ZOiAU;?^gk&Eovi zpF-LsdvC9bWyMN5-6@TY?Wa}miQ&PTU<0T2GonCP;nwe%E)j_g4QI0P$vSv;gDedL z&rysH!I>QUv8_n@;zD_Ut=B;}PfRa`@DpQvejH4W*48__U`<+`Th5v`-wrVliS8!- zurih~%Dd#0Zm09oQ$^kS!bsrpJx^+VQZpxdCKI zi=LUcH85brkZU5Q8F?YjD#rWMV<=bOavugmb##>k1jxVhJPB^F`g{DMO52&8<6Q6W!mrBRJ%hLJ zQInYW{DT}fKEs*c3SnW)cax9uuU)n{s%sm#>bmAp$_}D)G8WZ`j17?c)e{Ew=FLp( zV~Mef>iXH*hu6!_OV!#ND{xl^IzL_MI>}r0D5t3E2I#gv{ z8TuU-JV9=m5xtwSY+M|EPzH@3G78oDylep&vBF$;6@xuwA#{4ZIMi2G_$6f9bU zRS{p-r2-vh*Xk@LbILf0Z{q?CV{Qhc%OyCai1-yUki6!ZYpOCi-GsqPm#ae{?18T4 zQm$v`Oc4Ms0|sDCzyi=uQ;Wlb%{p;v8W zB?cp-UfMK&Gv0?0@h9Z$z-Wfu4m+SHavr?Tkg+IhwSRQO!eZGP^MyjVN@ChV*BR0< zT6?ls^%!XDF>IoAe6ST>6(#)?EoPsz`e9B1Y@Vuu#xE7)!1xW6D*sx{NO4L9Yp~JiFPFsqPFeBCy5r;d4GdKtwcGn0?%3xPpm& z`l{bud~~85BNV<4GSgxM|8Kl@Uef1ct5amYc}~E|`c&v;cpDdeBcv5Y&U25tulh+V z*J{_*{T=F~0(A4;y`G3mqX3es>a~QOuq8_tZhR?K_0EL?P6lFt5u2R#dR~E$mBJ7% zF)qf$FmJELb! zxBX#*(X#jMz^m5R`WKrSF)lmnK6KpFkMm4fpE!G6`~F|Um}FF}%M)17q3u2t2el$$ zg23&tV7-Q}=9f@x(6c7iIoIrhajG7sDeFUb{Hm%uWxz5Lq%S5z@sxFE@9Hm)7zSH9 zLbZF;lzO)H|iO0D*GoFI!47}Zmx>{`Eyce%Lp>(KG z4J7y6pHMGPKh{0?6Syp&tr`i40tFT+h^rJ+a}ajCXu*sa-XO|{tj_KY|Gtp~$kAy8fW zk%#-k50~{~(o5$J)xIwu&2XYh$n`#dboo>G5og>r`orG_K2IB9;G>&TMG!6$0|)y#O~Joi>Qo;NKJyxZqXFLkyK zkk-}J)wpj;I4^ZT81=`o=xYS=DLil6i0H)b16ST3GXYLcPBKAv>-WYCyuG}*EGMaZ z%8OCPU4(y7Xjf+!7YMWTnsFP4upnJ?bMyFWI|cBm-Nd)b*>_=$T2^A9mY$xlWnZ#^ zeRU=TkyOqguuXw~@1a5P5^Xkd8`Tz^c5Du~MYi;5xW6DhJ-y^@J4E~#7Z8PQCYI#D zi(Q=Vfb&<0#a?qXLvX0i#mU@5Za?n2-%XmT;+b4u-+r8(ojp4_aDbq+gN+%>-b8I< zV|(|nPdGVaZS5hbE=kZq;`Mg|6BCoSTj~Wh%xOUu?t-x+`~g;edjaT| zS*gc!87&p`^z;;vT?v9zX5c21b=#*0_3!p>-@Z+{3E`#qfv?RFz+8?7qM1Vy0|Emf zjypXck8_{(fSRhR&~6MzlUIiX^1-LHymHdgz84V*3k$#D6h*8JG*D7fhDH?_RH_(B z*An68bPDyoPu43-^_`u!A;_w^xp|$7e6wz+@3Y|S{rmTyAd@A$YhLMUXn@9b5X#GG zRKq7JX|g{GiM2J1ioNvudlLX$B8o+RIoYV?hQZJqo0 z=>6r1_SV)d!8>;hz@=&H;jCT=-1^+dXV!7=NX^i2y3|~Wn3x#usi&v6+?P`ROv&|m zpS;nVrD>9#NSK^`VxZXZuZ&P6YA{3IHdVLCkTpF0Y(O_E+vsCRNQmzMny=TAC6m@;@Z}r zYYL8Qb}zu>jE0@zBJa=>qgL^p$(hICwa<#6w2TZ+_8!vF(B#?UGgQkHKwVv31yUUA z_!a1Wh^|v5MX5)p==4`#3ZCRl!Fl1vEgT7q+ zok@(5eGM$Z{BN~JSf{moZq30#pXlc4QnYH>p5*Co8EYF&4UHD0&q3|n(aFgLHRl@c z*wfcL7Pgm?idfg2-emi;!BH?PEf6*?T0MWV5XMb29v>}jug+yvr5t5Y2*N%+6jV|9 z?$B_Q1<8nlf}%W5EoJ^a;ni86lXbSB`HEO}-BtW9e!O33+JYHS!CcQV0hh`$v}R7& zldRlHoHuSS1R_&XQr3l>YULy7pt~9%pXaujIB-eh5+0Al z#p*)9d$$L_?vi!9>Fd`&y15%^NJt*@!mi^dN-_%ca8T`p*!R%iaKpvLMe_I;<7Kr7 zioy&o&7I0c-Du&`{yG+0X3d!Ee5I>?pv-!h`Q?{1RRtS6PP-Xe`)9Q~!$DZ`gWAv& z{cQIjM)`1vJ_bv2J_Vo@d3y0SLOhG9H$_x;1pVK@|Ghr6(_&jWbms*Ek)eAwmB?%F z&2m@(o#OiaM3G8;rd-JQK|8DQt)oMs7MAS^DJa#}(a!S0%CLOjJ3VEtU;6GbGmHdF zFB>5PmU8Kd+26!*N?nbFxv}ReD}m3>LRe6d#SpT;GFx8@x!;oVZ*J}j6_Y8cK9@pH zobgYbq3!y4Fz4$#lg$5@nqZVc8%iA6hZ0YcA%s*XS8?*hKwF%1C6v8rpwGS#E8TE> zaw6D7S+N1Fw$`T_4{JRun!HrxulFbF4`#8D6zK>F(xm!ig)zLmyc(LCGt<*cVhi%y z1bgaWUJugu`lO=1%$cVgbWvc88OGm48RuR}_&&cQr8CK)VG_D>0Qo`vqNL}~pVN>L uSbhKB`PBcx=>Nfx|D8+!pH8U8a^x$#?v_j|<4Jevm;pHz*&^xJe*Xtr2}7U& literal 0 HcmV?d00001 diff --git a/img/online_delete-vs-ledger_history.png b/img/online_delete-vs-ledger_history.png new file mode 100644 index 0000000000000000000000000000000000000000..3268afa3fc12aac1588bbfe67ee889ba5e128cb3 GIT binary patch literal 8121 zcmbuEcT`hN*XU8iXcT=5F|*E-a?NcAiZ}35(R0}J7N?9h_uj~fD!4v zi1bJ^K?>?Zo@lBiOMq^9f^l|VL zw_2I~zD^RWj>gX$JAuz{>Gi#p^6XFFe|s-_KM5F`=s}$a4fwq90(t>>Ux%ER$(YXH z|185v{!jhy@*i=cf0X|z{?GD%-1OfyfRp%&T>9(sRY{k(Yum2v10NnfieX>&D|uUZ z`NI1lL+MAXh)L`r7J-oVUNIjs93b3){LFxVuW`jlVxI5R^r6;_FFQW7BBCwwpeu^B zAAWX1E9lmH`Wi>h1bkf#nIX@gx?20~ogEkq#wPYmyVyirSh)1(Pbf}W-eKh9la{dW zrIu%CQqH$`pD60+=tLZnrHr4#TdmS%eAjbgK_-m>c*9~7A@h3utinR0QZ?OuOmJ|H z@*Cw?4k5e2$@>+^B2TO8{>|BzjWVuVx7yIrBqStNjz3C(R_p8Qr>CbYxN^CuHCEDY zAz|VE$;rvy-rj2VRinW!3dXf{n3H~nWfr%_tL!ZuePytNZi7vY!ppx@;pPoK24pG5 z-0$rsT?C#^R)5U&!u1dn6ORB&ynp`yGn;H=;lI_&oK=Z`hGm9zISp* zeKH&DGCq%C2O3lf0$mz=gWc%%_V#-Aw>Ua=a6=|ZA$<&#kH5)yKIda}DKPa?L!?4Tor>{&~ zA&{|Jck5yYzJwvqY*3cBf$v@Gt*tHHntp`5t7(AkYmCzs>zXO#cmilCjEb!iKU9|J za|L~I{nVKvCHq%~I4H8~ql+@qXMbrhr{{CJ_{D(gUXvS>5*pJ;Hqm>wJy)g7Oy5bm zTC{{w;V;}Ykte2fiCS8ERv=qeRz}Duvu^j9YyYWX{ zU!>QEh5lc?nVF0H{QPP@agoKP%RfpHhE)uz%ThXszWPrxX(CarLk+$g%5PXzh1KeP zhlo@C4|*j;e7b=ZMA+YmgnFoXO>e+xVX4Jc{iyM*hO+ zm+U@qc0}vm%*86fq;a3t{o0Vp;Iud&6J4mS&1y938Koq+ppy$#fVJpQyCo8GOWfO~$9k?RqgMoTMgH z;vM_ds~=14C##F`ejasXh87kUy&2-zp7eQW2{Q3!UAYYSql#L81i|YPDK+4u@apFk zEW?}I+W|*wHQ!&GB3jo0&w9hB_yE;qj$#o1QFQJ{-uHB4;5IeqpD+0x>DYsiZ?@@9 znQxEX7%SsGA9=BJp)-*)=wKFwLPbYM8&}w}%!CUvqv?;2j{z^K0M>a?QIRLMPsprx z^JE`mw4i1}yoPV9tQ2ceU$<2?#wxi>s}}3!YZ55)0oM={^Xf?xX^-VhZBr-sQ?ZK0 zO^>{0>$7TTLwh7HG6|V|G!$CnyeyL-vMeiP@wrwbQ)0+E z5tgTgurM<#0>z&PQFcMeqRqi|EXsm z5ULmJJv=>~B?p0^OjHuvYGprOzPLJG;s5QTFQN+)!9=O z_Jy0N_UJs>ZHAKoHUAR&_Zpzti=8XX%)T3q`!n-IE-o$+5fMpOeL0C}1&MewA9462 zd$5k(DMIq62i9GY4K(Ln+*eXjQAy@EVCesWIXm6&&6I2o!1LR*PLmQ5QxdVa4iD>X zg*%U-W>bVrd3pY5Kz@(?e$18NY?Oz~5UwmPD z^Uj?TpS1}B_uhxs946IHEUjPvXyg(Fq?PYQ#(M8c@hjhf{B(`vU;m`m*4Ea0;~XsI zDX6G&AN(2Qfz;+oVQCk)&lu_n0|{V@^%qd8Am?vW{mlnjK_SPfBuo&u^iao^t2A$= zd~a+xfZS`qN+YW4Af2rc=n3->t(`5Fo_N-3>yX(pEbN*EJ1Vl4@*#(}$9|>@-;Hnz zO5*PDU0;6&x>xEHDV4q-+sp+Z^s?LJ#8GsD@atVtNFQzp8I!{G^A~M2l#Yxk1S$en zp(TccuS1A|G8|0%n2!FBNXkxY_>cDHd$3!J+IL8-V1guplCjXNNSP}^TqdR#<+7Vf z`j3?jW6Y*&&L(s(*i~8>DjknUy$+Vvv2`FDEe3Dqe(l#{94`s>bL{Lo`1IdV2tDb9OOW@BlqBa)%WSq9!( za`DG=y(e}`C>Nn7klx2&4%H&?=MVis>UET(n6HsM?xp5TRr*p~kT4{zAYh;(j2i0H zcgf9IrN8I&aP?3e^ZZAXV{9sEb4pS|7}Q{_j<%>v0{lz}GU>bC+1I!|{~?D_Hlykc zI@bEC+t=avGP<4*J&~#XUMO(AvPj-2EiMuB+*FD(#ZqZ7egQ7({f$ms=cWNm+X^sV z_jh9T*rl{FL+5wL&(T(ETFVcGA(chyHK6ix(+%b(3Nvfm1>2*g7HbpssBnqNVOqTz z{4llszE}P%ce}C@Vn`AnG@E;9d1vtK_(+KpQLr&CXQXD*7EN=ESU~y5E}z z?kzt8me_(4q96y8cJTD5Hm` zC$sISZa#Y}^N3VE_(STXI@Ra}CH(WM=_7qQH9>{3$MGPBL`>@El;?Ev=d9-Y_^CzM(4v!;FsZ-Lx(rE$6e-`-<(bq| zq}S>|pvq$`Zr;?&eDV&{3XXF^cRToYWcDz{(9Sju594K7XxaA|BO1or*o9abljG@7 zwwF&umPu;v!EG{=6aoy}n1ceFHm!W@!+37fxHdMaCb?VS3DH)l;qQy|#r#cnIc>+}25L;~dj0S~6fKvQZac*IG(W)=EKVL;#MGhE~s9P&1YMD7OZLVWY-w<57_YFc( zZaQ@Qo#QY0<4NDOtrqk{<~uxH{@`WJchbd_hTZbj<`qw@t#rG7t;tNw$+lKXI8EIQ zb~z66jJiGH(1+}Z2#DOw8AqHhkN0I(J2fYz9b1pZ7Yy5e6z(d4nrmtvBIM6x4X?x1cIjaLYj8;*SXGE-jNu{_&2kPRfZY zBX^Z_y;or;-&3u#;MwoZAgGB|RFRx1uf%4D;k?T`86kn7JJBkQP4=3P2ZhHEXP&gi zk|yttyOM_sh>@lEC66WFPpJ+vd^-4~@Jaq;I+xf@8(Kj=t*sle*G0fl3)&0HlVZZv zOW0`m>5-5Yaq@uhQ+J#8iWv=X* zVZj8wVQ^T?vDa}Pv3ehm^fsR0>UtSQ;5usu|mKB8uq?=(8AnD>-3DzPZ+ zi%>*;yCydQVnlgV%`@9bfO$D?-5yF|CV7s3oG+hm^K>VtIE_zHgi+O}z2wXTZes}( z>MFO#Q9EKHi^RLNYO-XoXd%o_`;xo{b2L*>Evp-&A!=a zL#Mx<$Bw0x-~8t}(*|KiE(lr570S`XwT3RmvG`5LjpmxR90qyDIhiy$tO-$$*1yRFLQ0;U?J^BsNTH!h zLqTTW))q@rWX9d?zJ=w^wi%PGcjdWXF!DkzjM|76EnWIjtz0RF+D=c+ncyZ2JreZoF^VBy+pLcYEh_|d!BMvyJC&Bbxo|Q5Q^}8#?sc`7P$408R3&_+{>NAK^=N1v9GWHTZ4%sR zp1DSt>&~S>>#pY1O;j##ICihd;qRa9u*64nB$iDeCSJs?bCq7+IemS0d-O_hJ=7SX zp2=la@g%3o4iR$ZCnpqMLVLBA(gq}cPQ{m@W zYT!7!8^Yd&y8UWxNoLe@9EpIZQy8BdgQZl`qCq+O0HL8+D|~% zt$5F{R;3eg;~~bvd*vR~oN{;QD}v_pEa`>Wh;c_k3R3=ObbPY(LD@YIS1D2mfESl( zYUpU+kq4*2c+NsO)pA$ji+LiX4iRk(a@CCshqO^D5w{@a%ZX{(QR9cD=7F7!`yd(N zIyUw~xTS|H9v6iTG8C?Bw@GSpTBp_oYDt!2{ zpJOC8@!GrPAr0z%j(eKSf-QBrL!!r^gM$L<{_&H?FMrR%nl}q*_^#`0LfaE4Cn6tdqm~}dmh2xs z&hX(peu{j)Rm#h21nV@{{|VJaZ=%?^2$pK z8+Y$fk!pa32fRxct$fg59X&&V?MdMrrKH7?7yRSw@fZ$aGyodj1Un6qWcG`7P`h#p zobuIO7m_^lQbx;br35*n*VRnXqB5zo{COPB7GVh^bHiX~9|*0WGKwsl-D+u5r(TJ^ z@8FC}|9N`i&*0>FhO-=wi-5b38ASnG;=$&mAd^>t?tbL%-MZ z=mgIAq>%9Yh(yK&Z#_Rc{XQdwKyN^Z74H7)==A#-6$;(zxcz%rI5aZu0g4Rhi;E0S zCwLL~A&%lxgrC<%UH|`xoF7E~YnePU^8c!v|67Dl@wX!X(1upXU){mpWpBTetFzN-bgmNU=WM zkcF=|%bWHZY9~4eA7m+-%F1D-r5i74KnLS?2u3C*=`HUY5JgNA3_$4`L5pttWGVkI z=#B9LXn{_S@^tXo2|!j^bZ4m#W)>VBi+2E+t z&DZh}$@ecwX@X`$nMb55HSY5AegvR4z$`ofnz6qx%+D_>#&CFeIAkIDY~Z~~l>@;3 z&}j7Ncw*Ew2`9)20iUaY0EnMYt#LV#_PQ3t5e)zhV*pYk4fBx5`;GU3ogLFuC|un0 z*JB!*SQP56dtO{zTmXKh*u24TQp|qn=3vI==H|$VegX5)-mkkDOct4+AGPI#X)-^$Cn;&F_ z@L}j|d8}+cp8Jv6{8xbjC||(afQEu4I|0IQb)&BTMdLHmu(Y(RE0=1kp{mpWLfWuG z0BM)IEp{cdiaQipHd~ll{!O%PBOFvNUFS4XjnWX=^qfV(%XpKn~M@&$qc*9GPkx?titpq zLi_SSh8$0g^W?8zzw+J)AV4q^2lQB(b(O|O(@}{bY4-`a0FEb7BSXx%F`!ByIhco0 z5FD%LD#>`_RMAvwT$)5DSX)S}?(k+LHLz3%m}h>2PaOyVjHPM-7c1q00UtJ|0B%)m zJMOZAX>LB(Tzt#55uakb*L3)c&AvAm)a6lX(Wo4KRmYZ^id@YU)8Om6Y;$;U;575u zn0RnNTj|lGZUC;A5pHaMS(<5Vc-P3Mm$#I;Kr5vkFZjZfJZ1WV30`lz>1aLib*Z4R zak*0EYFRr6z#=eBC#0OuH%(=q5U*?z_l@Rc|Gi(uI&Cnqev*)oCg9&AS|<_y^+&BU0u??>r-DtDKWukFz3V(66b9>A)($qo_zp! zpI;0~91;jCnG6NVd=S8`_osakuV25$Cq9VJyMZ1U7!a{;W0wyyQ&JNheXnq7yyDw0 zp8eOHK@bi`78Vu&z>9ufa4|ylB>dL)Xva|gd~F&w_&AUAGinobhV!*Jfnaockx2`J zWfJs`)kARU?^ZCL*f%r|gOvyUM~R@pXvAW!rc-$ngr5)$cQT<{>ym1-4 z;sXC0wXe`|iW0dgqxnTRy_ptx{aIHqIV(fCQG4fhGXhfM$5$$)wsYSi-mpmq{R|;R z_EJ$&<~;t7vgV(}`rk^p|AVIgo~!?ZH18&KDu02~1>+hTVFBb_B2}=aQi&oI{=We0 C?MYbx literal 0 HcmV?d00001 From a31fd23d6a8e299d7ac8bda1d580fd4a92a7eaaf Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Tue, 11 Dec 2018 20:03:29 -0800 Subject: [PATCH 02/14] Advisory delete, full history tutorials. History disk space estimates. --- .../the-rippled-server/online-deletion.md | 39 +++++--- .../configuration/configure-full-history.md | 98 +++++++++++++++++++ .../configure-online-deletion.md | 96 ++++++++++++++++++ dactyl-config.yml | 20 ++++ 4 files changed, 241 insertions(+), 12 deletions(-) create mode 100644 content/tutorials/manage-the-rippled-server/configuration/configure-full-history.md create mode 100644 content/tutorials/manage-the-rippled-server/configuration/configure-online-deletion.md diff --git a/content/concepts/the-rippled-server/online-deletion.md b/content/concepts/the-rippled-server/online-deletion.md index 41afbdcbd8..c19d81c226 100644 --- a/content/concepts/the-rippled-server/online-deletion.md +++ b/content/concepts/the-rippled-server/online-deletion.md @@ -57,6 +57,14 @@ With online deletion enabled and running automatically (that is, with advisory d When online deletion runs, it does not reduce the size of the database files on disk; it only makes space within those files available to be reused for new data. To actually reduce the disk space allocated to the database files, you must restart `rippled`. +### Interrupting Online Deletion + +Online deletion automatically stops if the [server state](rippled-server-states.html) becomes less than `full`. If this happens, the server writes a log message with the prefix `SHAMapStore::WRN`. The server attempts to start online deletion again after the next validated ledger version after becoming fully synced. + +If you stop the server or it crashes while online deletion is running, online deletion resumes after the server is restarted and the server becomes fully synced. + +To temporarily disable online deletion, you can use the [can_delete method][] with an argument of `never`. This change persists until the server restarts or you re-enable online deletion by calling [can_delete][can_delete method] again. For more information on controlling when online deletion happens, see [Advisory Deletion](#advisory-deletion). + ## Configuration @@ -95,16 +103,16 @@ The following settings relate to online deletion: Online deletion settings must be configured in terms of the number of ledger versions, not in terms of real time. You can estimate the amount of time represented by a number of ledger versions using the rate of ledger progress. Ledger versions typically close 3-4 seconds apart, with the overall rate of ledger closures ranging from about 20,000 to 27,000 per day throughout the years 2017-2018. There are hard limits of at least 2 seconds and no more than 20 seconds between ledgers as long as consensus is operating properly. -The following table summarizes some +The following table approximates the requirements for different amounts of history: -| Real Time Amount | Approximate Number of Ledgers | -|:-----------------|:------------------------------| -| 1 day | 25,000 | -| 14 days | 350,000 | -| 30 days | 750,000 | -| 90 days | 2,250,000 | -| 1 year | 10,000,000 | -| 2 years | 20,000,000 | +| Real Time Amount | Number of Ledger Versions | Disk Space Required (RocksDB) | Disk Space Required (NuDB) | +|:-----------------|:--------------------------|:------------------------------|:--| +| 1 day | 25,000 | 8 GB | 12 GB | +| 14 days | 350,000 | 112 GB | 168 GB | +| 30 days | 750,000 | 240 GB | 360 GB | +| 90 days | 2,250,000 | 720 GB | 1 TB | +| 1 year | 10,000,000 | 3 TB | 4.5 TB | +| 2 years | 20,000,000 | 6 TB | 9 TB | These numbers are estimates and depend on several factors: @@ -112,15 +120,22 @@ These numbers are estimates and depend on several factors: - The volume of transactions in the network. A high rate of transactions can increase the work needed to close a new ledger version, reducing the number of separate ledgers closed. An extremely low rate of transactions can also cause ledgers to close further apart (because there is nothing to be done). - Changes and optimizations to the XRP Ledger's consensus algorithm can affect the speed of consensus and the rate of disagreement either positively or negatively. -### Advisory Delete +### Advisory Deletion -By default, online deletion happens automatically and periodically. If the advisory delete setting is enabled, online deletion only happens when an administrator triggers it using the [can_delete method][]. (You can trigger this command with a `cron` job to run automatic deletion based on actual time instead of the number of ledger versions closed.) +By default, online deletion happens automatically and periodically. If the `advisory_delete` setting is enabled, online deletion only happens when an administrator triggers it using the [can_delete method][]. You can run this command with a scheduled job to trigger automatic deletion based on clock time instead of the number of ledger versions closed. If your server is heavily used, the extra load from online deletion can cause your server to fall behind and temporarily de-sync from the consensus network. If this is the case, you can use advisory delete and schedule online deletion to happen only during off-peak times. You can also use advisory delete for other reasons, such as if you want to manually confirm that the transaction data is backed up to a separate server before deleting it. -The `can_delete` API method can enable advisory delete (with the value `never`) or disable advisory delete (with the value `always`). These setting changes persist until you restart the `rippled` server, at which point the settings in the config file override them. +The `can_delete` API method can enable advisory deletion (with the value `never`) or disable advisory delete (with the value `always`). These setting changes persist until you restart the `rippled` server, at which point the settings in the config file override them. + + +## See Also + +- [can_delete method][] API reference documentation +- [Configure Online Deletion with Advisory Delete](configure-online-deletion.html) + diff --git a/content/tutorials/manage-the-rippled-server/configuration/configure-full-history.md b/content/tutorials/manage-the-rippled-server/configuration/configure-full-history.md new file mode 100644 index 0000000000..a81ba2e7a1 --- /dev/null +++ b/content/tutorials/manage-the-rippled-server/configuration/configure-full-history.md @@ -0,0 +1,98 @@ +# Configure Full History + +In its default configuration, the `rippled` server automatically deletes outdated history of XRP Ledger state and transactions as new ledger versions become available. This is sufficient for most servers, which do not need older history to know the current state and process transactions. However, it can be useful for the network if some servers provide as much history of the XRP Ledger as possible. + +## Warnings + +Storing full history is expensive. As of 2018-12-11, the full history of the XRP Ledger occupies approximately **9 terabytes** of disk space, which must be entirely stored on fast solid state disk drives for proper server performance. Such a large amount of solid state storage is not cheap, and the total amount of history you must store increases by approximately 12 GB per day. + +Acquiring full history from the peer-to-peer network takes a long time (several months) and requires that your server has sufficient system and network resources to acquire older history while keeping up with new ledger progress. To get a faster start on acquiring ledger history, you may want to find another server operator who has a large amount of history already downloaded, who can give you a database dump or at least allow your server to explicitly peer with theirs for a long time to acquire history. The server can load ledger history from a file and verify the integrity of the historical ledgers it imports. + +You do not need a full history server to participate in the network, validate transactions, or know the current state of the network. Full history is only useful for knowing the outcome of transactions that occurred in the past, or the state of the ledger at a given time in the past. To get such information, you must rely on other servers having the history you need. + +If you want to contribute to storing the history of the XRP Ledger network without storing the full history, you can [configure history sharding](configure-history-sharding.html) to store randomly-selected chunks of ledger history instead. + +## Configuration Steps + +To configure your server to acquire and store full history, complete the following steps: + +1. Stop the `rippled` server if it is running. + + $ sudo systemctl stop rippled + +0. Remove (or comment out) the `online_delete` and `advisory_delete` settings from the `[node_db]` stanza of your server's config file: + + [node_db] + type=RocksDB + path=/var/lib/rippled/db/rocksdb + open_files=2000 + filter_bits=12 + cache_mb=256 + file_size_mb=8 + file_size_mult=2 + compression=1 + #online_delete=2000 + #advisory_delete=0 + + You can store full history with either RocksDB or NuDB as the key-value store behind the ledger store. NuDB requires less RAM but RocksDB requires less disk space. For more information, see [Capacity Planning](capacity-planning.html). + + If you are using RocksDB, add `compression=1` to enable Snappy compression, which reduces the disk space necessary to store full history (at a cost of greater CPU usage when reading or writing history). + + {% include '_snippets/conf-file-location.md' %} + +0. Set the `[ledger_history]` stanza of your server's config file to `full`: + + [ledger_history] + full + +0. Set the `[ips_fixed]` stanza of your server's config file to explicitly peer with at least one server that has full history available. + + [ips_fixed] + 169.55.164.20 + 50.22.123.215 + + Your server can only download historical data from the peer-to-peer network if one its direct peers has the data available. The easiest way to ensure you can download full history is to peer with a server that already has full history. + + **Tip:** Ripple makes a pool of full history servers publicly available. You can resolve the domain `s2.ripple.com` a few times to get the IP addresses of these servers. Ripple offers these servers as a public service, so be aware that their availability to peer with other servers is limited and you may be blocked if you abuse them. + +0. If you have a database dump from another full-history server to use as a basis, set the `[import_db]` stanza of your server's config file to point to the data to be imported. (Otherwise, skip this step.) + + [import_db] + type=RocksDB + path=/tmp/full_history_dump/ + +0. Remove your server's existing database files, if you have any from previously running `rippled`. + + After disabling online deletion, the server ignores any data that was downloaded while online deletion was enabled, so you may as well clear up the disk space: + + rm -r /var/lib/rippled/db/rocksdb/* + rm /var/lib/rippled/db/*.db + rm /var/lib/rippled/db/*.sqlite + +0. Start the `rippled` server, importing the database dump if you have one available: + + If you have a database dump to load configured in `[import_db]`, start the server explicitly and include the `--import` [commandline option](commandline-usage.html#daemon-mode-options): + + $ /opt/ripple/bin/rippled --conf /etc/opt/ripple/rippled.cfg --import + + Importing a large database dump may take several minutes or even hours. During this time, the server is not fully started and synced with the network. Watch the server logs to see the status of the import. + + If you are not importing a database dump, start the server normally: + + $ sudo systemctl start rippled + +0. If you added an `[import_db]` stanza to your server's config file, remove it after the import completes. + + Otherwise, your server may try to import the same data again the next time it is restarted. + +0. Monitor your server's available history with the [server_info method][]. + + The range of available ledgers reported in the `complete_ledgers` field should increase over time. + + The earliest available ledger version in the production XRP Ledger's history is ledger index **32570**. The first two weeks or so of ledger history was lost due to a bug in the server at the time. Test nets and other chains generally have history going back to ledger index **1**. + + + +{% include '_snippets/rippled-api-links.md' %} +{% include '_snippets/tx-type-links.md' %} +{% include '_snippets/rippled_versions.md' %} diff --git a/content/tutorials/manage-the-rippled-server/configuration/configure-online-deletion.md b/content/tutorials/manage-the-rippled-server/configuration/configure-online-deletion.md new file mode 100644 index 0000000000..981cd4c27a --- /dev/null +++ b/content/tutorials/manage-the-rippled-server/configuration/configure-online-deletion.md @@ -0,0 +1,96 @@ +# Configure Online Deletion with Advisory Delete + +In its default configuration, the `rippled` server automatically deletes outdated history of XRP Ledger state and transactions as new ledger versions become available. If your server uses most of its hardware resources during peak hours, you can configure the server to delete ledgers only when prompted by a command scheduled to run during off-peak hours to reduce the chances of online deletion + +## Prerequisites + +This tutorial assumes your server meets the following prerequisites: + +- You are on a supported operating system: Ubuntu Linux, Red Hat Enterprise Linux (RHEL), or CentOS. + +- The `rippled` server is already [installed](install-rippled.html) and [online deletion](online-deletion.html) is enabled. + + The default configuration file enables online deletion after + +- A `cron` daemon is installed and running. + + Ubuntu Linux runs a `cron` daemon by default. + + On RHEL or CentOS, you can install the `cronie` package: + + $ sudo yum install cronie + +- Your server has enough disk space to store your chosen amount of history in its ledger store. + + See [Capacity Planning](capacity-planning.html) for details of how much storage is required for different configuration. With advisory deletion enabled, the maximum history a server may accumulate before deletion is equal to the number of ledger versions configured in the `online_delete` setting **plus** the amount of time between online deletion prompts. + +- You know which hours are least busy for your server. + +## Configuration Steps + +To configure advisory deletion with a daily schedule, perform the following steps: + +1. Enable `advisory_delete` in the `[node_db]` stanza of your `rippled`'s config file. + + [node_db] + # Other settings unchanged ... + online_delete=2000 + advisory_delete=1 + + - Set `advisory_delete` to `1` to require approval before running online deletion. (Set it to `0` to run online deletion automatically as new ledger versions become available.) + - Set `online_delete` to the minimum number of ledger versions to keep after running online deletion. The server accumulates more history than this until online deletion runs. + + {% include '_snippets/conf-file-location.md' %} + +2. Test running the [can_delete method][] to prompt the server to run online deletion. + + You can use the [`rippled` commandline interface](get-started-with-the-rippled-api.html#commandline) to run this command. For example: + + $ rippled --conf=/etc/opt/ripple/rippled.cfg can_delete now + + The response indicates the maximum ledger index that may the server may delete from its ledger store. For example, the following message indicates that ledger versions up to and including ledger index 43633667 are to be deleted: + + { + "result": { + "can_delete": 43633667, + "status": "success" + } + } + +3. Configure your `cron` daemon to run the `can_delete` method you tested in the previous step at a scheduled time. + + Edit your `cron` configuration: + + $ crontab -e + + The following example sets the server to run deletion at 1:05 AM server time daily: + + 5 1 * * * rippled --conf /etc/opt/ripple/rippled.cfg can_delete now + + Be sure that you schedule the command to run based on the time zone your server's clock is configured to use. Ripple recommends running all production servers with the UTC time zone. + + **Tip:** You do not need to schedule a `cron` job to run online deletion if you have `advisory_delete` disabled. In that case, `rippled` runs online deletion automatically when the server's oldest available ledger is twice as old as the number of ledgers to keep after deletion. + +4. Start (or restart) the `rippled` service. + + $ sudo systemctl restart rippled + +5. Periodically check your server's `complete_ledgers` range using the [server_info method][] to confirm that ledgers are being deleted as scheduled. + + The lowest ledger index in `complete_ledgers` should increase after online deletion. + + Deletion may take several minutes to complete when it runs, depending on how busy your server is and how much history you delete at a time. + +## Troubleshooting + +If online deletion does not seem to be running after configuring it, try the following: + +- Check that the user who configured the `cron` job has permissions to run the `rippled` server as a commandline client. +- Check the syntax of your cron job and the time when it is supposed to run. +- Check that the `rippled` executable is available at the path specified in your `cron` configuration. If necessary, specify the absolute path to the executable, such as `/opt/ripple/bin/rippled`. +- Check your `rippled` logs for messages that begin with `SHAMapStore::WRN`. This can indicate that [online deletion is being interrupted](online-deletion.html#interrupting-online-deletion) because your server fell out of sync with the network. + + +{% include '_snippets/rippled-api-links.md' %} +{% include '_snippets/tx-type-links.md' %} +{% include '_snippets/rippled_versions.md' %} diff --git a/dactyl-config.yml b/dactyl-config.yml index cdf9f06032..f798caf7e1 100644 --- a/dactyl-config.yml +++ b/dactyl-config.yml @@ -1040,6 +1040,26 @@ pages: targets: - local + - md: tutorials/manage-the-rippled-server/configuration/configure-online-deletion.md + html: configure-online-deletion.html + funnel: Docs + doc_type: Tutorials + category: Manage the rippled Server + subcategory: Configuration + blurb: Use advisory deletion to delete older ledger history on a schedule rather than as new history becomes available. + targets: + - local + + - md: tutorials/manage-the-rippled-server/configuration/configure-full-history.md + html: configure-full-history.html + funnel: Docs + doc_type: Tutorials + category: Manage the rippled Server + subcategory: Configuration + blurb: Full history servers provide a record of every transaction ever to occur in the XRP Ledger, although they are expensive to run. + targets: + - local + - md: tutorials/manage-the-rippled-server/configuration/enable-public-signing.md html: enable-public-signing.html funnel: Docs From 00cc82038485b1f354e3777b9e4d87652e3b7c1e Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Thu, 13 Dec 2018 15:23:19 -0800 Subject: [PATCH 03/14] Online deletion: simpler tutorial --- .../configure-advisory-deletion.md | 96 ++++++++++++++++++ .../configure-online-deletion.md | 97 ++++++++----------- dactyl-config.yml | 30 ++++-- 3 files changed, 154 insertions(+), 69 deletions(-) create mode 100644 content/tutorials/manage-the-rippled-server/configuration/configure-advisory-deletion.md diff --git a/content/tutorials/manage-the-rippled-server/configuration/configure-advisory-deletion.md b/content/tutorials/manage-the-rippled-server/configuration/configure-advisory-deletion.md new file mode 100644 index 0000000000..c94544627d --- /dev/null +++ b/content/tutorials/manage-the-rippled-server/configuration/configure-advisory-deletion.md @@ -0,0 +1,96 @@ +# Configure Advisory Deletion + +In its default configuration, the `rippled` server automatically deletes outdated history of XRP Ledger state and transactions as new ledger versions become available. If your server uses most of its hardware resources during peak hours, you can configure the server to delete ledgers only when prompted by a command scheduled to run during off-peak hours to reduce the chances of online deletion + +## Prerequisites + +This tutorial assumes your server meets the following prerequisites: + +- You are on a supported operating system: Ubuntu Linux, Red Hat Enterprise Linux (RHEL), or CentOS. + +- The `rippled` server is already [installed](install-rippled.html) and [online deletion](online-deletion.html) is enabled. + + The default configuration file enables online deletion after 2000 ledger versions. + +- A `cron` daemon is installed and running. + + Ubuntu Linux runs a `cron` daemon by default. + + On RHEL or CentOS, you can install the `cronie` package: + + $ sudo yum install cronie + +- Your server has enough disk space to store your chosen amount of history in its ledger store. + + See [Capacity Planning](capacity-planning.html) for details of how much storage is required for different configuration. With advisory deletion enabled, the maximum history a server may accumulate before deletion is equal to the number of ledger versions configured in the `online_delete` setting **plus** the amount of time between online deletion prompts. + +- You know which hours are least busy for your server. + +## Configuration Steps + +To configure advisory deletion with a daily schedule, perform the following steps: + +1. Enable `advisory_delete` in the `[node_db]` stanza of your `rippled`'s config file. + + [node_db] + # Other settings unchanged ... + online_delete=2000 + advisory_delete=1 + + - Set `advisory_delete` to `1` to require approval before running online deletion. (Set it to `0` to run online deletion automatically as new ledger versions become available.) + - Set `online_delete` to the minimum number of ledger versions to keep after running online deletion. The server accumulates more history than this until online deletion runs. + + {% include '_snippets/conf-file-location.md' %} + +2. Test running the [can_delete method][] to prompt the server to run online deletion. + + You can use the [`rippled` commandline interface](get-started-with-the-rippled-api.html#commandline) to run this command. For example: + + $ rippled --conf=/etc/opt/ripple/rippled.cfg can_delete now + + The response indicates the maximum ledger index that may the server may delete from its ledger store. For example, the following message indicates that ledger versions up to and including ledger index 43633667 are to be deleted: + + { + "result": { + "can_delete": 43633667, + "status": "success" + } + } + +3. Configure your `cron` daemon to run the `can_delete` method you tested in the previous step at a scheduled time. + + Edit your `cron` configuration: + + $ crontab -e + + The following example sets the server to run deletion at 1:05 AM server time daily: + + 5 1 * * * rippled --conf /etc/opt/ripple/rippled.cfg can_delete now + + Be sure that you schedule the command to run based on the time zone your server's clock is configured to use. Ripple recommends running all production servers with the UTC time zone. + + **Tip:** You do not need to schedule a `cron` job to run online deletion if you have `advisory_delete` disabled. In that case, `rippled` runs online deletion automatically when the server's oldest available ledger is twice as old as the number of ledgers to keep after deletion. + +4. Start (or restart) the `rippled` service. + + $ sudo systemctl restart rippled + +5. Periodically check your server's `complete_ledgers` range using the [server_info method][] to confirm that ledgers are being deleted as scheduled. + + The lowest ledger index in `complete_ledgers` should increase after online deletion. + + Deletion may take several minutes to complete when it runs, depending on how busy your server is and how much history you delete at a time. + +## Troubleshooting + +If online deletion does not seem to be running after configuring it, try the following: + +- Check that the user who configured the `cron` job has permissions to run the `rippled` server as a commandline client. +- Check the syntax of your cron job and the time when it is supposed to run. +- Check that the `rippled` executable is available at the path specified in your `cron` configuration. If necessary, specify the absolute path to the executable, such as `/opt/ripple/bin/rippled`. +- Check your `rippled` logs for messages that begin with `SHAMapStore::WRN`. This can indicate that [online deletion is being interrupted](online-deletion.html#interrupting-online-deletion) because your server fell out of sync with the network. + + +{% include '_snippets/rippled-api-links.md' %} +{% include '_snippets/tx-type-links.md' %} +{% include '_snippets/rippled_versions.md' %} diff --git a/content/tutorials/manage-the-rippled-server/configuration/configure-online-deletion.md b/content/tutorials/manage-the-rippled-server/configuration/configure-online-deletion.md index 981cd4c27a..ff31c6c947 100644 --- a/content/tutorials/manage-the-rippled-server/configuration/configure-online-deletion.md +++ b/content/tutorials/manage-the-rippled-server/configuration/configure-online-deletion.md @@ -1,6 +1,6 @@ -# Configure Online Deletion with Advisory Delete +# Configure Online Deletion -In its default configuration, the `rippled` server automatically deletes outdated history of XRP Ledger state and transactions as new ledger versions become available. If your server uses most of its hardware resources during peak hours, you can configure the server to delete ledgers only when prompted by a command scheduled to run during off-peak hours to reduce the chances of online deletion +In its default configuration, the `rippled` server [deletes history](online-deletion.html) older than the most recent 2000 ledger versions, keeping approximately 15 minutes of ledger history (based on the current rate between ledgers). This page describes how to configure the amount of history your `rippled` server stores before deleting. ## Prerequisites @@ -10,85 +10,64 @@ This tutorial assumes your server meets the following prerequisites: - The `rippled` server is already [installed](install-rippled.html) and [online deletion](online-deletion.html) is enabled. - The default configuration file enables online deletion after + If you followed the installation instructions for a recommended platform, online deletion is enabled by default. -- A `cron` daemon is installed and running. +- Your server has [enough disk space](capacity-planning.html) to store your chosen amount of history in its ledger store. - Ubuntu Linux runs a `cron` daemon by default. - - On RHEL or CentOS, you can install the `cronie` package: - - $ sudo yum install cronie - -- Your server has enough disk space to store your chosen amount of history in its ledger store. - - See [Capacity Planning](capacity-planning.html) for details of how much storage is required for different configuration. With advisory deletion enabled, the maximum history a server may accumulate before deletion is equal to the number of ledger versions configured in the `online_delete` setting **plus** the amount of time between online deletion prompts. - -- You know which hours are least busy for your server. ## Configuration Steps -To configure advisory deletion with a daily schedule, perform the following steps: +To change the amount of history your server stores, perform the following steps: -1. Enable `advisory_delete` in the `[node_db]` stanza of your `rippled`'s config file. +1. Decide how many ledger versions' worth of history to store. + + New ledger versions are usually validated 3 to 4 seconds apart, so the number of ledger versions corresponds roughly to the amount of time you want to store. See [Capacity Planning](capacity-planning.html) for details of how much storage is required for different configuration. + + Online deletion is based on how many ledger versions to keep _after_ deleting history, so you should have enough disk space to store twice as many ledgers as you set it to keep. + +0. In your `rippled`'s config file, edit the `online_delete` field of the `[node_db]` stanza. [node_db] # Other settings unchanged ... online_delete=2000 - advisory_delete=1 + advisory_delete=0 - - Set `advisory_delete` to `1` to require approval before running online deletion. (Set it to `0` to run online deletion automatically as new ledger versions become available.) - - Set `online_delete` to the minimum number of ledger versions to keep after running online deletion. The server accumulates more history than this until online deletion runs. + Set `online_delete` to the minimum number of ledger versions to keep after running online deletion. With automatic deletion (the default), the server typically runs deletion when it has accumulated about twice this many ledger versions. {% include '_snippets/conf-file-location.md' %} -2. Test running the [can_delete method][] to prompt the server to run online deletion. - - You can use the [`rippled` commandline interface](get-started-with-the-rippled-api.html#commandline) to run this command. For example: - - $ rippled --conf=/etc/opt/ripple/rippled.cfg can_delete now - - The response indicates the maximum ledger index that may the server may delete from its ledger store. For example, the following message indicates that ledger versions up to and including ledger index 43633667 are to be deleted: - - { - "result": { - "can_delete": 43633667, - "status": "success" - } - } - -3. Configure your `cron` daemon to run the `can_delete` method you tested in the previous step at a scheduled time. - - Edit your `cron` configuration: - - $ crontab -e - - The following example sets the server to run deletion at 1:05 AM server time daily: - - 5 1 * * * rippled --conf /etc/opt/ripple/rippled.cfg can_delete now - - Be sure that you schedule the command to run based on the time zone your server's clock is configured to use. Ripple recommends running all production servers with the UTC time zone. - - **Tip:** You do not need to schedule a `cron` job to run online deletion if you have `advisory_delete` disabled. In that case, `rippled` runs online deletion automatically when the server's oldest available ledger is twice as old as the number of ledgers to keep after deletion. - -4. Start (or restart) the `rippled` service. +0. Start (or restart) the `rippled` service. $ sudo systemctl restart rippled -5. Periodically check your server's `complete_ledgers` range using the [server_info method][] to confirm that ledgers are being deleted as scheduled. +0. Wait for your server to sync to the network. - The lowest ledger index in `complete_ledgers` should increase after online deletion. + Depending on your network and system capabilities and how long your server was offline, it may take between 5 and 15 minutes to fully sync. - Deletion may take several minutes to complete when it runs, depending on how busy your server is and how much history you delete at a time. + When your server is synced with the network, the [server_info method][] reports a `server_state` value of `"full"`, `"proposing"`, or `"validating"`. -## Troubleshooting +0. Periodically check your server's `complete_ledgers` range using the [server_info method][] to confirm that ledgers are being deleted. -If online deletion does not seem to be running after configuring it, try the following: + After online deletion runs, the `complete_ledgers` range reflects that older ledgers are no longer available. As your server accumulates history, the total number of ledgers available should slowly increase to twice the `online_delete` value you configured, then decrease when online deletion runs. + +0. Monitor your `rippled` logs for messages that begin with `SHAMapStore::WRN`. This can indicate that [online deletion is being interrupted](online-deletion.html#interrupting-online-deletion) because your server fell out of sync with the network. + + If this happens regularly, your server may not have sufficient specifications to keep up with the ledger while running online deletion. Check that other services on the same hardware (such as scheduled backups or security scans) aren't competing with the `rippled` server for resources. You may want to try any of the following: + + - Increase your system specs. See [System Requirements](system-requirements.html) for recommendations. + - Change your configuration to store less history. (Step 2 of this tutorial) + - Change your server's [`node_size` parameter](capacity-planning.html). + - Use [NuDB instead of RocksDB](capacity-planning.html) for the ledger store. + - [Schedule online deletion using Advisory Deletion](configure-advisory-deletion.html). + + +## See Also + +- [Online Deletion](online-deletion.html) +- [Configure Advisory Deletion](configure-advisory-deletion.html) +- [Configure History Sharding](configure-history-sharding.html) +- [Configure Full History](configure-full-history.html) -- Check that the user who configured the `cron` job has permissions to run the `rippled` server as a commandline client. -- Check the syntax of your cron job and the time when it is supposed to run. -- Check that the `rippled` executable is available at the path specified in your `cron` configuration. If necessary, specify the absolute path to the executable, such as `/opt/ripple/bin/rippled`. -- Check your `rippled` logs for messages that begin with `SHAMapStore::WRN`. This can indicate that [online deletion is being interrupted](online-deletion.html#interrupting-online-deletion) because your server fell out of sync with the network. {% include '_snippets/rippled-api-links.md' %} diff --git a/dactyl-config.yml b/dactyl-config.yml index f798caf7e1..6e6320b737 100644 --- a/dactyl-config.yml +++ b/dactyl-config.yml @@ -1030,6 +1030,26 @@ pages: targets: - local + - md: tutorials/manage-the-rippled-server/configuration/configure-online-deletion.md + html: configure-online-deletion.html + funnel: Docs + doc_type: Tutorials + category: Manage the rippled Server + subcategory: Configuration + blurb: Configure how far back your server should store transaction history. + targets: + - local + + - md: tutorials/manage-the-rippled-server/configuration/configure-advisory-deletion.md + html: configure-advisory-deletion.html + funnel: Docs + doc_type: Tutorials + category: Manage the rippled Server + subcategory: Configuration + blurb: Use advisory deletion to delete older ledger history on a schedule rather than as new history becomes available. + targets: + - local + - md: tutorials/manage-the-rippled-server/configuration/configure-history-sharding.md html: configure-history-sharding.html funnel: Docs @@ -1040,16 +1060,6 @@ pages: targets: - local - - md: tutorials/manage-the-rippled-server/configuration/configure-online-deletion.md - html: configure-online-deletion.html - funnel: Docs - doc_type: Tutorials - category: Manage the rippled Server - subcategory: Configuration - blurb: Use advisory deletion to delete older ledger history on a schedule rather than as new history becomes available. - targets: - - local - - md: tutorials/manage-the-rippled-server/configuration/configure-full-history.md html: configure-full-history.html funnel: Docs From 0bec4019632667cf864cef7018f27dd6786bf2f3 Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Thu, 13 Dec 2018 15:54:22 -0800 Subject: [PATCH 04/14] Move storage disk requirements table to capacity planning doc --- .../the-rippled-server/online-deletion.md | 22 +-------------- .../installation/capacity-planning.md | 28 +++++++++++++++---- 2 files changed, 23 insertions(+), 27 deletions(-) diff --git a/content/concepts/the-rippled-server/online-deletion.md b/content/concepts/the-rippled-server/online-deletion.md index c19d81c226..abd6406123 100644 --- a/content/concepts/the-rippled-server/online-deletion.md +++ b/content/concepts/the-rippled-server/online-deletion.md @@ -98,27 +98,7 @@ The following settings relate to online deletion: ![Ledger versions older than fetch_depth are not served to peers](img/fetch_depth.png) - -### Relation to Real Time - -Online deletion settings must be configured in terms of the number of ledger versions, not in terms of real time. You can estimate the amount of time represented by a number of ledger versions using the rate of ledger progress. Ledger versions typically close 3-4 seconds apart, with the overall rate of ledger closures ranging from about 20,000 to 27,000 per day throughout the years 2017-2018. There are hard limits of at least 2 seconds and no more than 20 seconds between ledgers as long as consensus is operating properly. - -The following table approximates the requirements for different amounts of history: - -| Real Time Amount | Number of Ledger Versions | Disk Space Required (RocksDB) | Disk Space Required (NuDB) | -|:-----------------|:--------------------------|:------------------------------|:--| -| 1 day | 25,000 | 8 GB | 12 GB | -| 14 days | 350,000 | 112 GB | 168 GB | -| 30 days | 750,000 | 240 GB | 360 GB | -| 90 days | 2,250,000 | 720 GB | 1 TB | -| 1 year | 10,000,000 | 3 TB | 4.5 TB | -| 2 years | 20,000,000 | 6 TB | 9 TB | - -These numbers are estimates and depend on several factors: - -- The rate of agreement among trusted validators. More disagreement slows the rate of new ledger creation. Disagreement can occur because of random chance, network latency, or validators running different software versions. -- The volume of transactions in the network. A high rate of transactions can increase the work needed to close a new ledger version, reducing the number of separate ledgers closed. An extremely low rate of transactions can also cause ledgers to close further apart (because there is nothing to be done). -- Changes and optimizations to the XRP Ledger's consensus algorithm can affect the speed of consensus and the rate of disagreement either positively or negatively. +For estimates of how much disk space is required to store different amounts of history, see [Capacity Planning](capacity-planning.html#historical-data). ### Advisory Deletion diff --git a/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md b/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md index c972995275..ecaa9d7c7a 100644 --- a/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md +++ b/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md @@ -82,18 +82,34 @@ Non-validator production servers should be configured to use NuDB and to store t NuDB does not have performance-related configuration options available in `rippled.cfg`. -#### History Sharding - -`rippled` offers a history sharding feature that allows you to store a randomized range of ledgers in a separate shard store. You can use the `[shard_db]` stanza to configure the shard store to use a different type of key-value store than the one you defined for the ledger store using the `[node_db]` stanza. For more information about how to use this feature, see [History Sharding](history-sharding.html). ### Historical Data -The amount of historical data that a `rippled` server keeps online is a major contributor to required storage space. At the time of writing (2018-10-29), a `rippled` server stores about 12GB of data per day and requires 8.4TB to store the full history of the XRP Ledger. You can expect this amount to grow as transaction volume increases across the XRP Ledger network. You can control how much data you keep with the `online_delete` and `advisory_delete` fields. +The amount of historical data that a `rippled` server keeps available locally is a major contributor to the amount of disk space `rippled` requires. The server does not need to store more than the most recent 256 ledger versions to follow the consensus process and report the complete state of the ledger, but you can only query your server for transactions that executed in ledger versions it has stored locally. You can control how much data you keep with [online deletion](online-deletion.html); the default config file has the server keep the latest 2000 ledger versions. Without online deletion, the server's disk requirements grow without bounds. -Online deletion enables the purging of `rippled` ledgers from databases without any disruption of service. It removes only records that are not part of the current ledgers. Data in current ledgers means any data that's used by ledger versions that are new enough not to be deleted. Without online deletion, those databases grow without bounds. Freeing disk space requires stopping the process and manually removing database files. For more information, see [`[node_db]`: `online_delete`](https://github.com/ripple/rippled/blob/develop/cfg/rippled-example.cfg#L832). +The following table approximates the requirements for different amounts of history, at the time of writing (2018-12-13): - +| Real Time Amount | Number of Ledger Versions | Disk Space Required (RocksDB) | Disk Space Required (NuDB) | +|:-----------------|:--------------------------|:------------------------------|:--| +| 1 day | 25,000 | 8 GB | 12 GB | +| 14 days | 350,000 | 112 GB | 168 GB | +| 30 days | 750,000 | 240 GB | 360 GB | +| 90 days | 2,250,000 | 720 GB | 1 TB | +| 1 year | 10,000,000 | 3 TB | 4.5 TB | +| 2 years | 20,000,000 | 6 TB | 9 TB | +| Full history (through 2018) | 43,000,000+ | 9 TB | (Unknown) | + +These numbers are estimates. They depend on several factors, most importantly the volume of transactions in the network. As transaction volume increases, each ledger version stores more unique data. + +The `online_delete` setting tells the server how many ledger versions to keep after deleting old history. You should plan for enough disk space to store twice that many ledger versions at maximum (right before online deletion runs). + +For instructions on how to change the amount of history you keep, see [Configure Online Deletion](configure-online-deletion.html). + + +#### History Sharding + +`rippled` offers a history sharding feature that allows you to store a randomized range of ledgers in a separate shard store. You can use the `[shard_db]` stanza to configure the shard store to use a different type of key-value store than the one you defined for the ledger store using the `[node_db]` stanza. For more information about how to use this feature, see [History Sharding](history-sharding.html). ### Log Level From ab6b4d44352b385574b02b691f422bb28aea140f Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Thu, 13 Dec 2018 16:29:29 -0800 Subject: [PATCH 05/14] Online deletion: more interlinking --- content/concepts/the-rippled-server/online-deletion.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/content/concepts/the-rippled-server/online-deletion.md b/content/concepts/the-rippled-server/online-deletion.md index abd6406123..0512656895 100644 --- a/content/concepts/the-rippled-server/online-deletion.md +++ b/content/concepts/the-rippled-server/online-deletion.md @@ -32,7 +32,7 @@ Backfilling history is one of the server's lowest priorities, so it may take a l Some servers in the XRP Ledger network are configured as "full-history" servers. These servers, which require significantly more disk space than other tracking servers, collect all available XRP Ledger history and **do not use online deletion**. - +For instructions setting up full history, see [Configure Full History](configure-full-history.html). ### History Sharding @@ -113,11 +113,11 @@ The `can_delete` API method can enable advisory deletion (with the value `never` ## See Also +- [Capacity Planning](capacity-planning.html) - [can_delete method][] API reference documentation -- [Configure Online Deletion with Advisory Delete](configure-online-deletion.html) - - - +- [Configure Online Deletion](configure-online-deletion.html) +- [Configure Advisory Deletion](configure-advisory-deletion.html) +- [Configure Full History](configure-full-history.html) From a3400378c4730406b9d09f62d7c39787d267580a Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Tue, 18 Dec 2018 14:21:51 -0800 Subject: [PATCH 06/14] Online deletion revisions - Split ledger history section into a parent concept page w/ online deletion and history sharding as child pages - Clarified how much data is stored in general - Covered the special case for backfilling with advisory_delete enabled (including updating the diagram) - Changed full history recommendation to NuDB only --- .../online_delete-vs-ledger_history.uxf | 120 ++++++++++++++--- .../{ => ledger-history}/history-sharding.md | 0 .../ledger-history/ledger-history.md | 56 ++++++++ .../ledger-history/online-deletion.md | 107 +++++++++++++++ .../the-rippled-server/online-deletion.md | 126 ------------------ .../can_delete.md | 9 +- .../configure-advisory-deletion.md | 10 +- .../configuration/configure-full-history.md | 16 +-- .../installation/capacity-planning.md | 19 ++- dactyl-config.yml | 20 ++- img/online_delete-vs-ledger_history.png | Bin 8121 -> 21388 bytes 11 files changed, 317 insertions(+), 166 deletions(-) rename content/concepts/the-rippled-server/{ => ledger-history}/history-sharding.md (100%) create mode 100644 content/concepts/the-rippled-server/ledger-history/ledger-history.md create mode 100644 content/concepts/the-rippled-server/ledger-history/online-deletion.md delete mode 100644 content/concepts/the-rippled-server/online-deletion.md diff --git a/content/_img-sources/online_delete-vs-ledger_history.uxf b/content/_img-sources/online_delete-vs-ledger_history.uxf index 051c9efe45..5a924b0964 100644 --- a/content/_img-sources/online_delete-vs-ledger_history.uxf +++ b/content/_img-sources/online_delete-vs-ledger_history.uxf @@ -4,8 +4,8 @@ UMLClass - 30 - 70 + 50 + 140 330 40 @@ -18,8 +18,8 @@ lt=.. Relation - 20 - 110 + 40 + 180 740 50 @@ -33,8 +33,8 @@ m2=newest UMLClass - 360 - 70 + 380 + 140 170 40 @@ -47,8 +47,8 @@ transparency=0 UMLClass - 530 - 70 + 550 + 140 210 40 @@ -61,8 +61,8 @@ fg=#f5f7f9 Relation - 520 - 30 + 540 + 100 70 60 @@ -72,8 +72,8 @@ fg=#f5f7f9 Text - 570 - 30 + 590 + 100 210 30 @@ -84,8 +84,8 @@ style=wordwrap Text - 150 - 30 + 170 + 100 170 30 @@ -96,12 +96,100 @@ style=wordwrap Relation - 300 - 30 + 320 + 100 80 60 lt=<<<- 60.0;40.0;60.0;10.0;10.0;10.0 + + Relation + + 40 + 430 + 740 + 50 + + lt=->> + +Ledger versions +m1=oldest +m2=newest + 10.0;20.0;720.0;20.0 + + + Text + + 110 + 320 + 230 + 60 + + online_delete setting, or most recent can_delete point, whichever is older +style=wordwrap + + + + UMLClass + + 50 + 390 + 330 + 40 + + Delete automatically +bg=#e1e4e8 +fg=#23292f +lt=.. + + + + Relation + + 310 + 340 + 90 + 70 + + lt=<<<- + 70.0;50.0;70.0;10.0;10.0;10.0 + + + UMLClass + + 380 + 390 + 380 + 40 + + Backfill if possible +bg=#1db4ff +transparency=0 +fg=#f5f7f9 + + + + Text + + 50 + 60 + 220 + 30 + + *Without advisory deletion* + + + + Text + + 40 + 280 + 220 + 30 + + *With advisory deletion* + + diff --git a/content/concepts/the-rippled-server/history-sharding.md b/content/concepts/the-rippled-server/ledger-history/history-sharding.md similarity index 100% rename from content/concepts/the-rippled-server/history-sharding.md rename to content/concepts/the-rippled-server/ledger-history/history-sharding.md diff --git a/content/concepts/the-rippled-server/ledger-history/ledger-history.md b/content/concepts/the-rippled-server/ledger-history/ledger-history.md new file mode 100644 index 0000000000..14ad3b72e5 --- /dev/null +++ b/content/concepts/the-rippled-server/ledger-history/ledger-history.md @@ -0,0 +1,56 @@ +# Ledger History + +The [consensus process](intro-to-consensus.html) creates a chain of [validated ledger versions](ledgers.html), each derived from the previous one by applying a set of transactions. Every `rippled` server stores ledger versions and transaction history locally. The amount of transaction history a server stores depends on how long that server has been online and how much history it is configured to fetch and keep. + +Servers in the peer-to-peer XRP Ledger network share transactions and other data with each other as part of the consensus process. Each server each independently builds each new ledger version and compares results with its trusted validators to ensure consistency. (If a consensus of trusted validators disagrees with a server's results, that server fetches the necessary data from its peers to achieve consistency.) Servers can download older data from their peers to fill gaps in their available history. The structure of the ledger uses cryptographic [hashes](basic-data-types.html#hashes) of the data so that any server can verify the integrity and consistency of the data. + +## Databases + +Servers keep ledger state data and transactions in a key-value store called the _ledger store_. Additionally, `rippled` maintains a few SQLite database files for more flexible access to things like transaction history, and to track certain settings changes. + +It is generally safe to delete all of a `rippled` server's database files when that server is not running. (You may want to do this, for example, if you change the server's storage settings or if you are switching from a test net to the production network.) + +## Available History + +By design, all data and transactions in the XRP Ledger are public, and anyone can search or query anything. However, your server can only search data that it has available locally. If you try to query for a ledger version or transaction that your server does not have available, your server replies that it cannot find that data. Other servers that have the necessary history can respond successfully to the same query. If you have a business that uses XRP Ledger data, you should be mindful of how much history your server has available. + +The [server_info method][] reports how many ledger versions your server has available in the `complete_ledgers` field. + +## Fetching History + +When it starts, a `rippled` server's first priority is to get a complete copy of the latest validated ledger. From there, it keeps up with advances in the ledger progress. If configured to do so, the server also backfills ledger history up to a configured amount, which must be equal or less than the cutoff beyond which online deletion is configured to delete. + +The server can backfill history from before it became synced, as well as filling in any gaps in the history it has collected after syncing. (Gaps in ledger history can occur if a server temporarily becomes too busy to keep up with the network, loses its network connection, or suffers other temporary issues.) To backfill history, the server requests data from its peer `rippled` servers. The amount the server tries to backfill is defined by the `[ledger_history]` setting. + +The XRP Ledger identifies data (on several different levels) by a unique hash of its contents. The XRP Ledger's state data contains a short summary of the ledger's history, in the form of the [LedgerHashes object type](ledgerhashes.html). Servers use the LedgerHashes objects to know which ledger versions to fetch, and to confirm that the ledger data they receive is correct and complete. + +Backfilling history is one of the server's lowest priorities, so it may take a long time to fill missing history, especially if the server is busy or has less than sufficient hardware and network specs. For recommendations on hardware specs, see [Capacity Planning](capacity-planning.html). Backfilling history also requires that at least one of the server's direct peers has the history in question. + +### With Advisory Deletion + +If [online deletion](online-deletion.html) and advisory deletion are both enabled, the server automatically backfills data up to the oldest ledger it has not been allowed to delete yet. This can fetch data beyond the number of ledger versions configured in the `[ledger_history]` and `online_delete` settings. The [can_delete method][] tells the server what ledger versions it is allowed to delete. + + +## Full History + +Some servers in the XRP Ledger network are configured as "full-history" servers. These servers, which require significantly more disk space than other tracking servers, collect all available XRP Ledger history and **do not use online deletion**. + +Ripple provides a set of public full-history servers as a public service at `s2.ripple.com`. This service is provided for the benefit of the larger XRP community. Ripple reserves the right to block those who abuse the servers or use more than their fair share of the servers' resources. + +**Tip:** Unlike some cryptocurrency networks, servers in the XRP Ledger do not need full history to know the current state and keep up with current transactions. + +For instructions setting up full history, see [Configure Full History](configure-full-history.html). + +## History Sharding + +An alternative to storing the full history of the XRP Ledger on a single expensive machine is to configure many servers to each store a portion of ledger history. The [History Sharding](history-sharding.html) feature makes this possible, storing ranges of ledger history in a separate storage area called the _shard store_. When a peer server asks for specific data (as described in [fetching history](#fetching-history) above), a server can answer using data from either its ledger store or shard store. + +Online deletion **does not** delete from the shard store. However, if you configure online deletion to keep at least 32768 ledger versions in your server's ledger store, your server can copy full shards from the ledger store to the shard store before automatically deleting them from the ledger store. + +For more information, see [Configure History Sharding](configure-history-sharding.html). + + + +{% include '_snippets/rippled-api-links.md' %} +{% include '_snippets/tx-type-links.md' %} +{% include '_snippets/rippled_versions.md' %} diff --git a/content/concepts/the-rippled-server/ledger-history/online-deletion.md b/content/concepts/the-rippled-server/ledger-history/online-deletion.md new file mode 100644 index 0000000000..411cfa957d --- /dev/null +++ b/content/concepts/the-rippled-server/ledger-history/online-deletion.md @@ -0,0 +1,107 @@ +# Online Deletion +[[Source]
](https://github.com/ripple/rippled/blob/master/src/ripple/app/misc/SHAMapStoreImp.cpp "Source") + +The online deletion feature lets the `rippled` server delete the server's local copy of old ledger versions to keep disk usage from rapidly growing over time. The default config file sets online deletion to run automatically, but online deletion can also be configured to run only when prompted. [New in: rippled 0.27.0][] + +The server always keeps the complete _current_ state of the ledger, with all the balances and settings it contains. The deleted data includes older transactions and versions of the ledger state that are older than the stored history. + +The default config file sets the `rippled` server to keep the most recent 2000 ledger versions and automatically delete older data. + +**Tip:** Even with online deletion, the amount of disk space required to store the same time span's worth of ledger data increases over time, because the size of individual ledger versions tends to grow over time. This growth is very slow in comparison to the accumulation of data that occurs without deleting old ledgers. For more information on disk space needs, see [Capacity Planning](capacity-planning.html). + + +## Background + +The `rippled` server stores [ledger history](ledger-history.html) in its _ledger store_. This data accumulates over time. + +Inside the ledger store, ledger data is "de-duplicated": data that doesn't change from version to version is only stored once. The records themselves in the ledger store do not indicate which ledger version(s) contain them; part of the work of online deletion is identifying which records are only used by outdated ledger versions. This process is time consuming and affects the disk I/O and application cache, so it is not feasible to delete old data on every ledger close. + + +## Online Deletion Behavior + +The online deletion settings configure how many ledger versions the `rippled` server should keep available in the ledger store at a time. However, the specified number is a guideline, not a hard rule: + +- The server never deletes data more recent than the configured number of ledger versions, but it may have less than that amount available if it has not been running for long enough or if it lost sync with the network at any time. (The server attempts to backfill at least some history; see [fetching history](ledger-history.html#fetching-history) for details.) +- The server may store up to twice the configured number of ledger versions if online deletion is set to run automatically. (Each time it runs, it reduces the number of stored ledger versions to approximately the configured number.) +- If advisory deletion is enabled, the server stores all the ledger versions that it has acquired and built until its administrator calls the [can_delete method][]. + + If you call [can_delete][can_delete method] on a regular basis, the maximum amount of data the server stores is either of the following, whichever is larger: + + - A number of ledger versions approximately equal to twice the `online_delete` value. (After deletion, this is reduced to approximately the `online_delete` value.) + - Ledger versions spanning an amount of time that is approximately twice the interval between `can_delete` calls. (After deletion, this is reduced to approximately one interval's worth of data.) + + For example, if you call `can_delete` with a value of `now` once per day and an `online_delete` value of 2000, the server typically stores up two full day's worth of ledger versions before running deletion. After running deletion, the server keeps approximately 1 day's worth, but never less than 2000 ledger versions. + +With online deletion enabled and running automatically (that is, with advisory delete disabled), the total amount of ledger data stored should remain at minimum equal to the number of ledger versions the server is configured to keep, with the maximum being roughly twice that many. + +When online deletion runs, it does not reduce the size of SQLite database files on disk; it only makes space within those files available to be reused for new data. Online deletion _does_ reduce the size of RocksDB or NuDB database files containing the ledger store. + +The server only counts validated ledger versions when deciding how far back it can delete. In exceptional circumstances where the server is unable to validate new ledger versions (either because of an outage in its local network connection or because the global XRP Ledger network is unable to reach a consensus) `rippled` continues to close ledgers so that it can recover quickly when the network is restored. In this case, the server may accumulate many closed but not validated ledger versions. These unvalidated ledgers do not affect how many _validated_ ledger versions the server keeps before running online deletion. + +### Interrupting Online Deletion + +Online deletion automatically stops if the [server state](rippled-server-states.html) becomes less than `full`. If this happens, the server writes a log message with the prefix `SHAMapStore::WRN`. The server attempts to start online deletion again after the next validated ledger version after becoming fully synced. + +If you stop the server or it crashes while online deletion is running, online deletion resumes after the server is restarted and the server becomes fully synced. + +To temporarily disable online deletion, you can use the [can_delete method][] with an argument of `never`. This change persists until you re-enable online deletion by calling [can_delete][can_delete method] again, even if you change the . For more information on controlling when online deletion happens, see [Advisory Deletion](#advisory-deletion). + + +## Configuration + +The following settings relate to online deletion: + +- **`online_delete`** - Specify a number of validated ledger versions to keep. The server periodically deletes any ledger versions that are older than this number. If not specified, no ledgers are deleted. + + The default config file specifies 2000 for this value. This cannot be less than 256, because some events like [Fee Voting](fee-voting.html) and the [Amendment Process](amendments.html#amendment-process) update only every 256 ledgers. + + **Caution:** If you run `rippled` with `online_delete` disabled, then later enable `online_delete` and restart the server, the server disregards but does not delete existing ledger history that your server already downloaded while `online_delete` was disabled. To save disk space, delete your existing history before re-starting the server with `online_delete` back on. + +- **`[ledger_history]`** - Specify a number of validated ledgers, equal to or less than `online_delete`. If the server does not have at least this many validated ledger versions, it attempts to backfill them by fetching the data from peers. + + The default for this setting is 256 ledgers. + + The following diagram shows the relationship between `online_delete` and `ledger_history` settings: + + ![Ledgers older than `online_delete` are automatically deleted. Ledgers newer than `ledger_history` are backfilled. Ledgers in between are kept if available but not backfilled](img/online_delete-vs-ledger_history.png) + +- **`advisory_delete`** - If enabled, online deletion is not scheduled automatically. Instead, an administrator must manually trigger online deletion. Use the value `0` for disabled or `1` for enabled. + + This setting is disabled by default. + +- **`[fetch_depth]`** - Specify a number of ledger versions. The server does not accept fetch requests from peers for historical data that is older than the specified number of ledger versions. Specify the value `full` to serve any available data to peers. + + The default for `fetch_depth` is `full` (serve all available data). + + The `fetch_depth` setting cannot be higher than `online_delete` if both are specified. If `fetch_depth` is set higher, the server treats it as equal to `online_delete` instead. + + The following diagram shows how fetch_depth works: + + ![Ledger versions older than fetch_depth are not served to peers](img/fetch_depth.png) + +For estimates of how much disk space is required to store different amounts of history, see [Capacity Planning](capacity-planning.html#historical-data). + +### Advisory Deletion + +By default, online deletion happens automatically and periodically. If the `advisory_delete` setting is enabled in the config file, online deletion only happens when an administrator triggers it using the [can_delete method][]. + +You can use advisory deletion with a scheduled job to trigger automatic deletion based on clock time instead of the number of ledger versions closed. If your server is heavily used, the extra load from online deletion can cause your server to fall behind and temporarily de-sync from the consensus network. If this is the case, you can use advisory deletion and schedule online deletion to happen only during off-peak times. + +Alternatively, you can use advisory deletion for other reasons, such as if you want to manually confirm that transaction data is backed up to a separate server before deleting it. + +The `can_delete` API method can enable or disable automatic deletion, in general or up to a specific ledger version, as long as `advisory_delete` is enabled in the config file. These settings changes persist even if you restart the `rippled` server, unless you disable `advisory_delete` in the config file before restarting. + + +## See Also + +- [Capacity Planning](capacity-planning.html) +- [can_delete method][] API reference documentation +- [Configure Online Deletion](configure-online-deletion.html) +- [Configure Advisory Deletion](configure-advisory-deletion.html) +- [Configure Full History](configure-full-history.html) + + + +{% include '_snippets/rippled-api-links.md' %} +{% include '_snippets/tx-type-links.md' %} +{% include '_snippets/rippled_versions.md' %} diff --git a/content/concepts/the-rippled-server/online-deletion.md b/content/concepts/the-rippled-server/online-deletion.md deleted file mode 100644 index 0512656895..0000000000 --- a/content/concepts/the-rippled-server/online-deletion.md +++ /dev/null @@ -1,126 +0,0 @@ -# Online Deletion -[[Source]
](https://github.com/ripple/rippled/blob/master/src/ripple/app/misc/SHAMapStoreImp.cpp "Source") - -The online deletion feature lets the `rippled` server delete the server's local copy of old ledger versions to keep disk usage from rapidly growing over time. Online deletion happens automatically by default, but can be configured to run only when prompted. [New in: rippled 0.27.0][] - -The server always keeps the complete _current_ state of the ledger, with all the balances and settings it contains. The deleted data includes older transactions and versions of the ledger state that are older than the stored history. - -The default config file sets the `rippled` server to keep the most recent 2000 ledger versions and automatically delete older data. - -**Tip:** Even with online deletion, the amount of disk space required to store the same time span's worth of ledger data increases over time, because the size of individual ledger versions tends to grow over time. This growth is very slow in comparison to the accumulation of data that occurs without deleting old ledgers. For more information on disk space needs, see [Capacity Planning](capacity-planning.html). - - -## Background - -The `rippled` server stores data from recent ledger versions in its _ledger store_. This data accumulates as the XRP Ledger network creates new ledger versions and validates them through the [consensus process](intro-to-consensus.html). - -The XRP Ledger stores the complete state of the current ledger, with all balances, settings, and other data, in each ledger version. Unlike many cryptocurrency systems, an XRP Ledger server does not require full history to sync with the network, know the full state, and contribute to making forward progress. - -Inside the ledger store, ledger data is "de-duplicated", with data that doesn't change from version to version only stored once. The records themselves in the ledger store do not indicate which ledger version(s) contain them; part of the work of online deletion is identifying which records are only used by outdated ledger versions. This process is time consuming and affects the disk I/O and application cache, so it is not feasible to delete old data on every ledger close. - -### Fetching History - -When it starts, a `rippled` server's first priority is to get a complete copy of the latest validated ledger. From there, it keeps up with advances in the ledger progress. If configured to do so, the server also backfills ledger history up to a configured amount, which must be equal or less than the cutoff beyond which online deletion is configured to delete. - -The server can backfill history from before it became synced, as well as filling in any gaps in the history it has collected after syncing. (Gaps in ledger history can occur if a server temporarily becomes too busy to keep up with the network, loses its network connection, or suffers other temporary issues.) To backfill history, the server requests data from its peer `rippled` servers. The amount the server tries to backfill is defined by the `[ledger_history]` setting. - -The XRP Ledger identifies data (on several different levels) by a unique hash of its contents. The XRP Ledger's state data contains a short summary of the ledger's history, in the form of the [LedgerHashes object type](ledgerhashes.html). Servers use the LedgerHashes objects to know which ledger versions to fetch, and to confirm that the ledger data they receive is correct and complete. - -Backfilling history is one of the server's lowest priorities, so it may take a long time to fill missing history, especially if the server is busy or has less than sufficient hardware and network specs. For recommendations on hardware specs, see [Capacity Planning](capacity-planning.html). Backfilling history also requires that at least one of the server's direct peers has the history in question. - -### Full History - -Some servers in the XRP Ledger network are configured as "full-history" servers. These servers, which require significantly more disk space than other tracking servers, collect all available XRP Ledger history and **do not use online deletion**. - -For instructions setting up full history, see [Configure Full History](configure-full-history.html). - -### History Sharding - -An alternative to storing the full history of the XRP Ledger on a single expensive machine is to configure many servers to each store a portion of ledger history. The [History Sharding](history-sharding.html) feature makes this possible, storing ranges of ledger history in a separate storage area called the _shard store_. When a peer server asks for specific data (as described in [fetching history](#fetching-history) above), a server can answer using data from either its ledger store or shard store. - -Online deletion **does not** delete from the shard store. However, if you configure online deletion to keep at least 32768 ledger versions in your server's ledger store, your server can copy full shards from the ledger store to the shard store before automatically deleting them from the ledger store. - -For more information, see [Configure History Sharding](configure-history-sharding.html). - - -## Online Deletion Behavior - -The online deletion settings configure how many ledger versions the `rippled` server should keep available in the ledger store at a time. However, the specified number is a guideline, not a hard rule: - -- The server may have less than the configured number of ledger versions if it has not been running for long enough or if it lost sync with the network at any time. (The server attempts to backfill at least some history; see [fetching history](#fetching-history) above for details.) -- The server may store up to twice the configured number of ledger versions if online deletion is set to run automatically. (Each time it runs, it reduces the number of stored ledger versions to the configured number.) -- If advisory delete is enabled, the server stores all the ledger versions that it has acquired and built until someone calls the [can_delete method][]. - - For example, if you call `can_delete` with a value of `now` once per day and an `online_delete` value of 2000, the server typically stores 2000 ledger versions at minimum and just over a full day's worth of ledger versions at maximum. - -With online deletion enabled and running automatically (that is, with advisory delete disabled), the total amount of ledger data stored should remain at minimum equal to the number of ledger versions the server is configured to keep, with the maximum being roughly twice that many. - -When online deletion runs, it does not reduce the size of the database files on disk; it only makes space within those files available to be reused for new data. To actually reduce the disk space allocated to the database files, you must restart `rippled`. - -### Interrupting Online Deletion - -Online deletion automatically stops if the [server state](rippled-server-states.html) becomes less than `full`. If this happens, the server writes a log message with the prefix `SHAMapStore::WRN`. The server attempts to start online deletion again after the next validated ledger version after becoming fully synced. - -If you stop the server or it crashes while online deletion is running, online deletion resumes after the server is restarted and the server becomes fully synced. - -To temporarily disable online deletion, you can use the [can_delete method][] with an argument of `never`. This change persists until the server restarts or you re-enable online deletion by calling [can_delete][can_delete method] again. For more information on controlling when online deletion happens, see [Advisory Deletion](#advisory-deletion). - - -## Configuration - -The following settings relate to online deletion: - -- **`online_delete`** - Specify a number of validated ledger versions to keep. The server periodically deletes any ledger versions that are older than this number. If not specified, no ledgers are deleted. - - The default config file specifies 2000 for this value. This cannot be less than 256, because some events like [Fee Voting](fee-voting.html) and the [Amendment Process](amendments.html#amendment-process) update only every 256 ledgers. - - **Caution:** If you run `rippled` with `online_delete` disabled, then later enable `online_delete` and restart the server, the server disregards but does not delete existing ledger history that your server already downloaded while `online_delete` was disabled. To save disk space, delete your existing history before re-starting the server with `online_delete` back on. - -- **`[ledger_history]`** - Specify a number of validated ledgers, equal to or less than `online_delete`. If the server does not have at least this many validated ledger versions, it attempts to backfill them by fetching the data from peers. - - The default for this setting is 256 ledgers. - - The following diagram shows the relationship between `online_delete` and `ledger_history` settings: - - ![Ledgers older than `online_delete` are automatically deleted. Ledgers newer than `ledger_history` are backfilled. Ledgers in between are kept if available but not backfilled](img/online_delete-vs-ledger_history.png) - -- **`advisory_delete`** - If enabled, online deletion is not scheduled automatically. Instead, an administrator must manually trigger online deletion. Use the value `0` for disabled or `1` for enabled. - - This setting is disabled by default. - -- **`[fetch_depth]`** - Specify a number of ledger versions. The server does not accept fetch requests from peers for historical data that is older than the specified number of ledger versions. Specify the value `full` to serve any available data to peers. - - The default for `fetch_depth` is `full` (serve all available data). - - The `fetch_depth` setting cannot be higher than `online_delete` if both are specified. If `fetch_depth` is set higher, the server treats it as equal to `online_delete` instead. - - The following diagram shows how fetch_depth works: - - ![Ledger versions older than fetch_depth are not served to peers](img/fetch_depth.png) - -For estimates of how much disk space is required to store different amounts of history, see [Capacity Planning](capacity-planning.html#historical-data). - -### Advisory Deletion - -By default, online deletion happens automatically and periodically. If the `advisory_delete` setting is enabled, online deletion only happens when an administrator triggers it using the [can_delete method][]. You can run this command with a scheduled job to trigger automatic deletion based on clock time instead of the number of ledger versions closed. - -If your server is heavily used, the extra load from online deletion can cause your server to fall behind and temporarily de-sync from the consensus network. If this is the case, you can use advisory delete and schedule online deletion to happen only during off-peak times. - -You can also use advisory delete for other reasons, such as if you want to manually confirm that the transaction data is backed up to a separate server before deleting it. - -The `can_delete` API method can enable advisory deletion (with the value `never`) or disable advisory delete (with the value `always`). These setting changes persist until you restart the `rippled` server, at which point the settings in the config file override them. - - -## See Also - -- [Capacity Planning](capacity-planning.html) -- [can_delete method][] API reference documentation -- [Configure Online Deletion](configure-online-deletion.html) -- [Configure Advisory Deletion](configure-advisory-deletion.html) -- [Configure Full History](configure-full-history.html) - - - -{% include '_snippets/rippled-api-links.md' %} -{% include '_snippets/tx-type-links.md' %} -{% include '_snippets/rippled_versions.md' %} diff --git a/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/can_delete.md b/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/can_delete.md index 1a3586bd68..db8f1ea407 100644 --- a/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/can_delete.md +++ b/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/can_delete.md @@ -1,7 +1,7 @@ # can_delete [[Source]
](https://github.com/ripple/rippled/blob/master/src/ripple/rpc/handlers/CanDelete.cpp "Source") -With `online_delete` and `advisory_delete` configuration options enabled, the `can_delete` method informs the rippled server of the latest ledger which may be deleted. +The `can_delete` method informs the `rippled` server of the latest ledger version which may be deleted when using [online deletion with advisory deletion enabled](online-deletion.html#advisory-deletion). If advisory deletion is not enabled, this method does nothing. _The `can_delete` method is an [admin method](admin-rippled-methods.html) that cannot be run by unprivileged users._ @@ -47,7 +47,7 @@ The request includes the following optional parameter: | `Field` | Type | Description | |:-------------|:------------------|:------------------------------------------| -| `can_delete` | String or Integer | The maximum ledger to allow to be deleted. For `ledger_index` or `ledger_hash`, see [Specifying Ledgers][]. `never` sets the value to 0, and effectively disables online deletion until another `can_delete` is appropriately called. `always` sets the value to the maximum possible ledger (4294967295), and online deletion occurs as of each configured `online_delete` interval. `now` triggers online deletion at the next validated ledger that meets or exceeds the configured `online_delete` interval, but no further. | +| `can_delete` | String or Integer | The [Ledger Index][] of the maximum ledger version to allow to be deleted. The special case `never` disables online deletion. The special case `always` enables automatic online deletion as if advisory deletion was disabled. The special case `now` allows online deletion one time at the next validated ledger that meets or exceeds the configured `online_delete` value. | If no parameter is specified, no change is made. @@ -68,6 +68,11 @@ Use this command with no parameter to query the existing `can_delete` setting. * `lgrNotFound` - Ledger not found. * `invalidParams` - Invalid parameters. +## See Also + +- [Online Deletion](online-deletion.html) +- [Configure Advisory Deletion](configure-advisory-deletion.html) + {% include '_snippets/rippled-api-links.md' %} {% include '_snippets/tx-type-links.md' %} diff --git a/content/tutorials/manage-the-rippled-server/configuration/configure-advisory-deletion.md b/content/tutorials/manage-the-rippled-server/configuration/configure-advisory-deletion.md index c94544627d..01073116cd 100644 --- a/content/tutorials/manage-the-rippled-server/configuration/configure-advisory-deletion.md +++ b/content/tutorials/manage-the-rippled-server/configuration/configure-advisory-deletion.md @@ -1,6 +1,6 @@ # Configure Advisory Deletion -In its default configuration, the `rippled` server automatically deletes outdated history of XRP Ledger state and transactions as new ledger versions become available. If your server uses most of its hardware resources during peak hours, you can configure the server to delete ledgers only when prompted by a command scheduled to run during off-peak hours to reduce the chances of online deletion +The default config file sets `rippled` to automatically delete outdated history of XRP Ledger state and transactions as new ledger versions become available. If your server uses most of its hardware resources during peak hours, you can configure the server to delete ledgers only when prompted by a command scheduled to run during off-peak hours to reduce the chances of online deletion. ## Prerequisites @@ -37,7 +37,7 @@ To configure advisory deletion with a daily schedule, perform the following step online_delete=2000 advisory_delete=1 - - Set `advisory_delete` to `1` to require approval before running online deletion. (Set it to `0` to run online deletion automatically as new ledger versions become available.) + - Set `advisory_delete` to `1` to run online deletion only when prompted. (Set it to `0` to run online deletion automatically as new ledger versions become available.) - Set `online_delete` to the minimum number of ledger versions to keep after running online deletion. The server accumulates more history than this until online deletion runs. {% include '_snippets/conf-file-location.md' %} @@ -57,6 +57,8 @@ To configure advisory deletion with a daily schedule, perform the following step } } + The server only deletes those ledger versions if the number of _newer_ validated ledger versions it has is equal to or greater than the `online_delete` setting. + 3. Configure your `cron` daemon to run the `can_delete` method you tested in the previous step at a scheduled time. Edit your `cron` configuration: @@ -67,9 +69,9 @@ To configure advisory deletion with a daily schedule, perform the following step 5 1 * * * rippled --conf /etc/opt/ripple/rippled.cfg can_delete now - Be sure that you schedule the command to run based on the time zone your server's clock is configured to use. Ripple recommends running all production servers with the UTC time zone. + Be sure that you schedule the command to run based on your server's configured time zone. - **Tip:** You do not need to schedule a `cron` job to run online deletion if you have `advisory_delete` disabled. In that case, `rippled` runs online deletion automatically when the server's oldest available ledger is twice as old as the number of ledgers to keep after deletion. + **Tip:** You do not need to schedule a `cron` job to run online deletion if you have `advisory_delete` disabled. In that case, `rippled` runs online deletion automatically when the server has approximately twice the number of ledgers to keep after deletion. 4. Start (or restart) the `rippled` service. diff --git a/content/tutorials/manage-the-rippled-server/configuration/configure-full-history.md b/content/tutorials/manage-the-rippled-server/configuration/configure-full-history.md index a81ba2e7a1..f6f534df7a 100644 --- a/content/tutorials/manage-the-rippled-server/configuration/configure-full-history.md +++ b/content/tutorials/manage-the-rippled-server/configuration/configure-full-history.md @@ -20,23 +20,17 @@ To configure your server to acquire and store full history, complete the followi $ sudo systemctl stop rippled -0. Remove (or comment out) the `online_delete` and `advisory_delete` settings from the `[node_db]` stanza of your server's config file: +0. Remove (or comment out) the `online_delete` and `advisory_delete` settings from the `[node_db]` stanza of your server's config file, and change the type to `NuDB` if you haven't already: [node_db] - type=RocksDB - path=/var/lib/rippled/db/rocksdb - open_files=2000 - filter_bits=12 - cache_mb=256 - file_size_mb=8 - file_size_mult=2 - compression=1 + type=NuDB + path=/var/lib/rippled/db/nudb #online_delete=2000 #advisory_delete=0 - You can store full history with either RocksDB or NuDB as the key-value store behind the ledger store. NuDB requires less RAM but RocksDB requires less disk space. For more information, see [Capacity Planning](capacity-planning.html). + On a full-history server, you should use NuDB for the ledger store, because RocksDB requires too much RAM when the database is that large. For more information, see [Capacity Planning](capacity-planning.html). You can remove the following performance-related configuration options from the default `[node_db]` stanza, because they only apply to RocksDB: `open_files`, `filter_bits`, `cache_mb`, `file_size_mb`, and `file_size_mult.` - If you are using RocksDB, add `compression=1` to enable Snappy compression, which reduces the disk space necessary to store full history (at a cost of greater CPU usage when reading or writing history). + **Caution:** If you have any history already downloaded with RocksDB, you must either delete that data or change the `path` field when you switch to NuDB. Otherwise, the server may [fail to start](server-wont-start.html). {% include '_snippets/conf-file-location.md' %} diff --git a/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md b/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md index ecaa9d7c7a..050dc27780 100644 --- a/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md +++ b/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md @@ -64,14 +64,18 @@ RocksDB has performance-related configuration options that you can set in `rippl ``` [node_db] type=RocksDB -path={path_to_ledger_store} +path=/var/lib/rippled/db/rocksdb open_files=512 filter_bits=12 cache_mb=512 file_size_mb=64 file_size_mult=2 +online_delete=2000 +advisory_delete=0 ``` +(Adjust the `path` to the directory where you want to keep the ledger store on disk. Adjust the `online_delete` and `advisory_delete` settings as desired for your configuration.) + #### More About Using NuDb [NuDB](https://github.com/vinniefalco/nudb#introduction) is an append-only key-value store that is optimized for SSD drives. @@ -80,8 +84,17 @@ NuDB has nearly constant performance and memory footprints regardless of the amo Non-validator production servers should be configured to use NuDB and to store the amount of historical data required for the use case. -NuDB does not have performance-related configuration options available in `rippled.cfg`. +NuDB does not have performance-related configuration options available in `rippled.cfg`. Here is the recommended configuration for a `rippled` server using NuDB: +``` +[node_db] +type=NuDB +path=/var/lib/rippled/db/nudb +online_delete=2000 +advisory_delete=0 +``` + +(Adjust the `path` to the directory where you want to keep the ledger store on disk. Adjust the `online_delete` and `advisory_delete` settings as desired for your configuration.) ### Historical Data @@ -98,7 +111,7 @@ The following table approximates the requirements for different amounts of histo | 90 days | 2,250,000 | 720 GB | 1 TB | | 1 year | 10,000,000 | 3 TB | 4.5 TB | | 2 years | 20,000,000 | 6 TB | 9 TB | -| Full history (through 2018) | 43,000,000+ | 9 TB | (Unknown) | +| Full history (through 2018) | 43,000,000+ | (Not recommended) | ~9 TB | These numbers are estimates. They depend on several factors, most importantly the volume of transactions in the network. As transaction volume increases, each ledger version stores more unique data. diff --git a/dactyl-config.yml b/dactyl-config.yml index 6e6320b737..695ac12322 100644 --- a/dactyl-config.yml +++ b/dactyl-config.yml @@ -536,24 +536,36 @@ pages: targets: - local - - md: concepts/the-rippled-server/history-sharding.md - html: history-sharding.html + - md: concepts/the-rippled-server/ledger-history/ledger-history.md + html: ledger-history.html funnel: Docs doc_type: Concepts category: The rippled Server - blurb: History sharding divides the work of keeping historical ledger data among rippled servers. + subcategory: Ledger History + blurb: rippled servers store a variable amount of transaction and state history locally. targets: - local - - md: concepts/the-rippled-server/online-deletion.md + - md: concepts/the-rippled-server/ledger-history/online-deletion.md html: online-deletion.html funnel: Docs doc_type: Concepts category: The rippled Server + subcategory: Ledger History blurb: Online deletion purges outdated transaction and state history. targets: - local + - md: concepts/the-rippled-server/ledger-history/history-sharding.md + html: history-sharding.html + funnel: Docs + doc_type: Concepts + category: The rippled Server + subcategory: Ledger History + blurb: History sharding divides the work of keeping historical ledger data among rippled servers. + targets: + - local + - md: concepts/the-rippled-server/peer-protocol.md html: peer-protocol.html funnel: Docs diff --git a/img/online_delete-vs-ledger_history.png b/img/online_delete-vs-ledger_history.png index 3268afa3fc12aac1588bbfe67ee889ba5e128cb3..702885f826cfade8fb19ed64d51b040fde553ac6 100644 GIT binary patch literal 21388 zcmeIa1yq&YxA04-h@ifpqDZ|WAuS*!Wq>rOAZ!|?CAM^jq9P&PsnSxL*mNn=_q$`9G5!uj_rnwGS!=GDzlqO7MX9rAuAafe!#gV@E%6u+ zkMK`CJp9(v1aRaNiRKC(p0cux#9bA~!KD#mtxMR$Bda`Ibq?hdVC`yiT2UZhV?h^2Lit?z*ny_?zLFh5f;8kyQ%!bw2iyh+ z2I}hSOtqc2mbHK6EcE4?X&cP4A~V%;Evw3D_|2ns5)R3hLO2xtJj4Cod&G?~;o76`r!o=!%cdCL$Str|;+AqaK6i0Aq8s!&T_M|_0d4zVxK6i`KsdRpS zh5M5;c0NTme7+~6yEjt}&0=I=u)VvB$HO4{X)#npbdI`XD5?yBn4eH(X9RyDivUtp0bCln-i-Deg!kf;$!$_8HXuQ@N&1u{o7hIf|mnZ3a zF}Rp#uil1+l2U!2Q~j!sk5354sO!3dOo~u@4F4&PwAAEe=}29VgC`5t6W@rPsAy<7 z2wR5GW_>Z=1IkwKza{O=*HcmH$cqitv4DAw(BFDWb7AijU`5M1UzY z?KY6Q;nJL(ocxlIsCnVKRKP7WjQnO;2ybD$<<8nP$Fk+6asx`QO;zD1%Egx22c*%f z*oFWa;pU;4wpc2ntkuZ|af2H2r%#^_77`E=*4Ead`877R0s5uXxmX!JSa`=RA z)2}JZb-HzkoBkOBRasfz#Xr&U>HCWvAH8gc%j_~MFpBry?X1s^A!~RjXTA_b*cSx9 z$6wgrSs%2kR4*`+F$gZ+oud#I9vc~@)}^lw7$-IjJx-C8eDGk99JL{RjcmOBJ(aM- zGW+A?cdH7^BV~5oo2#9jo#g)hTzcwp5qUp;=q+9BS(|QQ)aNtp`nLV{+LA#OdhZ)w zy;_HX^^6SzagxKzD5$e{fq~Ud>!}3=1-3`?nYq?+d}jD^)ulR)&dwTpuX^5Z{rVMD z{9XKLA4KZHg$sm)gq0N)8>-Rr8r&$3vDBq?=^6q&kp+A+*foP<<&LWpgJZ)!IQ>Wi z<1@E#pt^A3x*Oxm>Fvn3y8)Q}mHmT*1MI4|x3}D7A|mhSy`SGx3+0>jm3vyh@ zRL%YjnthWwXnpp(z&QlD&+g`u84{^mYCXL*>hgU`hA1X1ER4sn*`zp6x7@zXHRtT9 z+Ej(u+jjFRRjy8yPb!_UMr|=Ej`Ch&X1?B!4kqc2ItEJ}R&>mCgUVKUMx54XnjC#E zv2709eCI=sx^{k&+-+N%`L2%Q|ECusQR)ys+EF5=4`+SSrE~!dJpl!+maA)p{rz6% zMAxrfczB+kZ@g2qCHD5lj>L>wqc%2dDNg?%S>%6b&Hwur`QJQ&KXJ!fuXuWfv*9nSns@&`BT#%zw}#+OO+KTZDC5wrBKH&Q!^J&4DoErC%qInhMo(Lt z#mZ=9{X#`r+N>u&A*)LI!yeE~n~WCViBsN=)klm4$?zwYlx(Foh$vK;Oo9UX0I zYU=A#6V;tj=3aR=XFzz%16woLU_o*rmlq)m)wUsHiJfuKfJ@^9Eb;Z9R2r9r;-P zwt)fl+s(HpS5~;puw`}&7qhSO8u@1x-@SX6o15GE=k?5#s^{ud;Z|}NLwSWF@>8D5 z$jC6th9OE`d?dA5ov3RHWlj{dx_7X;cUvY0VaelPMGZUNxPp2O!x?dDN&Or`X`GV1 z&KGGhnS`Lcq#ued2h%MV!5-Z9ZkIFy{oWH}UZ>hv>R zo<&;Wv~Gl3z9|AB)lk0YZ83b#m*J0_LXfW8WKy*I!EV#)>*IGxeVlL5G*a$B)evL9 zG^lR&#M;^#h6mn2Zl!{fQtsmv-=gO%4^oOHM1>}wjMTq<VCjqh|`jCfmaGB3qI(FTI(A_c=7sJWTu zwRMXa=TOV8sE_x(n;X!5BoyrFDj6Z}1U}hx_$242LFMPKy?*<4UiWcA{_~(IN#C}| z3#;X5sRw+T`lZPKMaX^MU4WY+GIq!MCnlQ;pTu?VJSFO)Sq}Pr;4?!Ny(%I!D)o!jlmrQadGiVzs$_c)txCVVuZu;Fw3J)-{OR9%+1588Duiqb?8-o4@A^u zb7O6-OJ@vOd&|<&aodU~#|`x|rJ3d`?W!#x>YU0tbZXh0jp1_iyC>q@Ef zyZk+h`+g_u0KG}EKz)6E%uPTFaTG|}noL`Wpk(wb=Ou$Cx`T7Zl~K+HG%5qHt>8H6ux zz3Mg<${w6z)2S#)j((D<8ZF>|EL@y?d`|+TL3Qb^BAsp6^nAdB5)u|}QF-2zeoaa) zI5>Dd6aoJHxpU{hGOTTE+}a`3_DkXsYA@gy8o+>C@pW`mXV2@Y=2a#lCAmmWi=Omg z>b$0#;IOo{wyG#Ky0S89n`>EGUlB0r_|r|ucCp`Tsxf%+*_Yy@Abh#J3C*GHbhQU& z`YqX>Dg9@n%{9%(k{5QjR=C#f_BJinb#N1U=g|1(m3d6Iu*;^oecq2DW5ZU-rJ-Vv zW~Fn}1t%422ogkucEj|)rKhLAd-o3f_olnab=BQQmr1JJRO^WnxP*nvyVH~kYKRey z;$OdhWf${MFDrO)3lwXTRA3JH8qqw48pW10SFSYT#Ldap(Dh8QBad)EKtNSh6>bQ4 z{dKR9bbspM>(zeYj6l-i0?=~AuO&k8WNs_v_av*Wb4oE=HkH&+2@m7j@&~;wmaD)dhdwh4vxH=KHN&5CXXWV^>!Z zxwpr!T0?k-goF*^%ql-OaK*u||MF~ni0DoY!LNtA8*|E^B!~y9BiPh(VD8!f`2B9@ z+1S~$Z0EWLy8{Ub2oQh%euVXxnwNxzmxc9e@F-xmJemJM?N-flZ|UkDM#nITJAEP1 znToyaprjvG`eLp*K1^j^J+u4E?Dl3xgYJo}qia+Xnm)xWzRkwiT(0Ufr$zE@r?1Xa zvJdn&73*5aQAmmw(d$KrOlsRpUJU0Vj4)q$clEn@O~|uIB{Gztm*gWA&d98h6YK;O zl4PjUh>Ma|r~K&k9-Uz4Bzt^OvgQ-+P|XA>g`|~F5FEdV55f6XvwnYilX6-fwYA~P zD;^Oi_t+D^?20bl>&Pe=TrkYsS3K$cjhUfo*zIUCqT#0GWS{SuYDaA2soR-Y(WPcfDshP3-WUjLvPpX)-#JS(t-B)%^^@ zxc$(5eC^4>(sxaVVc80s#i{59N>tI(Fxy-+a;d(7Uy=ah)mwbI3)^sA=6UdN$MlD5 zeT0~!lSa*rBlc81ADbgQ4S84zDDeGw5rM3Fl_MTO74OQ!w;SqO%mfb8oMO*a?p0l; zq>-y}Ht(j_6Sasu{hkoBsl7*zWIoy*%4=}tU-&UzlPO>+ma%SpA`7C5J!GQas_rBR=lHY4*<ikDA2-dyAtEcaBuXwoV1o3Ya!Fb}Gaa zTbXu!HJ-`pK)r}gI+c@KG5mR{D_l2I^O+dO`pJC9x#qsx72}Itg?q6pyGPatQDq83 zOy8LMET+Ovewf1zHLjyCD&;>DxoMT~WzgkEU$QXE-Z@(S<-L~NmjX%pjCPX4{-X;X zHyU(8q(M^RNDi(qtKM_HH_`b)xkYs+gHX0{=fy$tJO2&?RA}1w=amNLv$b5=L%-;< z%L|py`A6g)noWJF_I~qLTgCO;ASJt5ZKiPpgB}Zoiu+l_W8^^$i^B8LmC>D#PdtW& zs~I!QhI(x)_trJ@51m*i&CNB2<9o*FyPauyq8b8TRvSKGOsVoOAFT%(;t| zzt^QI2!}q>&+H9g0#{d|#qp;4LuDmnY30e4@+AH524dbzq2I-Y!_j%-qf0 zbtjTRhdHaC#n)kCHh1Vkv=4VzJb0Q;VBT_zcPCxxdiF-QjAy(yV~yuax#siUr-i|@;t~=MkfT*CO#_l4n74H95yj&Bg{6vw zY8<>Q8lS2pwg(oube?18XE}y)r6dlnI@)%H1@`h9a_f!w`yMvPwH>ant7h*_n+X_D z@LLQuaV_bSAiN0zWCi>4%&M(&YhN}Xhr`A~v)QV3_PS?i$a(CSSJk#cB&c5`7dR6=3XXFt3&?l|HnCEnZMc5MsX2d$0F-^$Nqe=G;PzT1;?|t#QNhfo~uL z&K(Ah*u^V>ck@21S=Br*>_#m=D@%SEqp3h-Ghm1~klLM#a;cnoZJ;G5Z+E^mtN40m z%iY>$lx?UdW2tOcxB+|i-UctFl)s{|Xe;Br&_~fH*wqt4yyG0CAK#KDVH`H8qR@a5pnyt7)m!EK+c?#&CBpsdb4B-JOE z=((?vnI0ubd|xUsu_D)#u{%U?ZE1Jq=r*=Dsd!Y40237K5r5|S(Sz@g^eeE%xmB3e z1tcaz6EtlIO}ut&&i*ddka6=Ml{vDbovit3LZ(VhM7uLsSe3g&mHR zm=LQgNXGpA3cEC`QBbdvBXZ#ZqOFgQ0bv~><0G<@7K?B9hO9I zv`C$L==v?f1trh)F^a~G)Hfx#dgV(1RXUeSeuhIW9=@{7?(@vSc^}^2D zKDao0(2lzu-}kBft9Q=QlV@%_)w})NBVQ|gn!Ld)z<(<1gYuWP4COD^c7sAwU57n_ z(r8hs9=oNZ3)v1t+wBLDb(ujJ4VcGj8qpkGfkLa)tdm)~gn`ZZXUbowle}B2{6)3b z->#<}_|CV4((T~ERCV>h2Zyok#5Nud87hr~EpnZ>vPcH<9k&jtHIVKuv)&jx@U*M; z#%DWFepbcg8}YjlG%kfY!uCtm;f}`BeP+{ki0$d_FW2S+5hMs-nHxq@bY9O)r&f3o z1I7;RMiNXZMg1;ZniVt1erv`(*8Zg<&L+?~_98*?;um8~i!^$3)^u4mP3@rE+2qK(QFA9{(=~p4PdxQ0v<&C_ZNh42J9~xDD%c3HCv^Jqrvuouizfca^ z?C9MmHG6uYOL2bqY^+Q=?q8~QmiMMJtJRB|x#kp`i{f3d1q#LFmMepkTo#ID>$6Il z_UYo!IP_RRM&qjqBCsjJD>xyIh`P8fdnq|T^*dMV%D&S~PNjHAm@19O%& z&kY&%4qO=ztnUqmtWzp6=y4QZbkEq|Nk)8bh zPB`g1i=HXT=Qejjfl`}OY~m-fY^oFckk)-U_HH}9){-@kNs|oGEXu|vxxM%2?G6J; z$eG&XNP0(~Gr!(oUu(zPQgWT8)ZOkTj94u!lzVL^u>axwK+5b;edK8iVuWwR6{VI5 z0(C65xafNry&i}54t9a2++EOaS99guR3CQr=+by~wYtaBgQ@qion}=dFII9=hnk2u zM((EkG;~+sVB#Eq$wjuCJ#=Hee6!ji-fY%)W8=ErZN#(v6hd!;Px?sH;zTyFXWV)j z%ztg_RrV8E3qBcxZYm1)#h~cbLlP#7#~K?2$X74Kak~7SgBf(lz1>Jn5@HXRjd_Tb-UYPxh?&t{@pZ&dz8k( z>NYdQu9czGnk|;d(N~gPa_13D-7%X!xvec9#{#q}t!y?glPFkP7c=UyXguTL2t7$p z){*VB&;~wS38kIjul{5fx=dqPGF0+vJEy_|pRHzry=Zlg(v%P)^8nLQ{0A9jfsuuM z{TV@Wh6^L;1`pW7W3F2VhhE;(Y?!x&n1$T(uLl@Yhc|LJoPKzsU9keNA*2H0!~Q5Z z^iIr`9KJ=25Vsfzww+ZXek0|jIK55ghc72eTn9Ev0c_L(EZR7oI_70*XbIi+6CpYa+gq-Swad+Yk?+2;11K)?os9HaKlfgE!6g6 zY+Jm8ft&qiZtQS5O>31?^#Q)uYt*^Jx8Ph{P%_e3G+APDSU06SKjXrAot4GzTj%0Q z>>90?|7`aUQUZ#|)9VZ^tTU2wn-EVpJtUYmGD&Gilt6+#dpQac|6XkABK1pXYrXx8Y zHF*y*b|I+x{e!qKpKkcdz5eU-4PRW`Rr3k%7I~V*7k5hkwDJk|b_m1j81MOd{!_#T z7r~YP7_mKq=Ro)d5u4$>ZqIi|Bm2P1|y3Qp4U#_4huasS#epGpLa%Dk?$>jg*9B zl9a+Tt?|QMFUY1S)DQ#}6R1i{OQ$Ku&-Z2)8-GuzxUIWmR30tmK!{7A;LBa6qgz>8 z(k?O|fY(TaMEdv+y`GQ2sN332$fjMRDshU{0JoS@-Q*qw}>PF}JpmCrT@PmcxRTk;#hwM!Fz8oe#gi&sJv~oxg3597z9R_A* zc)BZ8?=`kIHa1wFd^RzQkT|POhs9%92uP-wOSuvop?BN%=LbQZt2gyLHwV9eS6X|& z+0gH672|(XL_|w)k&WAVwvm)zNNgU=#3I+a*F0niBPQM{ zk6E$?6lBxvX$Z4UlTsvPz)Qc+;(D{3Ur@k?Kz*plev%onapoN8baJkZm*w_LpR?6x zDj^$(6QFMXRgexers6%7Kr==QBjq##skQUx1F3OqFo1!4vU)&1NKY{o7>M67QTY}a ze{VLvfB!xGcg)7X5&7KVt* zUtdWErD6n-Vm33aQhDCtUS)w|yN&w{0{CoO8G(Z6rPK*QO{+z-rSve*H^!`bP#2O{ zJt0FoQ)|W|E1tl84l+M+dHr(S{1z9)0Yj*3)+w`{gC%l=TNnnIKuP>Sh3XKB z`w`kpFa->}mslQ8b$q-Z`0RY_Wx6jxYJ?HH@05pdQ+%nR3KJkpjBrdPh%jfHzGKrK zCp1&rB74nSV9O57SrIj1#4o6=UV8E6)k#<-NDa5A4V2%lsmlc;#>f+UG-HTFCPqZm)AA#sq)e7( z1rr;WEtXK^s3KeQmG=_rHPo$=l9J2?E|8ORYL^v4p_Y=@$fVLhyMwc;0rKo)Q#zoO zE-))fb3xu5xABBdWmURtK{CG|HjJavhbw7KrwIuN-kzgzUYn93e%ytEvR`wMx{*Ms zTCR?^{iW5_RnRU1rA;Hxz+!6ri9E^fS3Np^ET*y%>@04EC&MPwGnCWLQSwMsq5sTIILePjhIyzbJ8zb>srgR@ea{SQ(+N6`-hYy#6eUwz6{wRt5 z`gQET%fUOzd-*?HVNALTl#erRA$&Qv zZy88|OAP~<0Jm(K@a6{1BJ=2@WfUPTjgU&VX7MsqrqnOq89jqgZ!Tz8cR_)aH(urNsR{XJoC5K9I5dLI0LD+A+7 z)c=w3f*duNTb$`T{^|e5M@aJj?1ef5dqQc*oEE zyR-3MfBerplfPek{L_Sa&t9H|xb+_Y%Jj z0DLuY1_5=GeBPph0zW^$?y`;Lk@6aCsH66kHRfoHLLGU;byn28|K|EIV0GsB3s#&S zcu>bEI&=DT$3zbh4uzahF6k)C(kxb7cwAtl-~<%|Pv&y3>RkV|zvwq`Wx^8@bn*>k z24q}ZTv~=YIvzS{0~WC(WIZK4XT3DY$AJVyt<_*`4E7Phkr{TrclRcgy03CSlYsxr zg|qg;{sbaV!~6~q0_gAmIn)-TmaV}DdLQ5;x$_t9d!GyMW(p5&Oj`r<%v-qbDgP>PW?Y3BZ=dXEO#M zGIZqgqH}0S$fA)a{@3{U_@J3+RY_?nrIMUZ;}DB-YNJ`xowdQvPHrKgf#hHSMtT$j54I-*L8;e= zb}RN#>XI(?=OtBGj8>G2k#jiAGuX;DbS6o)EP%7uXy)RwH&l#_ZG8ufO>DmTbatV_ z0-N$JMSkZ!8|ue2s2<+!n`0Nj%g>@-9IMtHI=eDlS^$tCxTchpl%Y%tA)Eji*?=fc z)oCIt?W*xJ!Mh&RT(VBE53DE!p=~^@f(o2QNQZep>Mm2ZH-|-ImD}D`r3Em{;l3rbKM$oXfu`_ag@<65JPp&iQFn2wI*?xvr1!oY9smY?hp&|UDS00(^aE9CHqOGt^k7$(pkq5%80 z#R}*angqKSR5aO8T)aP1vVVyeht&P(JjFg7>2bK8NG<_^L~>$3AudD~DcRn3TzdOi zNo?nJ3K(UeVdTo+ZEjT`xwS@d&&|!T|ADg_VC(*CWBGr@9R8zql;8N-IeD-(Cg5@K zb7N)9ly-+p zPgEK12exn~IW^VU#RV$td4OR9i{i4i-0On27{YXS+thgCgH^ntS(FtSsRr(S1eO

-F7#EPYwZ37OEsO2BKTF zGZPC!z@?np3euRXRZ6eS4vc`j+xXWvMf>ehTsPL8j8h2l2viRRKr;dT2ypwiy1Je1 zZRv+#BVJWCpbH8#E1m70J$t67_aP~1vN@cs7rg&p&wdBbU607Us;fN!hs0s0fwb8S?3WA!I^4M`0Tr;J88rvN4ix=ts&T1Bv{q&|+Z2!zdDj z9TY{~%}9KCkJqON=<3E%tG_{Hx;I>gf%KWH1iEi_c{%4Cg2Fp8BaB_$dTV*4J}(vY zAVwoRE>5e$aTNf0Gh}ynccx0lr;{Xg=^GL0%*GW3#d%M`LWP(ExNlyyk>$4U4rPvM zChimvfMbr#*uX$KtBG5WrAj&aA)sSfZVz{HbH6)u>g-i%xs77jgt~dp-Z<}LC@zAh zdm8aQ0vC$Xa^Je8vcluK{Rd_(Pig7#brwd9q36%!r|}u^LM6u9Ie_ zDu-D%E*8c&K>o(YDA+G1b!mP|h+AzRt$+JBY|Lr6#41(*8L=NZ*{~v@r>zfVar?^KJ zbDSkPeHt1kegYK9s2L|{^$Bz*Pt@)t({K7Sw9jt7C)lUi`q$(f zb&sM&h*}1x7|uxBLYUS^M;WAF3mAR`r#35b66`IW)}9PyUS8hb3p|OwGMA|$hvJ1D zKn|0+yrGi-fWXHKj7{~>`BK6t|#8382nWbBH6lO&_4NGs}W3n*H1 z7)v+gX=-ZjL=gXTWOqInTqz+j-R7y>i;1U{o`xP<5*el5?n)bOJ<;}^{1h1xNg7bL zxD@l+y7ejWp%X4aze~7zXPQzX&|qlOZl;Ch^@y4FcuhzO!2Uuy2#b8jE8`v{g z0T!JKNATm^ccw2_me-A=gI%1R&t19s0UAaOTOzP)YvB0@EkmOa&tbvcP;VuT*bYdw z6prJ7E(jaq#683sh}FTHrLS#a2Nu(*G3Z+Tc^)*sa6PoL+=Nyagy78BSK*Zk@qDnS;ptv=geQkf8d!>O@ZqC9VNUhs zrbqn<>!_WKKmBSKuCM$<5^&zr(BdA`;r{w}nBVx@cB~D=(Q=B4oPaadc`#*1(TKLC z8xJ^E_1uqfB>{)4bF4i_~FjS^2NnPT;>&N*8ACRf7{B*YI#^F6q39;bT}J=efj>VB>_!SjVAW_@#6=? z7hqNuV}1K8oNNLvJ8cT2S1U}2VmIv z`t@s^UiO1PwX*NmcrC;bNN!M|1ULr;f1&9F`=rOJ3$U^SNiFUp1~;?IcN9c0Z;J)3 zH3=fn#Grw}^V6BAgUn7>R~HX2FN9=kkf56pyoT#z=r976S&)=QiDjqy9_cI{dNSqc zvhk#63SX}vl)Cxl#V1W#JpA3S5~#kc@Oe)Na&OI{XR7vdrZrlhCko>RNGe5n?D<6f z<+*`^w8+TF9jPpZpP<<4k0hmpJF$BBY(he%h`eVMTx6_9QcAq@>ih{@*dQj7<|*>0 z)6L-phb}%Hkmg;Po}Yid1gerz&qC;W!^6gVI5nAbl6@RR1d`CO)-V@8*^qY!*Effda`|D@b4yFv;zsm*d?1CZ;O#Z8a-XG$^mKAz0JYUiM~Dv& z{vW7cq;G)23YT*Ngr5K!xG0Wp%U3HWc%D2y9j?GBN%(^E zqLj5Nd3I{c_8{OjkW;3!7CG445;#67>i+^+kk_d!wOgHs{YUt|&M z<#pfLxvWCW z+lO@x)7Sq)MQ5($cX`H4p=$$jDd6?`2AU`bzkn3$UmE!VnZNyFe`kr6J|ugXnVF%< zvXZ+VN;S~=$flZ=?p|aJE^lj$d>#ZZxR$v@)#us4MTHlRUuN0~dJdo-a_;o$pgyQr zh2m{|CE+ru{GE80=ePiA-``~t?wdC|IyiXlirc|Kty9K^O!=OhRWUBo)kLW6*c|j?j7Ln%tpnZ*LDqE36pGl_5IdZQH(h5!AtyD{MDE zH9r2)O?8zly1743--Z-%%IhPD3@(%rupEYFMMj3`^G=(K>Xf*a*HJ?#1>q8K(bLcg zMk!AoD~14&8IX#rrgFk}ahpQb9R&s+3RtqAU)b8VHa12<3sumy`x{iK{IDY_btNk3 z=;A#}Y?}`>qYIvk@nB$NOaYHAWF6A)B$qiWK;LASPh1xomCY4+!V~fLjs=+(S91+( zo^uHBOS_eX!N?tSeFSvqed~nJ((iqAy>JKx9#T(=++`;=)B2_+nQHu&de0PXdaW|s zj3s=zy+&wCU@vN}t(}BG5drQv=os*^&;h=I{j@0m(&IA8sK_X!Fj{fybpq7uD)~oJ zBX^&NqPjSo`Phe1h~2pi-d ze%C>Y?L*6xIhUBW`uh=+vBQ_vam9C>@2!5V+;t#cnzd`R$Z|vjUVPKj@a7e`zDP^1I7>p#%Lu-OBi~Y80A4^Cu+BAeE9~_r0|>O9TD6a2~oSn6lB^WK@Kz z4<6J5&`!noTm`uEoo(<(vEBMOO#oRhU8rwh^{c$%dpB{v9Ep>Y{X-d`W%j(P6=K}~JGq?iRK zf@A3;c>?#yEhtT@QV^lt~0B?j1!w0Ok)OKaan#Gn)tj;z_@gxZ%;tg7K3LyK!yqGsp#0_KaTQQY$sJ zIL04}Wt)qco}PHn+z0oVdIfR5Yz@?J@%#wz67JN@?Pqg>RP>0^;?~sh6sW(^?d#kJVS^xYcjpk6lDETPx<7q_dlw+he zndEefC*zPX|L#S=CXT`UoNyeNkAp05-*JcjVmtuFz#YZW8^@GBjwx}<6?ywUM#S!F z%QZEMcdN$gsJJiA^=2#B4UdC_r=yj-n5^8|?&~4>N77a6__e8Qb1h$&&l)YEttcG? zD0?U4geQ#>c&yj7fi9_Ip1PQ@qUk>qvEfh>KtBTJhTJ^4WKR zVuw?!tu3MPz=U{WTQN9{ewOgR#OQH-1eoi@2;>$W1J{HlZk z^m@~oo+*6;K^?pAWA@EDA1_(B-s}1w#>z+VB9v@uLYx=+80*V8#|V7AAO;&gAL;Kt?~3$ylLl+5aA1tr+s=TZ(8aCHwlL z=+_Cb*wxi>?tI+kU%Eo(HCM19jU_j4O#&L3Eu8W4k!htzTaS3Jz@N(GnypJcHhHGq z?FK#l&(TRuz|#Z$EKkwX(@ojcvn`xq{H&|~q~RKy6b@fcApM_&uXErpn4Uwtx4(+f zmXEyk&?a+EkF%UK^GI5p#wNEfF38`^Cu8|#G~l*&LM8~x2upGlgPKfPh<}P*qC3;% zyH3+I?@|d99#&w@VS@j^XUaqJaQnD97T>JixPWoD(Xa7mKlEHI4o@QEf_BedHIE~_ zSQ~YM3>$=DWo1(Q+R-`5yuzl}RLBtET7I}!t>}jRnY|V6$!9R7oAzq3y3?lT3i4nn zGuLi^H`Z=G;s!y4fs}DyDqHno{bqDNkaX5j^3pDq2GiBLraR8&uJqM~TJ)|l|GzTe zW4FPjX3IU5#nd*`=vR|;*I~uOA^dZ9O)vBHcYPo3zk+(Dc(Nr8`bK`*7~Kjz62Xz9QL;Q3-4R&xAG;>i;&OJy{$80Nqy+7ax>^ zxD(lnsi-MDVm9Bs{7b(V+qsUJ0d3NKckN;0j`*_)%eoO+*Ksg+I!`|Ho+~Z* z3saP#o_wqK6Azc~Z+IF~t1{@hi6LJ?rLa(f3{^M)6zhESK4sCl!Ip2Am{j_$ycv^cxUgyZ>}J|ChAwg zTXS$>%&GqJA-syYox#sJb0wFY%_FM#!l_QMn<9 zPBH!vQ%<;JHx1o2#E8BN%ILlHn8o&XqTEA)NE7B920hcr(+!%{68+BAuxjSVd@@QUiUPap?hO_Pe;PQHsJpJr3(%%=($W@x7ksD zUoi?>S^6+5ifBG#_m)24ctN+#!S{_$Bos1SqI%KtW`oRCJY3%H!q(XOwgFxCTfomm zc&3>6G!@EIypy<7Jd-Ve9I=`->KU)mhrWlb1QyAfkb2?rTiJab7Hxyr{W9xQY`I$W z^74vHhjRAmpG*1J=vlCSLH?JEjL}#D=^LE{5gRd=L#+6?v6bZlr4C|x<4g{r{@5s_ zb?(s^)ps|4OY8G~->U|Ooun|v#z$pMl|x?^W4DIrZc3kC_<<~m3*uijNUaON0L+{i zv8q=!%kwGZu@rTi&O@#h?nXzkvrI@F>lqbEdux98ae?Xz=?mQ=>-Vck_{kBuYi469 z-iwor1FpclziClxv+=BZTDbm{T=!~8MV2mK0e`a9((q*N+3Uont6kyHQ`1@S&fscp z-UaQLLdO(4;hmDbjx8+IShIijJ0?aK#1uKUP3T`W$HDbc-y@s!uTrqf2K(#kEz zlS|xZb>A9kal6QSk={I%m;rbRx`}2dyT!yK4+>BE;d(RBrqQi+$)Q^jAN`-t?f5?s zE`26_+)OEIi=~^&@RkPHw-{wiaF;}40l|5v}?>R(jwzo_7UQNjPBg8xMY n|BDL#7ZviXcT=5F|*E-a?NcAiZ}35(R0}J7N?9h_uj~fD!4v zi1bJ^K?>?Zo@lBiOMq^9f^l|VL zw_2I~zD^RWj>gX$JAuz{>Gi#p^6XFFe|s-_KM5F`=s}$a4fwq90(t>>Ux%ER$(YXH z|185v{!jhy@*i=cf0X|z{?GD%-1OfyfRp%&T>9(sRY{k(Yum2v10NnfieX>&D|uUZ z`NI1lL+MAXh)L`r7J-oVUNIjs93b3){LFxVuW`jlVxI5R^r6;_FFQW7BBCwwpeu^B zAAWX1E9lmH`Wi>h1bkf#nIX@gx?20~ogEkq#wPYmyVyirSh)1(Pbf}W-eKh9la{dW zrIu%CQqH$`pD60+=tLZnrHr4#TdmS%eAjbgK_-m>c*9~7A@h3utinR0QZ?OuOmJ|H z@*Cw?4k5e2$@>+^B2TO8{>|BzjWVuVx7yIrBqStNjz3C(R_p8Qr>CbYxN^CuHCEDY zAz|VE$;rvy-rj2VRinW!3dXf{n3H~nWfr%_tL!ZuePytNZi7vY!ppx@;pPoK24pG5 z-0$rsT?C#^R)5U&!u1dn6ORB&ynp`yGn;H=;lI_&oK=Z`hGm9zISp* zeKH&DGCq%C2O3lf0$mz=gWc%%_V#-Aw>Ua=a6=|ZA$<&#kH5)yKIda}DKPa?L!?4Tor>{&~ zA&{|Jck5yYzJwvqY*3cBf$v@Gt*tHHntp`5t7(AkYmCzs>zXO#cmilCjEb!iKU9|J za|L~I{nVKvCHq%~I4H8~ql+@qXMbrhr{{CJ_{D(gUXvS>5*pJ;Hqm>wJy)g7Oy5bm zTC{{w;V;}Ykte2fiCS8ERv=qeRz}Duvu^j9YyYWX{ zU!>QEh5lc?nVF0H{QPP@agoKP%RfpHhE)uz%ThXszWPrxX(CarLk+$g%5PXzh1KeP zhlo@C4|*j;e7b=ZMA+YmgnFoXO>e+xVX4Jc{iyM*hO+ zm+U@qc0}vm%*86fq;a3t{o0Vp;Iud&6J4mS&1y938Koq+ppy$#fVJpQyCo8GOWfO~$9k?RqgMoTMgH z;vM_ds~=14C##F`ejasXh87kUy&2-zp7eQW2{Q3!UAYYSql#L81i|YPDK+4u@apFk zEW?}I+W|*wHQ!&GB3jo0&w9hB_yE;qj$#o1QFQJ{-uHB4;5IeqpD+0x>DYsiZ?@@9 znQxEX7%SsGA9=BJp)-*)=wKFwLPbYM8&}w}%!CUvqv?;2j{z^K0M>a?QIRLMPsprx z^JE`mw4i1}yoPV9tQ2ceU$<2?#wxi>s}}3!YZ55)0oM={^Xf?xX^-VhZBr-sQ?ZK0 zO^>{0>$7TTLwh7HG6|V|G!$CnyeyL-vMeiP@wrwbQ)0+E z5tgTgurM<#0>z&PQFcMeqRqi|EXsm z5ULmJJv=>~B?p0^OjHuvYGprOzPLJG;s5QTFQN+)!9=O z_Jy0N_UJs>ZHAKoHUAR&_Zpzti=8XX%)T3q`!n-IE-o$+5fMpOeL0C}1&MewA9462 zd$5k(DMIq62i9GY4K(Ln+*eXjQAy@EVCesWIXm6&&6I2o!1LR*PLmQ5QxdVa4iD>X zg*%U-W>bVrd3pY5Kz@(?e$18NY?Oz~5UwmPD z^Uj?TpS1}B_uhxs946IHEUjPvXyg(Fq?PYQ#(M8c@hjhf{B(`vU;m`m*4Ea0;~XsI zDX6G&AN(2Qfz;+oVQCk)&lu_n0|{V@^%qd8Am?vW{mlnjK_SPfBuo&u^iao^t2A$= zd~a+xfZS`qN+YW4Af2rc=n3->t(`5Fo_N-3>yX(pEbN*EJ1Vl4@*#(}$9|>@-;Hnz zO5*PDU0;6&x>xEHDV4q-+sp+Z^s?LJ#8GsD@atVtNFQzp8I!{G^A~M2l#Yxk1S$en zp(TccuS1A|G8|0%n2!FBNXkxY_>cDHd$3!J+IL8-V1guplCjXNNSP}^TqdR#<+7Vf z`j3?jW6Y*&&L(s(*i~8>DjknUy$+Vvv2`FDEe3Dqe(l#{94`s>bL{Lo`1IdV2tDb9OOW@BlqBa)%WSq9!( za`DG=y(e}`C>Nn7klx2&4%H&?=MVis>UET(n6HsM?xp5TRr*p~kT4{zAYh;(j2i0H zcgf9IrN8I&aP?3e^ZZAXV{9sEb4pS|7}Q{_j<%>v0{lz}GU>bC+1I!|{~?D_Hlykc zI@bEC+t=avGP<4*J&~#XUMO(AvPj-2EiMuB+*FD(#ZqZ7egQ7({f$ms=cWNm+X^sV z_jh9T*rl{FL+5wL&(T(ETFVcGA(chyHK6ix(+%b(3Nvfm1>2*g7HbpssBnqNVOqTz z{4llszE}P%ce}C@Vn`AnG@E;9d1vtK_(+KpQLr&CXQXD*7EN=ESU~y5E}z z?kzt8me_(4q96y8cJTD5Hm` zC$sISZa#Y}^N3VE_(STXI@Ra}CH(WM=_7qQH9>{3$MGPBL`>@El;?Ev=d9-Y_^CzM(4v!;FsZ-Lx(rE$6e-`-<(bq| zq}S>|pvq$`Zr;?&eDV&{3XXF^cRToYWcDz{(9Sju594K7XxaA|BO1or*o9abljG@7 zwwF&umPu;v!EG{=6aoy}n1ceFHm!W@!+37fxHdMaCb?VS3DH)l;qQy|#r#cnIc>+}25L;~dj0S~6fKvQZac*IG(W)=EKVL;#MGhE~s9P&1YMD7OZLVWY-w<57_YFc( zZaQ@Qo#QY0<4NDOtrqk{<~uxH{@`WJchbd_hTZbj<`qw@t#rG7t;tNw$+lKXI8EIQ zb~z66jJiGH(1+}Z2#DOw8AqHhkN0I(J2fYz9b1pZ7Yy5e6z(d4nrmtvBIM6x4X?x1cIjaLYj8;*SXGE-jNu{_&2kPRfZY zBX^Z_y;or;-&3u#;MwoZAgGB|RFRx1uf%4D;k?T`86kn7JJBkQP4=3P2ZhHEXP&gi zk|yttyOM_sh>@lEC66WFPpJ+vd^-4~@Jaq;I+xf@8(Kj=t*sle*G0fl3)&0HlVZZv zOW0`m>5-5Yaq@uhQ+J#8iWv=X* zVZj8wVQ^T?vDa}Pv3ehm^fsR0>UtSQ;5usu|mKB8uq?=(8AnD>-3DzPZ+ zi%>*;yCydQVnlgV%`@9bfO$D?-5yF|CV7s3oG+hm^K>VtIE_zHgi+O}z2wXTZes}( z>MFO#Q9EKHi^RLNYO-XoXd%o_`;xo{b2L*>Evp-&A!=a zL#Mx<$Bw0x-~8t}(*|KiE(lr570S`XwT3RmvG`5LjpmxR90qyDIhiy$tO-$$*1yRFLQ0;U?J^BsNTH!h zLqTTW))q@rWX9d?zJ=w^wi%PGcjdWXF!DkzjM|76EnWIjtz0RF+D=c+ncyZ2JreZoF^VBy+pLcYEh_|d!BMvyJC&Bbxo|Q5Q^}8#?sc`7P$408R3&_+{>NAK^=N1v9GWHTZ4%sR zp1DSt>&~S>>#pY1O;j##ICihd;qRa9u*64nB$iDeCSJs?bCq7+IemS0d-O_hJ=7SX zp2=la@g%3o4iR$ZCnpqMLVLBA(gq}cPQ{m@W zYT!7!8^Yd&y8UWxNoLe@9EpIZQy8BdgQZl`qCq+O0HL8+D|~% zt$5F{R;3eg;~~bvd*vR~oN{;QD}v_pEa`>Wh;c_k3R3=ObbPY(LD@YIS1D2mfESl( zYUpU+kq4*2c+NsO)pA$ji+LiX4iRk(a@CCshqO^D5w{@a%ZX{(QR9cD=7F7!`yd(N zIyUw~xTS|H9v6iTG8C?Bw@GSpTBp_oYDt!2{ zpJOC8@!GrPAr0z%j(eKSf-QBrL!!r^gM$L<{_&H?FMrR%nl}q*_^#`0LfaE4Cn6tdqm~}dmh2xs z&hX(peu{j)Rm#h21nV@{{|VJaZ=%?^2$pK z8+Y$fk!pa32fRxct$fg59X&&V?MdMrrKH7?7yRSw@fZ$aGyodj1Un6qWcG`7P`h#p zobuIO7m_^lQbx;br35*n*VRnXqB5zo{COPB7GVh^bHiX~9|*0WGKwsl-D+u5r(TJ^ z@8FC}|9N`i&*0>FhO-=wi-5b38ASnG;=$&mAd^>t?tbL%-MZ z=mgIAq>%9Yh(yK&Z#_Rc{XQdwKyN^Z74H7)==A#-6$;(zxcz%rI5aZu0g4Rhi;E0S zCwLL~A&%lxgrC<%UH|`xoF7E~YnePU^8c!v|67Dl@wX!X(1upXU){mpWpBTetFzN-bgmNU=WM zkcF=|%bWHZY9~4eA7m+-%F1D-r5i74KnLS?2u3C*=`HUY5JgNA3_$4`L5pttWGVkI z=#B9LXn{_S@^tXo2|!j^bZ4m#W)>VBi+2E+t z&DZh}$@ecwX@X`$nMb55HSY5AegvR4z$`ofnz6qx%+D_>#&CFeIAkIDY~Z~~l>@;3 z&}j7Ncw*Ew2`9)20iUaY0EnMYt#LV#_PQ3t5e)zhV*pYk4fBx5`;GU3ogLFuC|un0 z*JB!*SQP56dtO{zTmXKh*u24TQp|qn=3vI==H|$VegX5)-mkkDOct4+AGPI#X)-^$Cn;&F_ z@L}j|d8}+cp8Jv6{8xbjC||(afQEu4I|0IQb)&BTMdLHmu(Y(RE0=1kp{mpWLfWuG z0BM)IEp{cdiaQipHd~ll{!O%PBOFvNUFS4XjnWX=^qfV(%XpKn~M@&$qc*9GPkx?titpq zLi_SSh8$0g^W?8zzw+J)AV4q^2lQB(b(O|O(@}{bY4-`a0FEb7BSXx%F`!ByIhco0 z5FD%LD#>`_RMAvwT$)5DSX)S}?(k+LHLz3%m}h>2PaOyVjHPM-7c1q00UtJ|0B%)m zJMOZAX>LB(Tzt#55uakb*L3)c&AvAm)a6lX(Wo4KRmYZ^id@YU)8Om6Y;$;U;575u zn0RnNTj|lGZUC;A5pHaMS(<5Vc-P3Mm$#I;Kr5vkFZjZfJZ1WV30`lz>1aLib*Z4R zak*0EYFRr6z#=eBC#0OuH%(=q5U*?z_l@Rc|Gi(uI&Cnqev*)oCg9&AS|<_y^+&BU0u??>r-DtDKWukFz3V(66b9>A)($qo_zp! zpI;0~91;jCnG6NVd=S8`_osakuV25$Cq9VJyMZ1U7!a{;W0wyyQ&JNheXnq7yyDw0 zp8eOHK@bi`78Vu&z>9ufa4|ylB>dL)Xva|gd~F&w_&AUAGinobhV!*Jfnaockx2`J zWfJs`)kARU?^ZCL*f%r|gOvyUM~R@pXvAW!rc-$ngr5)$cQT<{>ym1-4 z;sXC0wXe`|iW0dgqxnTRy_ptx{aIHqIV(fCQG4fhGXhfM$5$$)wsYSi-mpmq{R|;R z_EJ$&<~;t7vgV(}`rk^p|AVIgo~!?ZH18&KDu02~1>+hTVFBb_B2}=aQi&oI{=We0 C?MYbx From 9b35835f0df7aafd9963e80921f1c4e4bb53f423 Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Tue, 18 Dec 2018 15:02:03 -0800 Subject: [PATCH 07/14] Capacity planning: unify storage sections --- .../installation/capacity-planning.md | 75 ++++++++----------- 1 file changed, 32 insertions(+), 43 deletions(-) diff --git a/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md b/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md index 050dc27780..eae5b89174 100644 --- a/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md +++ b/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md @@ -97,33 +97,6 @@ advisory_delete=0 (Adjust the `path` to the directory where you want to keep the ledger store on disk. Adjust the `online_delete` and `advisory_delete` settings as desired for your configuration.) -### Historical Data - -The amount of historical data that a `rippled` server keeps available locally is a major contributor to the amount of disk space `rippled` requires. The server does not need to store more than the most recent 256 ledger versions to follow the consensus process and report the complete state of the ledger, but you can only query your server for transactions that executed in ledger versions it has stored locally. You can control how much data you keep with [online deletion](online-deletion.html); the default config file has the server keep the latest 2000 ledger versions. Without online deletion, the server's disk requirements grow without bounds. - -The following table approximates the requirements for different amounts of history, at the time of writing (2018-12-13): - -| Real Time Amount | Number of Ledger Versions | Disk Space Required (RocksDB) | Disk Space Required (NuDB) | -|:-----------------|:--------------------------|:------------------------------|:--| -| 1 day | 25,000 | 8 GB | 12 GB | -| 14 days | 350,000 | 112 GB | 168 GB | -| 30 days | 750,000 | 240 GB | 360 GB | -| 90 days | 2,250,000 | 720 GB | 1 TB | -| 1 year | 10,000,000 | 3 TB | 4.5 TB | -| 2 years | 20,000,000 | 6 TB | 9 TB | -| Full history (through 2018) | 43,000,000+ | (Not recommended) | ~9 TB | - -These numbers are estimates. They depend on several factors, most importantly the volume of transactions in the network. As transaction volume increases, each ledger version stores more unique data. - -The `online_delete` setting tells the server how many ledger versions to keep after deleting old history. You should plan for enough disk space to store twice that many ledger versions at maximum (right before online deletion runs). - -For instructions on how to change the amount of history you keep, see [Configure Online Deletion](configure-online-deletion.html). - - -#### History Sharding - -`rippled` offers a history sharding feature that allows you to store a randomized range of ledgers in a separate shard store. You can use the `[shard_db]` stanza to configure the shard store to use a different type of key-value store than the one you defined for the ledger store using the `[node_db]` stanza. For more information about how to use this feature, see [History Sharding](history-sharding.html). - ### Log Level The example `rippled-example.cfg` file sets the logging verbosity to `warning` in the `[rpc_startup]` stanza. This setting greatly reduces disk space and I/O requirements over more verbose logging. However, more verbose logging provides increased visibility for troubleshooting. @@ -131,7 +104,6 @@ The example `rippled-example.cfg` file sets the logging verbosity to `warning` i **Caution:** If you omit the `log_level` command from the `[rpc_startup]` stanza, `rippled` writes logs to disk at the `debug` level and outputs `warning` level logs to the console. `debug` level logging requires several more GB of disk space per day than `warning` level, depending on transaction volumes and client activity. - ## Network and Hardware Each `rippled` server in the XRP Ledger network performs all of the transaction processing work of the network. Therefore, the baseline hardware for production `rippled` servers should be similar to that used in Ripple's [performance testing](https://ripple.com/dev-blog/demonstrably-scalable-blockchain/). @@ -145,7 +117,8 @@ For best performance in enterprise production environments, Ripple recommends ru - Operating System: Ubuntu 16.04+ - CPU: Intel Xeon 3+ GHz processor with 4 cores and hyperthreading enabled -- Disk: SSD (7000+ writes/second, 10,000+ reads/second) +- Disk speed: SSD (7000+ writes/second, 10,000+ reads/second) +- Disk space: Varies. At least 50 GB recommended. - RAM: 32GB - Network: Enterprise data center network with a gigabit network interface on the host @@ -153,25 +126,41 @@ For best performance in enterprise production environments, Ripple recommends ru You'll get the best performance on bare metal, but virtual machines can perform nearly as well as long as the host hardware has high enough specs. -#### Storage +#### Disk Speed -Here are some estimated `rippled` storage requirements: - -- RocksDB stores around 8GB per day -- NuDB stores around 12GB per day - -The amount of data stored per day changes with activity in the network. - -You should provision extra storage capacity to prepare for future growth. At the time of writing (2018-10-29), a `rippled` server storing the full history of the XRP Ledger required 8.4TB. - - - - -SSD storage should support several thousand of both read and write IOPS. Ripple engineers observed the following maximum reads and writes per second: +Ripple _strongly recommends_ using a high-grade solid state disk drive (SSD) with low-latency random reads and high throughput. Ripple engineers have observed the following maximum reads and writes per second: - Over 10,000 reads per second (in heavily-used public server clusters) - Over 7,000 writes per second (in dedicated performance testing) +#### Disk Space + +The amount of disk space `rippled` requires depend on how much [ledger history](ledger-history.html) you plan to keep available locally. A `rippled` server does not need to store more than the most recent 256 ledger versions to follow the consensus process and report the complete state of the ledger, but you can only query your server for transactions that executed in ledger versions it has stored locally. + +You can control how much data you keep with [online deletion](online-deletion.html); the default config file has the server keep the latest 2000 ledger versions. Without online deletion, the server's disk requirements grow without bounds. + +The following table approximates the requirements for different amounts of history, at the time of writing (2018-12-13): + +| Real Time Amount | Number of Ledger Versions | Disk Space Required (RocksDB) | Disk Space Required (NuDB) | +|:-----------------|:--------------------------|:------------------------------|:--| +| 2 hours | 2,000 | ***(TBD)*** | ***(TBD)*** | +| 1 day | 25,000 | 8 GB | 12 GB | +| 14 days | 350,000 | 112 GB | 168 GB | +| 30 days | 750,000 | 240 GB | 360 GB | +| 90 days | 2,250,000 | 720 GB | 1 TB | +| 1 year | 10,000,000 | 3 TB | 4.5 TB | +| 2 years | 20,000,000 | 6 TB | 9 TB | +| Full history (through 2018) | 43,000,000+ | (Not recommended) | ~9 TB | + +These numbers are estimates. They depend on several factors, most importantly the volume of transactions in the network. As transaction volume increases, each ledger version stores more unique data. You should provision extra storage capacity to prepare for future growth. + +The `online_delete` setting tells the server how many ledger versions to keep after deleting old history. You should plan for enough disk space to store twice that many ledger versions at maximum (right before online deletion runs). + +For instructions on how to change the amount of history you keep, see [Configure Online Deletion](configure-online-deletion.html). + +If you want to contribute to storing ledger history but you do not have enough disk space to store full history, you can use the [History Sharding](history-sharding.html) feature to store a randomized range of ledgers in a separate shard store. History sharding is configured in the `[shard_db]` stanza, and it can use a different type of key-value store than the one you defined for the ledger store using the `[node_db]` stanza. + + ##### Amazon Web Services Amazon Web Services (AWS) is a popular virtualized hosting environment. You can run `rippled` in AWS, but Ripple does not recommend using Elastic Block Storage (EBS). Elastic Block Storage's maximum number of IOPS (5,000) is insufficient for `rippled`'s heaviest loads, despite being very expensive. From 81e9d5486750f975d2df50b2ae71a0a0c50dceed Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Tue, 18 Dec 2018 15:02:31 -0800 Subject: [PATCH 08/14] Add 'empty' case to server_info/state ref --- .../public-rippled-methods/server-info-methods/server_info.md | 2 +- .../public-rippled-methods/server-info-methods/server_state.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/content/references/rippled-api/public-rippled-methods/server-info-methods/server_info.md b/content/references/rippled-api/public-rippled-methods/server-info-methods/server_info.md index 3a681634fd..35d47456ff 100644 --- a/content/references/rippled-api/public-rippled-methods/server-info-methods/server_info.md +++ b/content/references/rippled-api/public-rippled-methods/server-info-methods/server_info.md @@ -283,7 +283,7 @@ The `info` object may have some arrangement of the following fields: | `amendment_blocked` | Boolean | _(May be omitted)_ If `true`, this server is [amendment blocked](amendments.html#amendment-blocked). If the server is not amendment blocked, the response omits this field. [New in: rippled 0.80.0][] | | `build_version` | String | The version number of the running `rippled` version. | | `closed_ledger` | Object | (May be omitted) Information on the most recently closed ledger that has not been validated by consensus. If the most recently validated ledger is available, the response omits this field and includes `validated_ledger` instead. The member fields are the same as the `validated_ledger` field. | -| `complete_ledgers` | String | Range expression indicating the sequence numbers of the ledger versions the local rippled has in its database. This may be a disjoint sequence, for example `24900901-24900984,24901116-24901158`. | +| `complete_ledgers` | String | Range expression indicating the sequence numbers of the ledger versions the local rippled has in its database. This may be a disjoint sequence, for example `24900901-24900984,24901116-24901158`. If the server does not have any complete ledgers (for example, it just started syncing with the network), this is the string `empty`. | | `hostid` | String | On an admin request, returns the hostname of the server running the `rippled` instance; otherwise, returns a unique four letter word. | | `io_latency_ms` | Number | Amount of time spent waiting for I/O operations, in milliseconds. If this number is not very, very low, then the `rippled` server is probably having serious load issues. | | `last_close` | Object | Information about the last time the server closed a ledger, including the amount of time it took to reach a consensus and the number of trusted validators participating. | diff --git a/content/references/rippled-api/public-rippled-methods/server-info-methods/server_state.md b/content/references/rippled-api/public-rippled-methods/server-info-methods/server_state.md index 4fd9947433..37403a1ecc 100644 --- a/content/references/rippled-api/public-rippled-methods/server-info-methods/server_state.md +++ b/content/references/rippled-api/public-rippled-methods/server-info-methods/server_state.md @@ -252,7 +252,7 @@ The `state` object may have some arrangement of the following fields: |:---------------------------------|:-----------------|:-----------------------| | `amendment_blocked` | Boolean | _(May be omitted)_ If `true`, this server is [amendment blocked](amendments.html#amendment-blocked). If the server is not amendment blocked, the response omits this field. [New in: rippled 0.80.0][] | | `build_version` | String | The version number of the running `rippled` version. | -| `complete_ledgers` | String | Range expression indicating the sequence numbers of the ledger versions the local `rippled` has in its database. It is possible to be a disjoint sequence, e.g. "2500-5000,32570-7695432". | +| `complete_ledgers` | String | Range expression indicating the sequence numbers of the ledger versions the local `rippled` has in its database. It is possible to be a disjoint sequence, e.g. "2500-5000,32570-7695432". If the server does not have any complete ledgers (for example, it just started syncing with the network), this is the string `empty`. | | `closed_ledger` | Object | (May be omitted) Information on the most recently closed ledger that has not been validated by consensus. If the most recently validated ledger is available, the response omits this field and includes `validated_ledger` instead. The member fields are the same as the `validated_ledger` field. | | `io_latency_ms` | Number | Amount of time spent waiting for I/O operations, in milliseconds. If this number is not very, very low, then the `rippled` server is probably having serious load issues. | | `load` | Object | _(Admin only)_ Detailed information about the current load state of the server | From 4abb07f5dc65bd996977c7fed50ce96dec3d5118 Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Tue, 18 Dec 2018 19:17:52 -0800 Subject: [PATCH 09/14] Capacity planning: add 2-hour estimates --- .../manage-the-rippled-server/installation/capacity-planning.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md b/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md index eae5b89174..56608bf109 100644 --- a/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md +++ b/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md @@ -143,7 +143,7 @@ The following table approximates the requirements for different amounts of histo | Real Time Amount | Number of Ledger Versions | Disk Space Required (RocksDB) | Disk Space Required (NuDB) | |:-----------------|:--------------------------|:------------------------------|:--| -| 2 hours | 2,000 | ***(TBD)*** | ***(TBD)*** | +| 2 hours | 2,000 | 250 MB | 450 MB | | 1 day | 25,000 | 8 GB | 12 GB | | 14 days | 350,000 | 112 GB | 168 GB | | 30 days | 750,000 | 240 GB | 360 GB | From 16e8b560eef218107f41024c40d89f147da1a593 Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Tue, 18 Dec 2018 19:18:49 -0800 Subject: [PATCH 10/14] State DB error, how to fix, & relation to key-value store type --- .../configuration/configure-full-history.md | 2 +- .../troubleshooting/server-wont-start.md | 13 ++++++++++++- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/content/tutorials/manage-the-rippled-server/configuration/configure-full-history.md b/content/tutorials/manage-the-rippled-server/configuration/configure-full-history.md index f6f534df7a..a03952129a 100644 --- a/content/tutorials/manage-the-rippled-server/configuration/configure-full-history.md +++ b/content/tutorials/manage-the-rippled-server/configuration/configure-full-history.md @@ -30,7 +30,7 @@ To configure your server to acquire and store full history, complete the followi On a full-history server, you should use NuDB for the ledger store, because RocksDB requires too much RAM when the database is that large. For more information, see [Capacity Planning](capacity-planning.html). You can remove the following performance-related configuration options from the default `[node_db]` stanza, because they only apply to RocksDB: `open_files`, `filter_bits`, `cache_mb`, `file_size_mb`, and `file_size_mult.` - **Caution:** If you have any history already downloaded with RocksDB, you must either delete that data or change the `path` field when you switch to NuDB. Otherwise, the server may [fail to start](server-wont-start.html). + **Caution:** If you have any history already downloaded with RocksDB, you must either delete that data or change the paths to the databases in the config file when you switch to NuDB. You must change both the `path` field of the `[node_db]` stanza **and** the `[database_path]` (SQLite database) setting. Otherwise, the server may [fail to start](server-wont-start.html#state-db-error). {% include '_snippets/conf-file-location.md' %} diff --git a/content/tutorials/manage-the-rippled-server/troubleshooting/server-wont-start.md b/content/tutorials/manage-the-rippled-server/troubleshooting/server-wont-start.md index 84c517fdd6..7b6d98eace 100644 --- a/content/tutorials/manage-the-rippled-server/troubleshooting/server-wont-start.md +++ b/content/tutorials/manage-the-rippled-server/troubleshooting/server-wont-start.md @@ -107,7 +107,7 @@ Possible solutions: ## State DB Error -The following error can occur if the `rippled` server's state database is corrupted (possibly as the result of being shutdown unexpectedly): +The following error can occur if the `rippled` server's state database is corrupted. This can occur as the result of being shutdown unexpectedly, or if you change the type of database from RocksDB to NuDB without changing the `path` and `[database_path]` settings in the config file. ```text 2018-Aug-21 23:06:38.675117810 SHAMapStore:ERR state db error: @@ -133,6 +133,17 @@ rm -r /var/lib/rippled/db **Tip:** It is generally safe to delete the `rippled` databases, because any individual server can re-download ledger history from other servers in the XRP Ledger network. +Alternatively, you can change the paths to the databases in the config file. For example: + +``` +[node_db] +type=NuDB +path=/var/lib/rippled/custom_nudb_path + +[database_path] +/var/lib/rippled/custom_sqlite_db_path +``` + ## Online Delete is Less Than Ledger History From ca7beae0a7bcf573c772907d0751323f3367da89 Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Wed, 19 Dec 2018 12:36:04 -0800 Subject: [PATCH 11/14] Fix broken disk storage links --- .../the-rippled-server/ledger-history/online-deletion.md | 2 +- .../installation/capacity-planning.md | 6 +++--- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/content/concepts/the-rippled-server/ledger-history/online-deletion.md b/content/concepts/the-rippled-server/ledger-history/online-deletion.md index 411cfa957d..668610b820 100644 --- a/content/concepts/the-rippled-server/ledger-history/online-deletion.md +++ b/content/concepts/the-rippled-server/ledger-history/online-deletion.md @@ -79,7 +79,7 @@ The following settings relate to online deletion: ![Ledger versions older than fetch_depth are not served to peers](img/fetch_depth.png) -For estimates of how much disk space is required to store different amounts of history, see [Capacity Planning](capacity-planning.html#historical-data). +For estimates of how much disk space is required to store different amounts of history, see [Capacity Planning](capacity-planning.html#disk-storage). ### Advisory Deletion diff --git a/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md b/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md index 56608bf109..af2bf460bf 100644 --- a/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md +++ b/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md @@ -55,9 +55,9 @@ The example `rippled-example.cfg` file has the `type` field in the `[node_db]` s [RocksDB](https://rocksdb.org/docs/getting-started.html) is an embeddable persistent key-value store that is optimized for rotational disks. -RocksDB requires approximately one-third less disk [storage](#storage) than NuDB and provides better I/O latency. However, the better I/O latency comes as result of the large amount of RAM RocksDB requires to store data indexes. +RocksDB requires approximately one-third less [disk storage](#disk-space) than NuDB and provides better I/O latency. However, the better I/O latency comes as result of the large amount of RAM RocksDB requires to store data indexes. -Validators should be configured to use RocksDB and to store no more than about 300,000 ledgers (approximately two weeks' worth of [historical data](#historical-data)) in the ledger store. +Validators should be configured to use RocksDB and to store no more than about 300,000 ledgers (approximately two weeks' worth of [historical data](#disk-space)) in the ledger store. RocksDB has performance-related configuration options that you can set in `rippled.cfg` to achieve maximum transaction processing throughput. Here is the recommended configuration for a `rippled` server using RocksDB: @@ -80,7 +80,7 @@ advisory_delete=0 [NuDB](https://github.com/vinniefalco/nudb#introduction) is an append-only key-value store that is optimized for SSD drives. -NuDB has nearly constant performance and memory footprints regardless of the amount of data being [stored](#storage). NuDB _requires_ a solid-state drive, but uses much less RAM than RocksDB to access a large database. +NuDB has nearly constant performance and memory footprints regardless of the [amount of data being stored](#disk-space). NuDB _requires_ a solid-state drive, but uses much less RAM than RocksDB to access a large database. Non-validator production servers should be configured to use NuDB and to store the amount of historical data required for the use case. From db01bd32de021dbb74a2db1a33bff5e7e10e3c04 Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Mon, 7 Jan 2019 15:55:50 -0800 Subject: [PATCH 12/14] Online deletion: edits per reviews, new diagram of how it works --- .../_img-sources/online-deletion-process.uxf | 2005 +++++++++++++++++ .../ledger-history/ledger-history.md | 6 +- .../ledger-history/online-deletion.md | 34 +- .../server-info-methods/server_info.md | 2 +- .../configure-advisory-deletion.md | 8 +- .../configuration/configure-full-history.md | 4 +- .../configure-online-deletion.md | 2 +- .../installation/capacity-planning.md | 2 +- img/online-deletion-process.png | Bin 0 -> 45570 bytes 9 files changed, 2040 insertions(+), 23 deletions(-) create mode 100644 content/_img-sources/online-deletion-process.uxf create mode 100644 img/online-deletion-process.png diff --git a/content/_img-sources/online-deletion-process.uxf b/content/_img-sources/online-deletion-process.uxf new file mode 100644 index 0000000000..84d2f6218a --- /dev/null +++ b/content/_img-sources/online-deletion-process.uxf @@ -0,0 +1,2005 @@ + + + 10 + + UMLClass + + 90 + 60 + 180 + 190 + + "Old" DB (Read-only) +fg=gray +style=wordwrap + + + + UMLClass + + 280 + 60 + 180 + 190 + + "Current" DB (Writable) +style=wordwrap + + + + UMLClass + + 120 + 100 + 20 + 20 + + + + + + UMLClass + + 140 + 100 + 20 + 20 + + + + + + UMLClass + + 180 + 100 + 20 + 20 + + + + + + UMLClass + + 200 + 100 + 20 + 20 + + + + + + UMLClass + + 220 + 100 + 20 + 20 + + + + + + UMLClass + + 100 + 120 + 20 + 20 + + + + + + UMLClass + + 120 + 120 + 20 + 20 + + lt=.. +fg=gray + + + + UMLClass + + 140 + 120 + 20 + 20 + + + + + + UMLClass + + 200 + 120 + 20 + 20 + + + + + + UMLClass + + 220 + 120 + 20 + 20 + + + + + + UMLClass + + 240 + 120 + 20 + 20 + + + + + + UMLClass + + 100 + 100 + 20 + 20 + + lt=.. +fg=gray + + + + UMLClass + + 160 + 140 + 20 + 20 + + + + + + UMLClass + + 160 + 120 + 20 + 20 + + lt=.. +fg=gray + + + + UMLClass + + 180 + 120 + 20 + 20 + + lt=.. +fg=gray + + + + UMLClass + + 160 + 100 + 20 + 20 + + lt=.. +fg=gray + + + + UMLClass + + 140 + 140 + 20 + 20 + + lt=.. +fg=gray + + + + UMLClass + + 240 + 100 + 20 + 20 + + lt=.. +fg=gray + + + + UMLClass + + 180 + 140 + 20 + 20 + + + + + + UMLClass + + 220 + 140 + 20 + 20 + + + + + + UMLClass + + 200 + 140 + 20 + 20 + + lt=.. +fg=gray + + + + UMLClass + + 540 + 80 + 20 + 20 + + lt=.. +fg=gray + + + + Text + + 560 + 70 + 210 + 40 + + Objects not used in any recent ledger version +style=wordwrap + + + + UMLClass + + 540 + 140 + 20 + 20 + + + + + + Text + + 560 + 130 + 210 + 40 + + Objects included in recent ledger version(s) +style=wordwrap + + + + UMLClass + + 290 + 100 + 20 + 20 + + + + + + UMLClass + + 310 + 100 + 20 + 20 + + + + + + UMLClass + + 330 + 100 + 20 + 20 + + + + + + UMLClass + + 350 + 100 + 20 + 20 + + + + + + UMLClass + + 350 + 120 + 20 + 20 + + + + + + UMLClass + + 310 + 120 + 20 + 20 + + + + + + UMLClass + + 330 + 120 + 20 + 20 + + + + + + UMLClass + + 370 + 100 + 20 + 20 + + + + + + UMLClass + + 370 + 120 + 20 + 20 + + + + + + UMLClass + + 390 + 100 + 20 + 20 + + + + + + UMLClass + + 390 + 120 + 20 + 20 + + + + + + UMLClass + + 410 + 100 + 20 + 20 + + + + + + UMLClass + + 410 + 120 + 20 + 20 + + + + + + UMLClass + + 290 + 120 + 20 + 20 + + + + + + UMLClass + + 430 + 120 + 20 + 20 + + + + + + UMLClass + + 430 + 100 + 20 + 20 + + + + + + UMLClass + + 350 + 140 + 20 + 20 + + + + + + UMLClass + + 370 + 140 + 20 + 20 + + + + + + UMLClass + + 390 + 140 + 20 + 20 + + + + + + UMLClass + + 410 + 140 + 20 + 20 + + + + + + UMLClass + + 430 + 140 + 20 + 20 + + + + + + UMLClass + + 290 + 140 + 20 + 20 + + + + + + UMLClass + + 310 + 140 + 20 + 20 + + + + + + Text + + 30 + 30 + 200 + 30 + + *Before online deletion* + + + + UMLClass + + 460 + 360 + 180 + 190 + + "Current" DB (Writable) +style=wordwrap + + + + UMLClass + + 610 + 430 + 20 + 20 + + + + + + UMLClass + + 590 + 430 + 20 + 20 + + + + + + UMLClass + + 550 + 430 + 20 + 20 + + + + + + UMLClass + + 570 + 430 + 20 + 20 + + + + + + UMLClass + + 530 + 430 + 20 + 20 + + + + + + UMLClass + + 490 + 430 + 20 + 20 + + + + + + UMLClass + + 470 + 430 + 20 + 20 + + + + + + UMLClass + + 470 + 410 + 20 + 20 + + + + + + UMLClass + + 470 + 390 + 20 + 20 + + + + + + UMLClass + + 490 + 390 + 20 + 20 + + + + + + UMLClass + + 490 + 410 + 20 + 20 + + + + + + UMLClass + + 510 + 410 + 20 + 20 + + + + + + UMLClass + + 510 + 390 + 20 + 20 + + + + + + UMLClass + + 530 + 390 + 20 + 20 + + + + + + UMLClass + + 530 + 410 + 20 + 20 + + + + + + UMLClass + + 550 + 410 + 20 + 20 + + + + + + UMLClass + + 550 + 390 + 20 + 20 + + + + + + UMLClass + + 570 + 390 + 20 + 20 + + + + + + UMLClass + + 570 + 410 + 20 + 20 + + + + + + UMLClass + + 590 + 410 + 20 + 20 + + + + + + UMLClass + + 590 + 390 + 20 + 20 + + + + + + UMLClass + + 610 + 390 + 20 + 20 + + + + + + UMLClass + + 610 + 410 + 20 + 20 + + + + + + Text + + 40 + 330 + 200 + 30 + + *During online deletion* + + + + UMLClass + + 90 + 360 + 180 + 190 + + "Old" DB (Read-only) +fg=gray +style=wordwrap + + + + UMLClass + + 120 + 400 + 20 + 20 + + + + + + UMLClass + + 140 + 400 + 20 + 20 + + + + + + UMLClass + + 180 + 400 + 20 + 20 + + + + + + UMLClass + + 200 + 400 + 20 + 20 + + + + + + UMLClass + + 220 + 400 + 20 + 20 + + + + + + UMLClass + + 100 + 420 + 20 + 20 + + + + + + UMLClass + + 120 + 420 + 20 + 20 + + lt=.. +fg=gray + + + + UMLClass + + 140 + 420 + 20 + 20 + + + + + + UMLClass + + 200 + 420 + 20 + 20 + + + + + + UMLClass + + 220 + 420 + 20 + 20 + + + + + + UMLClass + + 240 + 420 + 20 + 20 + + + + + + UMLClass + + 100 + 400 + 20 + 20 + + lt=.. +fg=gray + + + + UMLClass + + 160 + 440 + 20 + 20 + + + + + + UMLClass + + 160 + 420 + 20 + 20 + + lt=.. +fg=gray + + + + UMLClass + + 180 + 420 + 20 + 20 + + lt=.. +fg=gray + + + + UMLClass + + 160 + 400 + 20 + 20 + + lt=.. +fg=gray + + + + UMLClass + + 140 + 440 + 20 + 20 + + lt=.. +fg=gray + + + + UMLClass + + 240 + 400 + 20 + 20 + + lt=.. +fg=gray + + + + UMLClass + + 180 + 440 + 20 + 20 + + + + + + UMLClass + + 220 + 440 + 20 + 20 + + + + + + UMLClass + + 200 + 440 + 20 + 20 + + lt=.. +fg=gray + + + + UMLClass + + 330 + 140 + 20 + 20 + + lt=.. +fg=gray + + + + UMLClass + + 510 + 430 + 20 + 20 + + lt=.. +fg=gray + + + + UMLClass + + 300 + 480 + 20 + 20 + + fg=green + + + + UMLClass + + 320 + 480 + 20 + 20 + + fg=green + + + + UMLClass + + 360 + 480 + 20 + 20 + + fg=green + + + + UMLClass + + 380 + 480 + 20 + 20 + + fg=green + + + + UMLClass + + 400 + 480 + 20 + 20 + + fg=green + + + + UMLClass + + 280 + 500 + 20 + 20 + + fg=green + + + + UMLClass + + 320 + 500 + 20 + 20 + + fg=green + + + + UMLClass + + 400 + 500 + 20 + 20 + + fg=green + + + + UMLClass + + 420 + 500 + 20 + 20 + + fg=green + + + + UMLClass + + 340 + 520 + 20 + 20 + + fg=green + + + + UMLClass + + 360 + 520 + 20 + 20 + + fg=green + + + + UMLClass + + 400 + 520 + 20 + 20 + + fg=green + + + + Relation + + 170 + 460 + 120 + 70 + + lt=<<<- + 100.0;50.0;10.0;10.0 + + + Relation + + 440 + 490 + 70 + 40 + + lt=<<<- + 50.0;10.0;10.0;20.0 + + + UMLClass + + 400 + 500 + 20 + 20 + + fg=green + + + + UMLClass + + 490 + 450 + 20 + 20 + + fg=green + + + + UMLClass + + 510 + 450 + 20 + 20 + + fg=green + + + + UMLClass + + 550 + 450 + 20 + 20 + + fg=green + + + + UMLClass + + 570 + 450 + 20 + 20 + + fg=green + + + + UMLClass + + 590 + 450 + 20 + 20 + + fg=green + + + + UMLClass + + 470 + 450 + 20 + 20 + + fg=green + + + + UMLClass + + 470 + 470 + 20 + 20 + + fg=green + + + + UMLClass + + 530 + 450 + 20 + 20 + + fg=green + + + + UMLClass + + 610 + 450 + 20 + 20 + + fg=green + + + + UMLClass + + 490 + 470 + 20 + 20 + + fg=green + + + + UMLClass + + 510 + 470 + 20 + 20 + + fg=green + + + + UMLClass + + 530 + 470 + 20 + 20 + + fg=green + + + + UMLClass + + 550 + 470 + 20 + 20 + + fg=green + + + + Text + + 490 + 210 + 170 + 80 + + The "current" DB can contain outdated objects, but it's less likely to. +style=wordwrap +fg=blue + + + + Text + + 20 + 570 + 140 + 70 + + Online deletion drops entire "Old" DB +style=wordwrap +fg=blue + + + + UMLClass + + 90 + 720 + 180 + 190 + + "Old" DB (Read-only) +fg=gray +style=wordwrap + + + + UMLClass + + 240 + 790 + 20 + 20 + + + + + + UMLClass + + 220 + 790 + 20 + 20 + + + + + + UMLClass + + 180 + 790 + 20 + 20 + + + + + + UMLClass + + 200 + 790 + 20 + 20 + + + + + + UMLClass + + 160 + 790 + 20 + 20 + + + + + + UMLClass + + 120 + 790 + 20 + 20 + + + + + + UMLClass + + 100 + 790 + 20 + 20 + + + + + + UMLClass + + 100 + 770 + 20 + 20 + + + + + + UMLClass + + 100 + 750 + 20 + 20 + + + + + + UMLClass + + 120 + 750 + 20 + 20 + + + + + + UMLClass + + 120 + 770 + 20 + 20 + + + + + + UMLClass + + 140 + 770 + 20 + 20 + + + + + + UMLClass + + 140 + 750 + 20 + 20 + + + + + + UMLClass + + 160 + 750 + 20 + 20 + + + + + + UMLClass + + 160 + 770 + 20 + 20 + + + + + + UMLClass + + 180 + 770 + 20 + 20 + + + + + + UMLClass + + 180 + 750 + 20 + 20 + + + + + + UMLClass + + 200 + 750 + 20 + 20 + + + + + + UMLClass + + 200 + 770 + 20 + 20 + + + + + + UMLClass + + 220 + 770 + 20 + 20 + + + + + + UMLClass + + 220 + 750 + 20 + 20 + + + + + + UMLClass + + 240 + 750 + 20 + 20 + + + + + + UMLClass + + 240 + 770 + 20 + 20 + + + + + + UMLClass + + 140 + 790 + 20 + 20 + + lt=.. +fg=gray + + + + UMLClass + + 120 + 810 + 20 + 20 + + + + + + UMLClass + + 140 + 810 + 20 + 20 + + + + + + UMLClass + + 180 + 810 + 20 + 20 + + + + + + UMLClass + + 200 + 810 + 20 + 20 + + + + + + UMLClass + + 220 + 810 + 20 + 20 + + + + + + UMLClass + + 100 + 810 + 20 + 20 + + + + + + UMLClass + + 100 + 830 + 20 + 20 + + + + + + UMLClass + + 160 + 810 + 20 + 20 + + + + + + UMLClass + + 240 + 810 + 20 + 20 + + + + + + UMLClass + + 120 + 830 + 20 + 20 + + + + + + UMLClass + + 140 + 830 + 20 + 20 + + + + + + UMLClass + + 160 + 830 + 20 + 20 + + + + + + UMLClass + + 180 + 830 + 20 + 20 + + + + + + UMLClass + + 280 + 720 + 180 + 190 + + "Current" DB (Writable) +style=wordwrap + + + + Relation + + 230 + 550 + 280 + 180 + + lt=<<<- + 10.0;160.0;260.0;10.0 + + + UMLSpecialState + + 160 + 600 + 20 + 20 + + type=termination + + + + Relation + + 160 + 550 + 30 + 60 + + lt=<<<- + 10.0;40.0;10.0;10.0 + + + Text + + 280 + 360 + 170 + 120 + + Online deletion copies objects to the "current" database if those objects are still being used by current ledgers. +style=wordwrap +fg=blue + + + + Text + + 470 + 780 + 150 + 70 + + New "Current" DB begins empty +style=wordwrap +fg=blue + + + + Text + + 420 + 620 + 110 + 70 + + "Current" DB becomes "Old" DB +style=wordwrap +fg=blue + + + + Text + + 40 + 690 + 200 + 30 + + *After online deletion* + + + + UMLClass + + 520 + 40 + 260 + 150 + + Legend +-- + + + + Relation + + 340 + 160 + 160 + 100 + + lt=.. +fg=blue + 10.0;10.0;140.0;80.0 + + diff --git a/content/concepts/the-rippled-server/ledger-history/ledger-history.md b/content/concepts/the-rippled-server/ledger-history/ledger-history.md index 14ad3b72e5..3181c128d6 100644 --- a/content/concepts/the-rippled-server/ledger-history/ledger-history.md +++ b/content/concepts/the-rippled-server/ledger-history/ledger-history.md @@ -2,7 +2,7 @@ The [consensus process](intro-to-consensus.html) creates a chain of [validated ledger versions](ledgers.html), each derived from the previous one by applying a set of transactions. Every `rippled` server stores ledger versions and transaction history locally. The amount of transaction history a server stores depends on how long that server has been online and how much history it is configured to fetch and keep. -Servers in the peer-to-peer XRP Ledger network share transactions and other data with each other as part of the consensus process. Each server each independently builds each new ledger version and compares results with its trusted validators to ensure consistency. (If a consensus of trusted validators disagrees with a server's results, that server fetches the necessary data from its peers to achieve consistency.) Servers can download older data from their peers to fill gaps in their available history. The structure of the ledger uses cryptographic [hashes](basic-data-types.html#hashes) of the data so that any server can verify the integrity and consistency of the data. +Servers in the peer-to-peer XRP Ledger network share transactions and other data with each other as part of the consensus process. Each server independently builds each new ledger version and compares results with its trusted validators to ensure consistency. (If a consensus of trusted validators disagrees with a server's results, that server fetches the necessary data from its peers to achieve consistency.) Servers can download older data from their peers to fill gaps in their available history. The structure of the ledger uses cryptographic [hashes](basic-data-types.html#hashes) of the data so that any server can verify the integrity and consistency of the data. ## Databases @@ -18,7 +18,7 @@ The [server_info method][] reports how many ledger versions your server has avai ## Fetching History -When it starts, a `rippled` server's first priority is to get a complete copy of the latest validated ledger. From there, it keeps up with advances in the ledger progress. If configured to do so, the server also backfills ledger history up to a configured amount, which must be equal or less than the cutoff beyond which online deletion is configured to delete. +When it starts, a `rippled` server's first priority is to get a complete copy of the latest validated ledger. From there, it keeps up with advances in the ledger progress. If configured to do so, the server also backfills ledger history up to a configured amount, which must be equal to or less than the cutoff beyond which online deletion is configured to delete. The server can backfill history from before it became synced, as well as filling in any gaps in the history it has collected after syncing. (Gaps in ledger history can occur if a server temporarily becomes too busy to keep up with the network, loses its network connection, or suffers other temporary issues.) To backfill history, the server requests data from its peer `rippled` servers. The amount the server tries to backfill is defined by the `[ledger_history]` setting. @@ -39,7 +39,7 @@ Ripple provides a set of public full-history servers as a public service at `s2. **Tip:** Unlike some cryptocurrency networks, servers in the XRP Ledger do not need full history to know the current state and keep up with current transactions. -For instructions setting up full history, see [Configure Full History](configure-full-history.html). +For instructions on setting up full history, see [Configure Full History](configure-full-history.html). ## History Sharding diff --git a/content/concepts/the-rippled-server/ledger-history/online-deletion.md b/content/concepts/the-rippled-server/ledger-history/online-deletion.md index 668610b820..880862fa63 100644 --- a/content/concepts/the-rippled-server/ledger-history/online-deletion.md +++ b/content/concepts/the-rippled-server/ledger-history/online-deletion.md @@ -14,7 +14,7 @@ The default config file sets the `rippled` server to keep the most recent 2000 l The `rippled` server stores [ledger history](ledger-history.html) in its _ledger store_. This data accumulates over time. -Inside the ledger store, ledger data is "de-duplicated": data that doesn't change from version to version is only stored once. The records themselves in the ledger store do not indicate which ledger version(s) contain them; part of the work of online deletion is identifying which records are only used by outdated ledger versions. This process is time consuming and affects the disk I/O and application cache, so it is not feasible to delete old data on every ledger close. +Inside the ledger store, ledger data is "de-duplicated". In other words, data that doesn't change from version to version is only stored once. The records themselves in the ledger store do not indicate which ledger version(s) contain them; part of the work of online deletion is identifying which records are only used by outdated ledger versions. This process is time consuming and affects the disk I/O and application cache, so it is not feasible to delete old data on every ledger close. ## Online Deletion Behavior @@ -25,12 +25,16 @@ The online deletion settings configure how many ledger versions the `rippled` se - The server may store up to twice the configured number of ledger versions if online deletion is set to run automatically. (Each time it runs, it reduces the number of stored ledger versions to approximately the configured number.) - If advisory deletion is enabled, the server stores all the ledger versions that it has acquired and built until its administrator calls the [can_delete method][]. - If you call [can_delete][can_delete method] on a regular basis, the maximum amount of data the server stores is either of the following, whichever is larger: + The amount of data the server stores depends on how often you call [can_delete][can_delete method] and the how big an interval of time your `online_delete` setting represents: - - A number of ledger versions approximately equal to twice the `online_delete` value. (After deletion, this is reduced to approximately the `online_delete` value.) - - Ledger versions spanning an amount of time that is approximately twice the interval between `can_delete` calls. (After deletion, this is reduced to approximately one interval's worth of data.) + - If you call `can_delete` _more often_ than your `online_delete` interval, the server stores at most a number of ledger versions approximately equal to **twice the `online_delete` value**. (After deletion, this is reduced to approximately the `online_delete` value.) + + For example, if you call `can_delete` with a value of `now` once per day and an `online_delete` value of 50,000, the server typically stores up to 100,000 ledger versions before running deletion. After running deletion, the server keeps at least 50,000 ledger versions (about two days' worth). With this configuration, approximately every other `can_delete` call results in no change because the server does not have enough ledger versions to delete. + + - If you call `can_delete` _less often_ than your `online_delete` interval, the server stores at most ledger versions spanning an amount of time that is approximately **twice the interval between `can_delete` calls**. (After deletion, this is reduced to approximately one interval's worth of data.) + + For example, if you call `can_delete` with a value of `now` once per day and an `online_delete` value of 2000, the server typically stores up to two full days' worth of ledger versions before running deletion. After running deletion, the server keeps approximately one day's worth (about 25,000 ledger versions), but never fewer than 2000 ledger versions. - For example, if you call `can_delete` with a value of `now` once per day and an `online_delete` value of 2000, the server typically stores up two full day's worth of ledger versions before running deletion. After running deletion, the server keeps approximately 1 day's worth, but never less than 2000 ledger versions. With online deletion enabled and running automatically (that is, with advisory delete disabled), the total amount of ledger data stored should remain at minimum equal to the number of ledger versions the server is configured to keep, with the maximum being roughly twice that many. @@ -44,7 +48,7 @@ Online deletion automatically stops if the [server state](rippled-server-states. If you stop the server or it crashes while online deletion is running, online deletion resumes after the server is restarted and the server becomes fully synced. -To temporarily disable online deletion, you can use the [can_delete method][] with an argument of `never`. This change persists until you re-enable online deletion by calling [can_delete][can_delete method] again, even if you change the . For more information on controlling when online deletion happens, see [Advisory Deletion](#advisory-deletion). +To temporarily disable online deletion, you can use the [can_delete method][] with an argument of `never`. This change persists until you re-enable online deletion by calling [can_delete][can_delete method] again. For more information on controlling when online deletion happens, see [Advisory Deletion](#advisory-deletion). ## Configuration @@ -55,7 +59,7 @@ The following settings relate to online deletion: The default config file specifies 2000 for this value. This cannot be less than 256, because some events like [Fee Voting](fee-voting.html) and the [Amendment Process](amendments.html#amendment-process) update only every 256 ledgers. - **Caution:** If you run `rippled` with `online_delete` disabled, then later enable `online_delete` and restart the server, the server disregards but does not delete existing ledger history that your server already downloaded while `online_delete` was disabled. To save disk space, delete your existing history before re-starting the server with `online_delete` back on. + **Caution:** If you run `rippled` with `online_delete` disabled, then later enable `online_delete` and restart the server, the server disregards but does not delete existing ledger history that your server already downloaded while `online_delete` was disabled. To save disk space, delete your existing history before re-starting the server after changing the `online_delete` setting. - **`[ledger_history]`** - Specify a number of validated ledgers, equal to or less than `online_delete`. If the server does not have at least this many validated ledger versions, it attempts to backfill them by fetching the data from peers. @@ -79,23 +83,31 @@ The following settings relate to online deletion: ![Ledger versions older than fetch_depth are not served to peers](img/fetch_depth.png) -For estimates of how much disk space is required to store different amounts of history, see [Capacity Planning](capacity-planning.html#disk-storage). +For estimates of how much disk space is required to store different amounts of history, see [Capacity Planning](capacity-planning.html#disk-space). ### Advisory Deletion -By default, online deletion happens automatically and periodically. If the `advisory_delete` setting is enabled in the config file, online deletion only happens when an administrator triggers it using the [can_delete method][]. +The default config file schedules online deletion to happen automatically and periodically. If the config file does not specify an `online_delete` interval, online deletion does not occur. If config file enables the `advisory_delete` setting, online deletion only happens when an administrator triggers it using the [can_delete method][]. You can use advisory deletion with a scheduled job to trigger automatic deletion based on clock time instead of the number of ledger versions closed. If your server is heavily used, the extra load from online deletion can cause your server to fall behind and temporarily de-sync from the consensus network. If this is the case, you can use advisory deletion and schedule online deletion to happen only during off-peak times. -Alternatively, you can use advisory deletion for other reasons, such as if you want to manually confirm that transaction data is backed up to a separate server before deleting it. +You can use advisory deletion for other reasons. For example, you may want to manually confirm that transaction data is backed up to a separate server before deleting it. Alternatively, you may want to manually confirm that a separate task has finished processing transaction data before you delete that data. The `can_delete` API method can enable or disable automatic deletion, in general or up to a specific ledger version, as long as `advisory_delete` is enabled in the config file. These settings changes persist even if you restart the `rippled` server, unless you disable `advisory_delete` in the config file before restarting. +## How It Works + +Online deletion works by creating two databases: at any given time, there is an "old" database, which is read-only, and a "current" database, which is writable. The `rippled` server can read objects from either database, so current ledger versions may contain objects in either one. If an object in a ledger does not change from ledger version to ledger version, only one copy of that object remains in the database, so the server does not store redundant copies of that object. When a new ledger version modifies an object, the server stores the modified object in the "new" database, while the previous version of the object (which is still used by previous ledger versions) remains in the "old" database. + +When it comes time for online deletion, the server first walks through the oldest ledger version to keep, and copies all objects in that ledger version from the read-only "old" database into the "current" database. This guarantees that the "current" database now contains all objects used in the chosen ledger version and all newer versions. Then, the server deletes the "old" database, and changes the existing "current" database to become "old" and read-only. The server starts a new "current" database to contain any newer changes after this point. + +![Diagram showing how online deletion uses two databases](img/online-deletion-process.png) + ## See Also - [Capacity Planning](capacity-planning.html) -- [can_delete method][] API reference documentation +- [can_delete method][] - API reference documentation - [Configure Online Deletion](configure-online-deletion.html) - [Configure Advisory Deletion](configure-advisory-deletion.html) - [Configure Full History](configure-full-history.html) diff --git a/content/references/rippled-api/public-rippled-methods/server-info-methods/server_info.md b/content/references/rippled-api/public-rippled-methods/server-info-methods/server_info.md index 35d47456ff..64d6bf3345 100644 --- a/content/references/rippled-api/public-rippled-methods/server-info-methods/server_info.md +++ b/content/references/rippled-api/public-rippled-methods/server-info-methods/server_info.md @@ -283,7 +283,7 @@ The `info` object may have some arrangement of the following fields: | `amendment_blocked` | Boolean | _(May be omitted)_ If `true`, this server is [amendment blocked](amendments.html#amendment-blocked). If the server is not amendment blocked, the response omits this field. [New in: rippled 0.80.0][] | | `build_version` | String | The version number of the running `rippled` version. | | `closed_ledger` | Object | (May be omitted) Information on the most recently closed ledger that has not been validated by consensus. If the most recently validated ledger is available, the response omits this field and includes `validated_ledger` instead. The member fields are the same as the `validated_ledger` field. | -| `complete_ledgers` | String | Range expression indicating the sequence numbers of the ledger versions the local rippled has in its database. This may be a disjoint sequence, for example `24900901-24900984,24901116-24901158`. If the server does not have any complete ledgers (for example, it just started syncing with the network), this is the string `empty`. | +| `complete_ledgers` | String | Range expression indicating the sequence numbers of the ledger versions the local `rippled` has in its database. This may be a disjoint sequence, for example `24900901-24900984,24901116-24901158`. If the server does not have any complete ledgers (for example, it just started syncing with the network), this is the string `empty`. | | `hostid` | String | On an admin request, returns the hostname of the server running the `rippled` instance; otherwise, returns a unique four letter word. | | `io_latency_ms` | Number | Amount of time spent waiting for I/O operations, in milliseconds. If this number is not very, very low, then the `rippled` server is probably having serious load issues. | | `last_close` | Object | Information about the last time the server closed a ledger, including the amount of time it took to reach a consensus and the number of trusted validators participating. | diff --git a/content/tutorials/manage-the-rippled-server/configuration/configure-advisory-deletion.md b/content/tutorials/manage-the-rippled-server/configuration/configure-advisory-deletion.md index 01073116cd..3e704ec630 100644 --- a/content/tutorials/manage-the-rippled-server/configuration/configure-advisory-deletion.md +++ b/content/tutorials/manage-the-rippled-server/configuration/configure-advisory-deletion.md @@ -1,6 +1,6 @@ # Configure Advisory Deletion -The default config file sets `rippled` to automatically delete outdated history of XRP Ledger state and transactions as new ledger versions become available. If your server uses most of its hardware resources during peak hours, you can configure the server to delete ledgers only when prompted by a command scheduled to run during off-peak hours to reduce the chances of online deletion. +The default config file sets `rippled` to automatically delete outdated history of XRP Ledger state and transactions as new ledger versions become available. If your server uses most of its hardware resources during peak hours, you can configure the server to delete ledgers only when prompted by a command scheduled to run during off-peak hours, so that online deletion is less likely to impact server performance. ## Prerequisites @@ -22,7 +22,7 @@ This tutorial assumes your server meets the following prerequisites: - Your server has enough disk space to store your chosen amount of history in its ledger store. - See [Capacity Planning](capacity-planning.html) for details of how much storage is required for different configuration. With advisory deletion enabled, the maximum history a server may accumulate before deletion is equal to the number of ledger versions configured in the `online_delete` setting **plus** the amount of time between online deletion prompts. + See [Capacity Planning](capacity-planning.html) for details of how much storage is required for different configurations. With advisory deletion enabled, the maximum history a server may accumulate before deletion is equal to the number of ledger versions configured in the `online_delete` setting **plus** the amount of time between online deletion prompts. - You know which hours are least busy for your server. @@ -48,7 +48,7 @@ To configure advisory deletion with a daily schedule, perform the following step $ rippled --conf=/etc/opt/ripple/rippled.cfg can_delete now - The response indicates the maximum ledger index that may the server may delete from its ledger store. For example, the following message indicates that ledger versions up to and including ledger index 43633667 are to be deleted: + The response indicates the maximum ledger index that the server may delete from its ledger store. For example, the following message indicates that ledger versions up to and including ledger index 43633667 can be deleted: { "result": { @@ -71,7 +71,7 @@ To configure advisory deletion with a daily schedule, perform the following step Be sure that you schedule the command to run based on your server's configured time zone. - **Tip:** You do not need to schedule a `cron` job to run online deletion if you have `advisory_delete` disabled. In that case, `rippled` runs online deletion automatically when the server has approximately twice the number of ledgers to keep after deletion. + **Tip:** You do not need to schedule a `cron` job to run online deletion if you have `advisory_delete` disabled. In that case, `rippled` runs online deletion automatically when the difference between the server's oldest and current validated ledger versions is at least the value of `online_delete`. 4. Start (or restart) the `rippled` service. diff --git a/content/tutorials/manage-the-rippled-server/configuration/configure-full-history.md b/content/tutorials/manage-the-rippled-server/configuration/configure-full-history.md index a03952129a..fd7a96d5a1 100644 --- a/content/tutorials/manage-the-rippled-server/configuration/configure-full-history.md +++ b/content/tutorials/manage-the-rippled-server/configuration/configure-full-history.md @@ -52,12 +52,12 @@ To configure your server to acquire and store full history, complete the followi 0. If you have a database dump from another full-history server to use as a basis, set the `[import_db]` stanza of your server's config file to point to the data to be imported. (Otherwise, skip this step.) [import_db] - type=RocksDB + type=NuDB path=/tmp/full_history_dump/ 0. Remove your server's existing database files, if you have any from previously running `rippled`. - After disabling online deletion, the server ignores any data that was downloaded while online deletion was enabled, so you may as well clear up the disk space: + After disabling online deletion, the server ignores any data that was downloaded while online deletion was enabled, so you may as well clear up the disk space. For example: rm -r /var/lib/rippled/db/rocksdb/* rm /var/lib/rippled/db/*.db diff --git a/content/tutorials/manage-the-rippled-server/configuration/configure-online-deletion.md b/content/tutorials/manage-the-rippled-server/configuration/configure-online-deletion.md index ff31c6c947..64b8f96dd9 100644 --- a/content/tutorials/manage-the-rippled-server/configuration/configure-online-deletion.md +++ b/content/tutorials/manage-the-rippled-server/configuration/configure-online-deletion.md @@ -21,7 +21,7 @@ To change the amount of history your server stores, perform the following steps: 1. Decide how many ledger versions' worth of history to store. - New ledger versions are usually validated 3 to 4 seconds apart, so the number of ledger versions corresponds roughly to the amount of time you want to store. See [Capacity Planning](capacity-planning.html) for details of how much storage is required for different configuration. + New ledger versions are usually validated 3 to 4 seconds apart, so the number of ledger versions corresponds roughly to the amount of time you want to store. See [Capacity Planning](capacity-planning.html) for details of how much storage is required for different configurations. Online deletion is based on how many ledger versions to keep _after_ deleting history, so you should have enough disk space to store twice as many ledgers as you set it to keep. diff --git a/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md b/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md index af2bf460bf..7296e9a286 100644 --- a/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md +++ b/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md @@ -53,7 +53,7 @@ The example `rippled-example.cfg` file has the `type` field in the `[node_db]` s #### More About Using RocksDB -[RocksDB](https://rocksdb.org/docs/getting-started.html) is an embeddable persistent key-value store that is optimized for rotational disks. +[RocksDB](https://rocksdb.org/docs/getting-started.html) is an embeddable persistent key-value store that works well with both rotational disks and solid state disks. RocksDB requires approximately one-third less [disk storage](#disk-space) than NuDB and provides better I/O latency. However, the better I/O latency comes as result of the large amount of RAM RocksDB requires to store data indexes. diff --git a/img/online-deletion-process.png b/img/online-deletion-process.png new file mode 100644 index 0000000000000000000000000000000000000000..fc38c8a69794b61ce4b31b6d4361f339b6a88753 GIT binary patch literal 45570 zcmd3ObySsW7bhwxh)5|&2-4C50@6w;E$N|;q?9yBD+VPJ%ArA!4nex{peWs-v`R?` zN;k7#^xp5@Z)VoaTJz7a)pBO6yBFf3oVr&ezn<2NzCK zyHVb3;yG7|SNW?mcIu|=>06|esP|E$_dof6;o!WWF4gGp_%RN>6R#x+z9r+)?1O^d zH@&Ma?ySv^l)v~Si+fn)pEBW}ZX^PdVfhEw;yQPzi( zxh;*2u3HiOS}5*p(cPb~-_I><@j>IqI{kFA#yeGH+I89 z1or(o{dS+6r&^+NZxyH}ik4n`c$|TOL0w&)Y$k-auJDemx3_l~orHB?mU@AaMD(u2 zFV>Z3=f&3H5q0wQ$i8-8rEoBv-Mm*4=9uEVhJuQBzT|mntbM?~X3{T4ETY z^SBzFg9o$JnUvyr^-7uqJRCvRc1uig?i;kEq0KEV=T$GL2?z+R;n#YuMTQUL>w_dL z7k5e$o;-PykZ`!f>Nzd7fW_0P)0ov?WKQEg8|NiFS9{f@h)$ozY}yRu@wxwK*xg#0 zGQgX*$HuoEJ9>1??_kgLbC-Glgc&WX{u(??+0<;btEOM^}U=%%>0b#t-tP_9o~LE~+h6&GYc$wF;$dG?K6X z7|x8oBqE|qY}Jz%#2~Y%8{0ZTS9}%AVIgw_#~<%LdTaCaz*K8=rb>K9r@nEuTmGX! z&d*~nclrx_r`S<`2Rq|yui}O62V{<)I(I=$Mq0Yf(5w9J=+olr8T*XF*p05uV%$vf!bjqHX z(e&Q(pX$4bb02-=0;-NOkMq`RTs%C5z}4#I)#{}>0SlG07f~O+^!21Gwa4lo&usIb zN~(`-uvjF=E`zL$)p{8|!5=N=kH$(^)cfw$&@yuA{k0dU!lw)k_wV2L+nYD`eDwak zEJ3zz@igpzXR=gsT%4%Gm$$;>-kKt6PLyx1xGls?-h6E!(mC6YX~Z{JF29-8=EI13 z%#rl+K=!RwWHp>Hf5rO*e)C}3;Nakvrl!KPc_u<YsO&9nBs`<3_B@|!nrKzTfuo7sryofK1@+=XXB8KtGAqd2s| zHeH4d10|meC@LWBPV~=DdoFZIU}1= zW;^IK*O#3l6QI~iBRTm(mNM_{+qZHd6tnfp!OxxBxX&%(dZ#Hx0|Q$B$A{1FN`7(x!>ymD@X(3i)Tut(|~)wT)Bm)Qm2Q-*`~E zy1FSTDW^ygYkVI)y6exv%)%mf{W=>PTe6t@;uq{g=jpcC%T|$#@@w;-)RdIOe70AA zetp;O+>;ntfseU#r|W`GWVKlnvpeFzNb=%k3-4AZmDvr?1z2<@jTYX17akryRBA(f z?$Xt9{{>h=WEz}pe~$M2U~#7G;&^?1M7$lDJGP~zMNwJ#GQU~u@!wa{k4s2Ms9vnH z87i^5jOUMsgR{zc(`Debp%}>Sb2c@wci!K-Qs9XKPM=TJgGch{P?3>FK2F_0|CPXbFjaY?6XegKXr>%%D2X8swIG! zp2tE`O>%3}xl`iSXD!TBVMzE_flhIOyCYX%2Vg1Qyr|~p=7a4|^&sLb`B#GJ?k$c~ znzv$VS9_F9KRk^%p!eNOd2~jn(AHUKWz1`7R_Ssmh+R*b0xaQReb^opbeif~%%$56 zpj!r>{31&&QJfOaNusWEAL;$z9HxqRP@wD+AH#U#u2JX~R=LbFe!8Ea#~7&m{Q=?x z!Wm@d2J&gBsi`R`Kh^o{fVM^r2p()sgb`C;&U(vFb5865?uip0-@g~%{rS$mZfiP< z^OiUh{#gz!lAyuD+YfsO$;ru25Yy#JVZ0d`8J}{h&^-zW0Eag!Ke<%5qq=MWexaJ! zV<2l1cOK*|A|isvo)g)CqsauARb}3~&2RelwXWaZ2Z5ehn3>h=tq+qiewC!m^H`bA z9wDM&me$i7dcm$a-H~Xs>L!o-wc#;gpfdC|2t2LUJ zcix|+DTGl)ugc#Qv@1@~COA6JxZcQU1l+67_H5?CiqwJWPSs{6de&>OrIg|N!FqXp zf0p_yuo4=SO1Rg?j)LLKN4-hSf7#j%@|`H=KBwlsG{$YYJC|FqnrZC2WeyU~($M^z zZH;8>B!i-LLema)R05g+0;H&EkTMc3+}!tgN}@Z_#=e8QiAGhaR=pW}YXyF5B}anR zzXs5=varkwS54j|#clnvI@=3N+xr&6Ys<7ST&`BMy}z?0Q5@MZR&xZy@)gcBUH*gS zFOgY1mbwHRP}u1m@>UFYuf0#ub8ZMvu0$&datB)Ot|T9vp^s9X4;3E!TKPS)DzD;4MavL7O@9ABsT+r@B&Pz`BCWQUr)OPd{Zqy^;?FQ}+CFQG%h1 zic1S4Dw@|A^K*68MZ!RpP2_uRp2OzP+ajeN73;?dF_)Db90~_@2w6}^jvRrAI7&G| zT2}FoPdlropg?NhWvmlKb?)577Ea1Y+R#JSb(?Pq9UBxDRuASzg2~LxZ1*d%ABkXu zQe$xuh^)_Cc4iyHXqBti=GTC=kEtqtTdl^EXO}(|B-3f+>j`GRv>UCZWfXV(Cc89N zn;x{fBo!Idr`~sygR`Awj-3&LHr4zbCgV%Y&zY`2LqZJ-|0xL092P$FHJgEKV16F# z)E_)|;5$`JbfbW2^q~G=&wfJwlM-+3n7=68ECtb0)wn| zks=mZ195(Sjl4d5Qh__8Ri$*rpj;Nkv)!+*KM*wu$w$}zY#>ZM4~vvY%hAsD-T6^o zrYo`eyDzOUMWISCU&9-k7c6q^2*9bz&pi9{T&Q=Bo2s z4V7d&-q4fe;pvD>QOccbnqtP&ulJQeLSqKhJ|k+`NAD;kLw2&{0vGGhRPOO@q(wzV z?;q{2W=bWBy0!@9Ryxn<;kMdZXrSMzr=8rMbib{|v)p&i{W&#aE6%>_eTO$oSzBNs zFaN}x#CfpD{6j4TX&!i$_g9$hF5<16$$J+-tTedl<@?RD=Yo8s24x<^E{e+8THUxd zP2mg_SpI;(7{%#pmAJ=`AIHUQ()3wXGVW5OdN83R^n66Bxozaj zOP7(ay*!Ug(210*@-(DvgOE2rJV6|s+f2721p=^0(N85oe$ ze_)TRbk-uWx-$S@&cuMVl8~HD^|Dy-1MdSNR(3B|gHr4M$>!&c=hyhdXoMGAdFxl( zWZ%nCcU8z_jL%0VWnYnbR8YTPZ=3#(E3eNC;owgpkhwU+ZZyxCd zK0xk$Xl5vl@VZeUk73mrIy#=Syf4rS9>-79qG+*0^?voO%r}uDudhJJhyfBS0sYC$2si`S9 zRaHq&2aSD=I^@dnZCZd!;I`7^wH7JA0~aWlp8z?6?|OL%y4ih|kdgL;F?jKmdK9IzyCD~Di-oNGM)0jFe-`arquOoAi+6vN zk1HNilu<}e%lzc$ePNlScI~xc^?RMExCq-?O0H{0(ciy+2a$qpY$lYj?4b3LP6|O2 zQD44c`yqyPsTLh{vGpv^3@E$~;*XB5oZ<2bymay6_xXbQ^)|&hIx;eoN)cvzOt`^Q z^~Nx%{VNfE)14@(_dbGW=~3bh>bfC3Thx{1y&YoxlwQgxY`VzzK|WfSf_e{* zzfFqlvRZnis>%Ao;DUE0GJb>o|SuWt0W#R@c{)0`u#&Ls{eNqDU~ zM5GneZ7a^DyC;v`(lpMvy*$}m)_VCKJBq`-OHS}CWhm@sqem%WH?-etPB!tDMl5;U!nOC1vvb0_u*%TW%3H3HO!M0Js9 zGxNGK5Vm6Z<*2Ln`97odYCY4|1E$rNB z(M5?yPtaT8udau!&-Pxgb9%vfOLc!^u3zb(^k&dWo!c|46tOgJjJSjlk-t5$Fk0;n z*in+rm8g-&4`FIiV`4TQLqRbn-c2yv)fNw`c z8+k&q-5fI;hCY(v#1taox4$cBGmtfMb~*UxTmSu6Bzu%XqG0Z_^~)Cl)4GkX@wEvz4iu7}*=^Co#c;QWY#}EYkE>I!(!IbY^(x z=R4xHJDF<97$H92OV7ZMzCApKDFN~JHmloX;QC}6gzd-jg3D>^>y8JQ^HJ(Abw|H- zHGcKksvcLQ`t!qIA+%A|yTK72-MFn~;;Q{^EiC}ad2NikuU&mZZ^a1smemP-wl*=A zikS->tgfHvA*Pszc%Cz#Xo-{ymym?(tyas`HzD39atr;_d1G!ROl_{ob=&&s|EM&u z0}E(kY6>}N6(szo>SQ74!Nur8(~qY`S{FyFLuJ5k`b(`-K`c4%G?O-y)g}r zuM!}F_qO8B&U&ZhHfV~`YqPx{KVx3r^fwD7ObYYMUg* zHfE8d+^FbsyjdzyrPJ2l7pntgNiS3P;h_oZ*TRYpw<0Haj~T(pbccsyVrjD`Kr| zPGl}$(bnupRdbT^*>*``t)#q&;t^vx6Z5b{9)&{PxX}bzCnqD1sVK;sZ>Wvu=?eJ79-n$ce}7WMeym15 zl=^W%fD+=FXhN>tV?yG3Iw4yWz*JnVm$UrKG#rM8u<7Duh@?g8a2VIsjNKpO?Y;RD z%|4SzR@0mUu$cx&ga7~vXDTj{a9u4_#7&MLG!2swGI@{BBkeaGah>kh42#-d!nd3L zx$Zn87e5shm7}Ahuh=?n1GtW3$BuzxdGhxo4G^uRrM|g4TGiClG=8wZF*rDA@Mu5i zySWyL8r%t1S5x=~2j=;JvpeF8ga+6z*Q> zR;qHHpYANK^f59XtN)FfTtNA%+t%|7MB@#Ra{cE;;i}KVA4wq8`Rh}{57C1?ph3SP zAqT zCJ_)2KpL5?mEH69!Lc%2WWWmKs=%O1vpiPCv(um#Dod=mie`Ae*mK2LIW@~Z%a0AK>LPGQ$MtIRQ@~e(91!cd8uN(P;ec?rsQz`iMO|PPDSDU>}~CukKBNaKA__lv(&R z(RtB6?dy-`IZ0TQd_=lScK+^@8aF|W>@Q&WET?rrAj7Bde$ICcW0FA9UP z$X*I0WrC{48{a)o5kn1)NRM~)(HL0-&H}TM*HiOwiWy}$RKhn-5&ZP2L9J&gDdW-Z zDpwnyWYcauC8x|y`RVgwZkJCZ;V$UG7$HV!8uHiMw{Ms2a4M5c_fL_mIr7`~q&0)M zwypIC;Dal*eP0RE)GqS~R8)J!x1?#KZSUu+ar;&Sy*0VQ8zJbeN;H9jf&vIDfRHmC ziDLJDegSj_q6+4(5G*|l3etUJ(zvxUBYbz{jQ>$`w>ViEK2q3C?WYN0DvD!{8ucZ>207}Wdv0OkPI9}Y$q zyj;D~ocASt?vQ&xDx|2Q(%#mVaK$mzJW>qs^QeyVc^WUGicr%%IlTUIkRTu6DaA;5 zmEDoOapMNa?}=0A!XqM5oUT{=G1`wFKZY=?4V8gqudAjA6czvZ_82KC>0;~U3bUlB zsHm726ez59f6m&%U~iGK40xJQ83^bghOPH{GKN0@cz({U0v^1vGCTbT?dHut!CXKV zU<1L-y?Yh1G=S;)5-S_lE5Cy+1}kM!t$Ykw=`%F@6Z}?bA5yL!hy<%Xo0I-(_z$M- zszj(0aXYY3xsMXd$PJ$##ra?1+hy}iIR37L1Qarl_Q(7AWN+w!p0LzsJ|8sl( zTwL4i93vO`Zy7{ZrNfT@uLnPRwGc|I`OWEt#CJJaT1;Y<*J19s$)5v^RMY88;BtH$7A6{ zHNK&hN?0{aHJ2|WIghx+tFZn8qqLqlTcd=PF>CV4!_!1Q?wX5_F$pC3eRv4Yh_N?j zeuDq^`;XCsY-aGjv<@80A*R23xK)220)YHGjC`_z=!oB235GzD0ny1K|HC5ze2v#X z{T!n#qLIjMzzf5aK@8G=pAZ9Toq!774?p|YQvFD|8o~F1Ui?}6c-;-8!t>`G2(d+K zP4NEXJVZQj;3T#}kYo7AqPN!N{(dTvZg7N@M)$DmUC$x79&zp*EsQ*~;v%Os21t@+ zvd;jU!d(Yv26*<)Dh1>+W_TbK!EXUh+gY6@r=$eqE3!47dyAKV-3sdo!08Dh>PP4; zHo1GQBzgQ7P;&CoXW!|aTzgs4p|5Ao!>y=Q_=WX2mP6$9c~Px=y;3M9-5synTb^o# zij*yc$&YT_d*H9lTIFy+n#Ygp52Z96D?Z4i7^_vSTt(8gr+ca{?P?abe2aRBj&j8= z+bMnc`0-Po?n(f??kf@IB@z-==JT9)(~XUqa6vB?6WM z@R3z#-erGNRY+(-@65rW>5#IAJa1RZZG*yod0*~ZlCRu6s5-@a`@35&U%q75%s@35 z`>bgbNO`U2jS(qH^Ch*lwY_+8W;Q6jeEsqh0&>>Nwx2Hv3a&vlj!ebYwmFi8Oww%u zrBDR;3cw$Bp9@o;PEn;w zxv@XdY&49lc!BJWqA_A<@pW{oosd-UZ7V~={@zYC;Q(w8vjYOad#^eSD~Klj zT+tq32~wg$uFl&Agav)w-+IJ=!Wh-RRanF@NWY;E;rl+J33cRhv)2Kg@xq-eHGcvT z4(sQvDo==Lba1aZio6o>08xNg(vH@8q~Z;q>^W-cIgzm$t!vTDEDR`~Mu(gVC9-J# zih5BnYpzU-Xg=Ad^I#{j9KDcM_x}9;Dl8*26R9;qPCeI^^3Xi?xKHE`kO0;c7C+R| z7xVpZiq(%oK&9X3q27E03Js0$-OCn6NVO8mciZpnZbNcW>$Nckv9Ft(o1UH?QmG3_ zB?#`NYH$7+N^`5)%`Q7nYp#*f__CT(wEwKY0D{pcy;(#SE=kF?fou{xY!X+&X~f6- zA6n16BDX2T0X&A-D0t-khY!b3oCwi_cwz+(d-}8_AKx1&cJ+AN%DQNQ@aVN+5>HnWdkb*^0V8~SAgTwR^d4DeG$H$ zTVe5m+Gk+L1Noh8pUaLv0y-?T6 zEXJo1vJKC;#2ZMAPvKtJq72e@JGKsp7EmK&MS*(+N&u8mg-!=`mE0ZGo4SifPC;Q& zjaSqZepjji@*6o_7k~g%qGP~@Uy0HTei0u2Qp%|2-kXc6Zaia_753wj*?DggQ|-(f zv&JAVowy10Z8_StJW;oWJC8!NfNvz@xBCT39tXQCcu8_oi$5<0ydV3%3UXi*x?hiv z1e%I}xm$?U;GDTofM!Y;zT`4I$My9XzK#O=%17`3X<8<9NscZNJ=eS1{7Zdy+=d-C zMqZ2_|6l?95rkO@)s3f5pMLrB1*v4>Qw8s(rKLTk5|{>d5$TB@a_Xpl8#9VLNuTZS z8*dHDt^z7&1MZDbJY?G{Fr1!@@emIJn$VoT>ns#rfBu~9&Gh=v4EQM#5qBR+iqc)Y zlb9AmqOZSzU9z|K2~N!y(lHZ@&%g0Q_ArJ9D1Xp6E`ki126ibFf&ttsPTtFVTbwnb zun1Y!^p7AocLZ*6%_YQyL)3Eqd}2a^H>eKO)j9wO2X@OgLj&-gg#r9)&cBM)0~7Ax-~a;O{>l{uz;|yOsCo8Kb&^$V(J8go4$2c}t^5Rdl%SxX zfPgbpou?HRRsCIeHyue8CxoJUbsvG#4y6_hE@wkhl|vvaozH9Cwwt;;5Vi&AOc$7# zWQaQtSxa01_duTNxxQd!l#$}Ku_%9UxXccaZDTa-$Y050ehDyCohzcLjA}hy$7(#_ z@+XOjqho-4^U409F2?HIZ_rRY+RA~9E6U5JJ?6=j3kGn}H(eM1T1DeJk%)+h`>5-u z606?3-{yY++{DMnx46taCjuedSUec4LrCvTbzmNp{GlSm24p1P-A#Z&-o1Ng7tIY+ zNd#lW(MP>{kLQUcR&QEcb9`b(G6MwkQU{jPAtneg(w99X1A&vt zXjnV-n-Vp8!N^zltScMBF}rv0TRhy-(z7Q%mY%o+>v+3E31W(aU1k{^qqb1H*>?$gh&2!<*jF&x0dBkHI|&F{Sy3-OV&+v`ssMB-y!BP_)_)H*O>$~ z`*Sl#2LR%_zrlOBMNHF)#JS6{kS**c3ojg0Yugn(x4ujBMx2(K)a9`9Y8sk!yX)-AhxPI&$z&Iee&24eF)KS zBAZUEdx-4%gGPO*;?1_N03}FM(YAw*cJZ+`7MDYqwmDKcF&! zGB^j{N6vhU5N;F;r}z7M6q{r|?|=R%2i`F}|8K$gA|_k7+U~Y`@Sj&`>JE%XecWRc zVj3SG`MoWe`eN_O?N;NzZ}`@o_R32*2Rhr4ti4GR;IaP}E^qlTI&;&o^E{{E1?0h6 z&Yd`&HIPsG*Gk`Jeh;|RvmYEQBR=J#-sk>v=>Xpo z^P7%q%dP)es@Q3I9i`~$Se@C^)K0hIkq)g*F`rvr+2HB_Ja!D{EiG^FvBmEHe3ND% zzjr^x)NUjYIGn#VvQB1U?s$D;j19_cB73u_vtG!an@ZTv=gkGSclUg+9?VSkAz4q|rt)a!y6>7FCsK)nM z0rWcdb=h{=lT_FCIkdX;?B5+9Bl)u|HL6P&3X+93u)ZuP3yXG*l$3$1xUGyRVu>_~ z3OALIpXK%0)PhBy2Wn~%7L8w+fNIFn%^sR;Q|s~IgfC) z(}Q!g6o_cEUK{l{I6L({rOLT$$gS}3xHZ;a8XuFED5BY4P|(nz)tdplyX6KNL531I=G&3K@ z_t~C|x?ZyI=HqLn@2k>>hYNKf*ZI_q0h@uzA9RLgxSW@6a_gmaB;U%;<`C86HpqIc z7W0x?s85>5P=lR)=-yx?%h<_*&u`)wK{PuOd*!>f*SS%JN00W6R1V$GQ0GtzRPsp3 zxtSn_viLU9zqc_g+thT^Z}Z`_x77Zf=TVwW-xIE+cy0_>n~m(au+RSL`ev zWo6!qLWk1E9;{Z*^gUP0;nu(G6~1V^fAm;syuxyc>iL3>B9CGUkC~1Jf`M>`zwb1D zQfI0~h_oX?-EaGO)t+0adqH_2j=BXww12?C8jO?2c-7v@I&BAXJ+Qk;nNIA4y zhAWHS$~QHu3))zlK0ZNg{VMp=>ujf~z9MtHzN3iNAw{%`XKl*h>65HKs(~u9Qc+iEH)&%5AB}nEh9Zm zKai)ybNu`s4<%@OcNim>ifg2?;=04aRi_9bI!S9UR#hFCzC$mbHQlEEoYljN@l96D zxl)tILdISbA9lA!Cq85km6q;^T6X488uzU>K-UIunC98zL_IpFOBv0SiiNKYd#6)c zXRZ4++T(NcxM;9F0r(I7`#u?a_2WzJXK^PATED2er*1c#vBmON6mTXQ47cEk+WHMY zESW{;fbTQ~S*mXL{UbwR(>2U&*K3!9g|2gODbpATV64Kc*qF1#7H>K_>ODCxdPcg? zpk+eq@!n4VG2*O1{ErMu`q-XK`5+-W$m zv5yzgjIRuX2w?RU;wNqzfFa+fm*Sz6*VjAtyUN7%lgi4<>GqxC0>bKju|r>h$nVX! zkALlq&$RoWKq%8^;IrfSI&NidXbp7&tr{4gbK?T~d{m7vO zM2_-eFq+Ey=L2f-&FSok=W5nck2zhdTw8bYO{m)G|2prB7&O%>`lJmn8L9U>Q{DD4 z=t|0Fr6U{-MN7zf57&&rwHXz)8@cIs(+#gxE%*_5@87|lj^xKM8!3TR0uhm+BFx&n zO^A_@U0nV0%v9^pXi<)KX*zHGWp#*7AD3IFd#?9U;N@$`>DxPhx$3%>Lvh#0q8C+L ztd?Apd6qNhBc0~BPjqz9)@h996N-eER|&$w>s*l)*$JE+ij5y~jA~0v&K;f9(&|)B zYqV&)={j$%R4jDoB?VWLRH@SbUtYHwO|`q#DK|pVg*d3&|=PDEL|l66j**;d124+Vq8)!)WMhvSnUDM0?2r`oSX zeUvK{54n7=kX2s4EK4)9#KBaCE;`lIg8#O^n^v&rY^0(>w@|HaQ5wnCu|KYU$v-MF zADp{ap^KHeuSJ%qgN^|GBU!k#NzMI#E&8Ei> z%8+~3vWIQ>M@6NNrUSpCCi9z`qvQHfGoBn4a&`r?w6grr!E(KF9)rK1x)dGzx?CK~ z(Rchqys(YCuG?bClcmookC(<0>3hlxJSQW>>8VBdrABIalQPlinKsoYKfKSnf4_#I zEQ-?(x0PMSzGb2ot#F(9xzEnBX_i|qASDKQGJ%9t0|E3@It zfS-RZJ?vr6^VhjJ5E5W6Sd)>z(#h+ZjqxC320te^q5qVHle=eaNVdf8miyCO%g-{Odx zdaPx#cA}V#Qthq0EW2j!A94K*PDrdWv^26TCi_>ZLW>Vj{MvT#pP6fMyj4(ON*QrXSA~;GYBaHVlG|N?-du+ z9!^vH7P!i#8|i*$0ma?CJT-*jrxBX8Z!oQsbNT9DRtjtuz2##kO4_Y^dQMVs4Qo{5 zW0s-77cIAz;b>dTA*~4!D|BiG)&(t#ElmbozMD`$naZycKTS{O9u!jD!NBYRn%w;K z!9}_GfuUnQnv<2zcBdCQ|9WmRqlf$1Kp z^vel9Cao?pQbexA@XaN8U9K%`Z)Zf6r7xd>$itf*Wkk^*|G}~4?>%SQCi(iUaf@MN zY+gwAVr|_Z!>Z%k*h3 zWKV0FTjDd203T=f1(@=Dy&>6!Hf2G_)IH{nV#}3`=kpr@ZK9Dc(o$b(I~BPR&wjZk zX!$JeGW$|^g{tyXy*JW{QUyYH+ma`Fj($pgER?B_syA9&%rCp!uf@x)vcvak@~HCm+6g(F8vM>W(mJ}P?a7;?hq_gkY&#aOg+ zKJ)g5$7B(e1hHvnz099}$xDmYowHiE{ZZ}|X)tN=7AGnqf->p(Df=L0(6eG{^Bc`B z)WR00NDtRgiJR-`+I|n*Ul>=IeYFZ6o|WH&2OJl!dg*086Wn@2r1!}5<1%xz6;Cwu z055bo4cCfEr2+iV#8e~u`~J+RU%!T2vSyMy!)mOaPhT~W#CKPWSyi_i`bh?82o8>L1awgN4^?V{DJr-Y&6EApLD0t&tJ@xzzF4J1bnd0^3e88& zkhes%F$Y@oj0QDVq$bC_s2Fh(owzHCCF6P)^+);4{#E~Vul@)fqqRt5X|#7`#=A6*w$X{zH{j)5)_UBZS7h;P6n zMzvBKn~f}fe%naG(W>m_sZ#p0q72KcZ;x>>eycL4enQmi-BAS04M2IIAMyiJSe9jS zI@NF4Ahr2qEhR8g7W$Xiy>GNyDalYxXE}WTU!D4g>isWS{BODZ>*L5$|N8q-bpNlu z8uI_|_<;Z0p#}d@(Z3%5f7*(xg?IUVe}xz$BM5+Yy)mOP8Y3;avrz1WIoK{Z;Fy(P z=@dEvnkM&q@W2$=xpU{JsC1#ap9uvEXq%8N+|?-r*3?8S8zPxs-2Gi=65U%SpmupM z#p_4fei9!a>5fzx_@176L0B0v8*kO{lxEgG8Gl-`$~UUjBZlgIusHN474DZnZ}Rf$ zt@`b#!BgKRCcJ=O2t8fUt1vjmIR`i_bbJGa=J3T?5aB^+G6?Qh7_vr)JP-{fdHqOfOxEGF)8$m??;aFdR&RnSfczp|E z0}8}+U()m(RJ8PRCL*KK@2T?92wJ-guW|i7eE`I5xfbbCK5_iG;>)H7x=8)_8zTxC zBfygXyai@CAUruJ66YFLmx)IyufBppbtKXqf{}ruPEu0TbF@N|(<-pa#eh2rTL=w3 zv}ewgShQ2yt7^SOi?A&8=SGX$jaIEeSBKQjf`EuA3iTN(54K8Kw=#j6{xOXH00`RL z{f^Bo8sgD0W2ZRryL4#wJhey@DUnYB5R~|e^{}`*)--rTkWenjsP~VQKO@^WeSx1U3!;GX&jp7nI z6t1UYm z0@Qs3lKMTMku7wt!qS{4%&YAvUTzG`Spa-)>#)xzZCr8-##z{KqPA-oL(s97memJF zY8%Hw&_J!g-tss1Vzh_W>=ehI)z#IbG`+Q=i@Jl1T!ks{5;{LwiRKNfVAnQ)_8^n;JL;T+EZ;ULvI~0*gu>qK1$;ZL^>Xs zN60*eFLatl_y}Qq;3jMYK86=>9_o4UXoSSXu;@HrtuxP>!;<5vC1hkP^1eL=;LuBCNQ%@Sm({+Z60Gm}!MdkbzhcEH=byoV) zd+SNP%wKxV1*>GXI!r(PnzkmA7u!efWONVvAuzck<=zg~jVNB;fuzlU&3#ijcQ zs~j3R(zm-|*vbsB?u#T!pskfya`UCuoG-A3K$`)KTR)!AI@9lBrSWhBI=^972`9P# z+T`=O${Aa_orJrieZ`#=sh8w(kM3Ozy@~q_x{xZJrl9IZxih*t>HN5!LLV$y10nAj zws&CcEJi9Ek(Popwe%{gs{0^AQHhD%R*mYbZ-HV2rPGD|9NcLR|mPReh;eB9UT$B#hs9xg>jcqInD_SFZ6W z1zq4zIpKS-w;}W76!hENh2abj)(YUKaakHOG#+EWa^(su z>ooLc`yvg|29@8au4Ua!@xRdqjF*Itq?nj1_kN=H-b|8{lHT?{+58X+q=@JS?m~TE zAqgtpt{V)UNEbMGlS};kJyT3bd5k1anh@i?U9#5_ z7!<;c5H?~bC#Qclx_hTmkv;+z`Hg94dU(O;VZm~J%2_kmTH#`q+n2XT6iR{01-xZKdr%@``v9xhk%d+D|a6>1v>vPG$KKSkBTm-zo~gdf{Pv>pfw2Qz;A zAJ--3ll|AaWQb>zX`EK={dV-Ba0xA-46bivNhT-Wz;m7D)G66pX1xv*T-;iQS);*O zw^G$_&vEX}LVLR|QgC%xo>Y*KtuqRaq-I;LlV@Q=NkfS=y%K@%yjf8_gZ0qKIlW}J z>?i;%_4@s_f+V2}-I6d<02%@J!cN_c-#vz8KJUKJxMeFdx=Fj(e}Pa~9M%ness4P`fx|`)j~1NhRal zfHrh$dHd%kNoY&^=V-_Qkb3;ImV9>NBD#{fxmjJn;?7pUlbqv3Y55CEEa$@~NzSI5 zn3jrJM^^Hnc$(Ys9D)aGt5+$u{HomyqZLKFzP{7$&*5N}6|_oibEh(}b|L(C=e=+? zv)?+K7zIuEM>4XrXZwrHm4-TWa@On$NVwYQk6~=Nq=RdEPnS!`PH#F5ulAtEu$Xm}P)I zgc2i_&MF#7GyPWxNkC0*jm~R<@6{zwR+xYQgzyL%XbZnAX$KUAz8ukI0-a3NLjvtM z{@=~_)!RsiIOo^x>zTTR?+qtqR$$l*Xk_HaNCih+rB1|-m$?_ps!SS`jX!}ms$eD& zC6+9cFX^>zv3$*(6m)z-Ct`))fiGzsaQ{5V$PhQd(7me05^b%wIr&^ILpjz7h%7LB z1;`;G@4#1lAjyMJ4FpSpTDOCzidEE{lxs_f>+9_7tY<}ns#BMg!eg*xt=aAGN?BZP zH?^KUla`_U=1Ki_4#n#*jbMG;<2iC6B~5=SKd3^8g<>yV6SU33Gpb6{(9lwT%3qtb zb&4);wj09@Es8&$Joj#x_bE|waz0{^T6v1}86+xwBtf^96;X>05Kxua*D}+BWUZsNbF9UAF3i zCPIlch&VN_4Oe-I$le3y127|$$Y;kxS;(_rWu<88#3}V@%C=}0CP2q>owp(@v|A&5 zs-IYcY3P$zU_Bf*b1;6em;8vnJ@KLG=lkA8H#QC+Xd15QUV~m$4=FbbP87lzIu}NY zt%)Cl!CM!NOyZ*^i%l9&b|$3XfN>x%c%MO||JEuMOsxVERa+eky<3upqOzmn)|vIX zxJX%(yAk^bY(~)PJJH98`cQk(@FaL8pU!xHMmONHlMi%aZ{IFsoz(80c`FeIF)#;Tj{8`%R9}UouX40}#VlT{f6X=K41-!Za)c%t!=x`g z>7{38Hfh}(!e-GWy3zPN=DbS5gBX>tWTRtCC;AL<=O0rVk^H#C7_-Enov*hxY+qm6 zK8HO)@4de@)6THW01PHpzzApY5gHfElHNtzcPINciVOEYMN>1)@nZL>w0 z^^{m?PK1|uC)DV_?(9m}cAb1Bf3nP|P9MSNN6^^FC1tXE+pAVgaWU6^;k8x_n>`DU zieh1f5dwR&(B=grP|77B25nb9X?rDD>ssVIWwL*~P)*e}fBiB9|kQUO!$Mj;8Oat7SzkHisUqxb95G<-vKQWVCvPcQR6HQrCH%pvtzD)L*Jd#$rOsG-U#|v1h z%E~@+WNXuKcgHHKsVxDU^y14gR{#A_Rerv^VFb{Y>vW@j%g*e3PYzNmg82LYG?E~q zbMKAmN1&z~O1Vrnk+K)_ogY(%&L7A-xg;LlYXjpmH4A*g_kxBw?Eizb?~cc^eg9Wc zLRldSQ7SuR&+NT-Rtedg5K0pIWt9LIT#_whcYE(9W^Z&W=JsP>Y{8`>3GEpiEMxxCz-Z&296H<%J>MAvWV zBGhUvsR~hzPqeG6jmR4uUm2C(;>P(&)ggo02#vzZQld5)HncPMy{~|pWW1O73Y*_~ z{t+otTY#0^o2v5U%lwN50&?>4xSBw72k5*QTHsxg5i$@coIk;GF+WH0X=G(7fILZ5 z)Q8Fe)E`!z0`Z4YwmLc%lR!y94pG()$c~~Cc%pnDUn3a~>CmovEy>=;N6S7_-I;ul z0osFq9>HtYG29(ha@gJ_NR$qM{QCzWOiZQN-mhpkiK}<{!m5q}S{eLu;*ykh0Nm{Z z-<%)^p?&b#yUXwAwksxk(*AcEgYemxG_mT>Lx(--wEy-Zoq|9?x8#DrYwVHGD_C|x zq||u|D%1c2#tlf+$iK-CrsSeliy#MW6!$yWN%aj4X+zadSbXTF$RB&D z;Ie81Ld!LBl4aO!CbmTQd#J6S@cuv-({n&@J^?1D`#^KiE_0dfp*Z038MJb(NY%)Y zO#WUe14`ri2GGg^qu<1d`9f-ZIG(<)#thKvq=R*_$_n~9LAC%Sf5r&7srI&-N1k0V z^RC1We~MpeH(n29fKzJI9o%DgZNfY*61JWzda6};xMfu-|HA%$=hFpCz=5s_sw&#WVW`k7*{Ax!f-5lQPB#9N(wL^qZ{MH0s$b$FL}g|E?SLuNTm z$&kBcgs!|CgXPpH=6Jgnh^e16BS;oBfbE?WG z6>gAKZEWn}t4;a`ZhQy#MTjl?nj7taJF9kip1!J+laqtP8XRtuVn)D3xA#batd8RD zOxpm*5+>gL#)v7MGXZ+9gxWY6c>XWj2Rx%(r>1w-#>l-YVu(K1H!xm zbdKGZG92980F_SBB4zRthXm3fomYQ-hMXN3P)Bh%S!l!pnonjL zfyN0DGlf-tods5fAV|iS@s$+9(I=${b1h46+F700e9W>y_5&x}B=H+iSv4Un1?mjD z*WRf-`*p2{-Fb(NtT)74T&GW+ z`7m-j*M*poO(l#qUW>{&@@&LpxHBr|lOKoR?`Ln{pn$L8#~o{$vqL5n1WX)jbW;Bl zNrS77sAxT<>o6f18Mo12&mjHak}PUtNECF74>1zGH_jd6hW)QB4W+1EDIdT^{-OgI zx}#1F3me-v_6u^D>O}LzsDR382bPxBgYp?#?KsYmJb(A-H@K~4T8|zj(`c(c1Ni#q z#>O|k7pR4Xd0v`OG7V{V9KLma^HP$K>36kJn1py-W4BGaCNK9hNFUL?su&j@mgDd< zoP_DlzsMQHy{*$C>;Xc6P5R-%*769yMepq{XLrP!J&hQ_01d!lt=<4Ya|l~c{6F?E z0s#4A^aAb0UB|N-s_uvHWBS{W&bZwdLLn9Z> z3CwRJ>93b6tsmY4(-HC!VSB);sk#Hfz4pjVFHaH(eQ;?QdWMmC?G{i7>1pIx=sTNUq=A;0~ zh`{G+Ywk3?>)YF$YAe<^;j&Wp+@F3KgeaTJ3;kIfW<{}P0iMR<*pus;F#wlLNq%fD zaaw=`2}#gwLP)|)#P6n1(IY=nF9-&z{DffO&hs-c%wgfC%ppA){@yNWt-8w$ZfepH zp*ZOb4jy2*%&*wWJx@yNQ;L4Y|J}#y5(3`>C2;{)Y>aGoFE*8PBR0r;@TXLg{e%EB z0nMEy=B!kO9+!tDx8L0e3Q&g_h7_^Z;}KmClx@7Jbd%)kxU!gvk7ll!*=ZY>w57Xl zEPr+(%YH|>9`CZ035fcH|MJFMw%_IJ9Vfm}@)!LEbwX@!OCv}isL!u5Y9!@s|{WV0MZ2@p;Nm=ooMZekW14XysH$be^Gz%{l8wzxF_|1UfdL79Ti{E_y8sBy=_(z`wc6{(yo$w8xpF4 zaGf~E_lp8j2p=)5Pe$3RXMyb>VDIn zwM0(z7Eq2}0GJC2@BQvQZ*;OBn9@%l1W@I4#2Ms^Mn9^ypKwMCZ&Pgu4f_#W-z<+8 zjOOmpWBT%mM^rV0%aET}r?T4VcS010ltNVU%RB&+Zfo8zwU7)6Pd%Jgyr^Tx=e*jr ztggeBNXlGPzQi|rsXL^x`UlJTSj@H*Qgj+4dfvBjub-ff%5O2rb6UELuwQ6+1tbhl zSddsEpL4f-iYS)#gVV-3r`lBG3(YKreL^RTEwfIYQI~v0dOLWgLoJZPywvOloNx!u zqgH7BP$6N+RGQyVys6uej>5pg)*oy4cF;jr~c>b3n59wz} z9wmBPfU6P0gbSQio1_r{5rE-2fnyp1=&$5a?}~SxLgeXxo`T4Lli!?(3S1qC2qK_e zmVo==HX}u`iJr}Y)Kr|uch?3vbNEOY9t#T(mk}a)pIF%6FRVy`C$tva( zbDVwK(0QpfB-t$bMVqZ09K|;KJNCyFoy`7mWF_eaz`M%6d_=F^7H;n1Jc;d!q;~?t zfUM8d=sB{PSWO4pl|Sjt&I{P#k>)+Up4(xQubBX^1X(MjAHGVz_u_6I8y)q!4@uCr zGPrP(qmq+8fkKKG#IlL?sA6>`s|ugfFFwB+E2AYp>c3tdfzrXSki^LN?*a{EENklU_$c6!do z%@J3(@i;U4x8~}Zj%L=8YP@_7kB&|!D`sfsbZ6hD6$MZ>vxqH||9me6X#l@=?Y_Pj z4IRnJ2xqGjWYdmO(o5*j`ic zKPE8C#zkc=6-0!5;%yC~b7MPmrtg!);1;LDT?B=x-N>)~;+*ALzKK7dv#i|hZvrbu zu{?>^FRs&pm+%?Cb{tz^pnEdjo}?hH3`<7x^7y#Lccz?HKp3ke6$QEu-*1pX`O>TP z=6QXlM6$G>8HL@eQ-$2+mwS7fD5hBtJpapXK$U7es7E$H6cU@mtWoUl?DgidPGY{o2MInty|u^szjc_{1WTjB)yk3wOO=QV z#9mu17x5OK{j=(hNwR4hAnPUF*3aNFz#J4&j%JBimdylNMv-DAR^JLpmfqL?C9*Uf zE|8;ki**`A6kA91Dv2xvUyB7ko8t|ZcCjX2UHW>jQYATF_ee9hNZ4Oap5CI?YnM$; zwJ2$(lQ*@sMP+Hwd?HNjwS|(zDO{@yf0jS@I3*-tL5jDWI*QA>b*%b_K8nw|LJ`GO zSj$Y~Uc&G4X-eIJd&tg^vzLl_e1%!z*-dwa~BrqZzIwvi^l9w)l< zmxB_UE=pY$=ONbJ4{x+p#P0iwGF>`-_9YR)s^}zbdU`_9>9cjBVXR+)bUDrZUSbN@ zaN_B;e!IfkY_w;wZ-`!7KfYjATiCWd5UNlh;}W{{{KlU{3+B>uy!dSCga1-Cx(d1!jB`Du!|o^lZO z_f&KYt{gsP99+qkGLxmQ8E){q1|nci#U7;&JR2HAV=&AGh~?U6ch;ZafwZ2j1g@ri zzWXw7^^J5d&%VMm%jmNz?~`wkkffNsY6|L?dnZe$UH?g9FUt`ih$^8J4vumJKo_Wy zXW$-m#!y9WMFyN^>J6_TO{UuL#Xi@Cg?@gek;E6HzO?M!DGF3(`7QJcVSDHBzn^B1 zfqEuM5!D2=4*t2XNBecbB3$kBi30@QHSG@)Adzv{>QsHP z#m3zULEX5G7MWN;PJ>POs&Nz9xhjbC&%(ik0r>45gJu2Z>)MdF#!Cg`;aUT&K>J#G z!#qMJVBUOYgjH!o5AE7jKPO=d8tJz5F$%MIu_N1*WQ|P<&Os5Bk2I{lYs6D!g_c{CeY0+4EN&e?D`GntW89__!_HJ#Fngv7U!{oN!+0 zcPE~W(T^jQS@^ji7um;FEggvHXR)R zG!*3Mwy80b2+01W!MQKBPq$;ey8VGA>(SJ~(`vO{Io3I;MM~aJ@s|rdN)_WdL z!ld_76?8*1a|#|m%d-d@N-|$tiwR04v|r`IGrT1$*CX*gw=XDZprm87oqAhttM7^7 z)5Bt~y}+2<^ya?i!D0>fiiKdIBfYY`bt2kiCM1XEH;$&;*&`(NwcXfTBepbi@T#u| z%Q*$WQUlw9(GS2GMRky>oJEphfImhqU?iM=j5>$&=wq)+y!S^5xzocP;2U(Bw; zA@!_ms@&n!z%*psRz*x!srKbctf~aZyz}ZC`n-=OEFK*nrPSCN&PbdrA(CDoS`jYH z)U`5*TmN2!wMN3Mt(I0X8l`l0xYiK2&MdzCVoGxQkWCNkTmL>vSIA5OcE)*N{yuZ+ zrGBN87a`&9dE6dHfwdB2DWn6IcTI-Cm5UBT0hg3GU7pC9qA zwVal8y0>UHc-Oh%f-Z%4GyryXr(Ozf^~oAjJxRcVoFn)-{WT+(cPR6TcpDYCv5XLbnZ`# zKWNZ zVYKTuUemeojq8s%;?zf7qN)FSi77u)$Wsru$n$)7MHNLg#GPIES(OBLlPx*jL6~h< zn+27mj8txJp90F^IS$iV&NUE~opZA)u{<6n5@()jWDS#+$Ma3;O|rb3vc6oSxUn0WoVwUjHdyd1HZwYscVmJr3&xA3(qP9y&5CoR7L?C^N9<%KG$+KYb+NJ#%R z<^?gLf@(B%-|UXY`Z#oVhbVh}W|K*-Ej(_YUau_Ei2=QQ#U;H?GSJ88zrR+82LAfc}Fowq=x&&N90) zElIwD$Y10_dVy4w%~_HtpUhINwptssHqH2^Qx3RIGV^_=Rv*XC-_qt>s#QXtK$@t` z`D1^Lhg$7M85V*OFO2#gA#xinR%KDxgx^K#h;Yld4XW?E7n)kktM_Y8qlKY!2j|`@ zCm`!mC6beuF>#n@05>m(=2U`Q<|k#KZl(! zjaUpX2Zu(fjr;xeAKHh1T`e> z_w+q-Mz7aM;rCOm*03}RBqd9)|C(S{wOTI8ixzn3JNkR%ApdEBb2_f3!u|>|;t}v6 z5uc$G*H)}!kK2xjn0Y!HkGFod&~$F5+G-?IL*}BNTehwVjrM~F7P}6IRl+R;W%(|b z&a_;sKv_oYZQbW|D-e>)z+@CQ6qcM$iP#tjx(d01!_}3(uToMgI)UD3gxphqbD&Vy zb@XXZ zju+}OGmlv{{q)&hnBa{Hx|fZog-t-bXk(UrTzM%`MJGcxLxfv6q?EjGoeRoWM&Ulf@=tX!L-Wa!J@V`^I7kM zB#XXQPDj(ucE4S}%e%rPW-9<^m`V}NmnPcAZh&$k89S$m+fHqsS4$$nI zE|B5FnWnVhcFm2DD%ZQdenZc{YU#%}QBi(yzyPX*?Otv~VP_Ij_xC=l=5)Zr_i5vDzQCV-+7NjmjC{ z@NTV)WcgDIEk-OqdD8N;t~2JBc6S`uT^G0?_pnn;WIX_qP&Y|KE$fn*NHZsKRU+a2 z_QLFM7;RWeH7w#f%vr8V`L#neHtdMHf*fcmIGP%SZ#`JFG@@Um89zDKCT`Muj8zYE zG|4E8LHsgx9+mg-CKpR5t=abSeB$dxWX${J!g$3W>lN3@v1Tmfs!e>!8m(q-rMRG7 zknBYEkX0LoSQ506%_vt)#`M7ebOH<3N9BXt*_mAD!i>n{>e8%XYQw$D$W?&zyrnb(*ZDy<|fzi;qH z*vG`X)Y_&%&oo{r_vLz@5~_9G-WQ!PtU+9d^Es#$-#T)Jb>O4?CX|!iL=R7eXr*5G!G_-H2)Q~XDQR?6ty^Ms!9Pce&|9oA!sQgEM z7b_&hXCdEac~H1ss{sdRqxhp)(O7ETbY|a_U{P{hp;?YU8Clj$T#sdPV}tvalXZ`g z8;uuEc*jUddCP13EQ9I|MbbX!=(C9U12;nWpAIT;*s_HHK<#1p?dI|&u6PGcYvD)d zL2x2d3MarY${e#pJ0659ULa2(mMeahbcm5TX-{ zG(2xIh%P6gG6(gD`6TZ>-#c7!MlmtSj<{!y(s0~(f0o-OJHo853||N72XYHdz4Gx+ zQ`M=u%&)b|5+Gd>XT~4Vd3rCR9uHYsq!YJk>N6KFPj8=+`}}*H)8+z?4~gzE7GOzW z+hqo!rGEkiu4b5WPh1ayZ`W(YM}x`edvcsyYsQjWDf+Bbe2Bz#u`TKtf+9hm1;!;7 z&Gnguj;{-SY9}&y00(#Tdu$-ZoK$N7`NUwu`Yf?tsYt&rmEyGp2%h7}4D2sm0xw%fWs^(fk!oBm)Le`rJH-8xK z4eCXKQFewHP`>?-pw=TEP~e* zx#Mfh)S_v$LvNPLl8|KqaQ$9WOK6|Rih2B*KMDnQcpI;%BAjbcKE=kbk9>9Z@#8-LAGF;hbw?@4#KXVSU{qvR%=sV>?}Q$FFD7C@eVtC?_AIPAA54h z?eQ=1nTK1Uw*<+`4Lx!ULRse2QuXqiM^d@)p~8x)@s~^>e{N;YXdV9=8AuUzcf5=8 zkJi~i$^`Zh%ZWwk?Yxlr@iM1ZRcnWHD%mCQlx(=*<><~GQVf@xUuLTh15*6?=jUW| zJH${d2I_L*>ACuw$q|z=Li+j@$gW3QYoe%OOva)uz1&PLt+sb6dfB;^ih{=i-U-;G zpMb0lS?`odSD)@A#-;8}UkM9Msy%a=u+9ee&!0c@;5Y@WsVFCS9eY$y3!ZEnP4F~} zpM!1TZ+~ePNm)mG#QuVx;{6#>+e}P) zC006scma+pAS%`A2{-%3hUCj4oBFa8<9opA(A~HJw*xeFYHAApcw#=oP~5$<2rv#H zFoX7i3Kt+7iT{4MQFBX6Fh40JC6t~z=3`{^e+Mc0h#wJo-RCsI!rpa0SV*}gk8J=S zTNmraNqhkuI$EiCcs1vAV-^DV>`~zMGiTN-gMvoy|9${6GS43X6x7ep511l>n77X~ zRF&^6;+j2qvA)pQ*x0Bj3K9|^F=@6y$(X-%2~Zv;i}w5r9YaR8VkZ0A+8UH^S+L^) z#^CpnkuqPfGikUk4PAZPz=57%q^l&N4<3vI+VdC7byiV(x09JaX~u^n`H`vV#mjg8 z8nxNqqW|MuK=dL}ln=$Wf^1=7;oRjv3kG?8g3ExZ17_+^Nl8G)^1?e*75&P{ zpd!-p5i%oXb`O9-RDpX0K+sVMBgvQ0^8vp@OAs~~m|XoKfD*IiVBwNZ&dgj#-W&1J zo-_ayjb1S3Qc_ZFthDL>ty7{`pi{u}5(q!ARPd1S(ys52xSqnBan9c-b0KJ?<&t%7z}Vc z0l^WvQxLsEM(gSi38=$3ito0^ducdwq^@0b$n72F!1PwJHNAtSHUC zcS*o?Qp`f^LLQtmcC=LrEnmp|a^<8Ijfp#4^bi2He){wPIH+Vf-G18fRK3oBOPc+D z2w888PFEFVWc+|J-I)&1_rRNO03d}|_|$Q6aW(K_N8RY*XG%W}CDkBtjR zSIC)Mfk6xK2P(WCAN~xX%T*q`YfygziV7Ts^^cOnj_gUecg7gQ5_nw!_W}zbOBPe^ zF_buAtpp} z5@!Q`4`|aK!r*rcfBpoB3FevH-0Ce|&C2U1OCjKL^amFJxUsO<08BwbLIPwRpzw0{ zrEmIhI?XC0yIHx8NF#*mwkin}q-{ha!Bzz*1Q|8; z7{c&4QsLG`+KgQa5HWZL&LU4#M8vJ54^S-REj6IXq2G4nqCxNDx7o9Slo3`pN;Dqf z!$c$vW(ZPlfr+X8J_@>Qa?sY<3EZ984)2n7cXwfpC@3fd`ukh{K#c>QAHbeQwHndo zSHhKHKG}?$jI>Uk;!A`e9H-pL5<$Rn58E{WdI}lUqBrU$rlzv26?3W4co|Xrt~Q#Q z;qh^gmS2IvIRYcN>I!vCka2!PLu#@fwga=VJ^Sqp?1FJ{M=)94i0F&E#%WDpmECv^aO>f zGl<}pTYgTL#WGt3-chunU=cJ>)1U0I#Me`6E^L_~r;2Sx_J9u~{9Z1=9$?FHjRi`f zEDDVL)IWdzoRXUQR^83}bDqG&?XCtS%q5Dtc7xE2J-n0~>No-$8M{0PQX}A~gI>pg zC;*ZBsCgK~5vTzmr+t*22(pa`vWfQ2A|e2pN(%}Y=dGlqc*-+%nJqqlPGAD2^!M+c z|4VWqG82)<$p8QM=gU&h;nycZ3XuY6!SQdw)u)YWJy25|PY+qwOf=GH!JoK&BA-OQ zq}LP>`G~e4CqE**({+3_sM<0nL&Ix30+7VulP;Y!OTg5ml3Dv1?dAHE@fK$^h#pIb%x%=Qk{ zi~qFhg9ZHt@%`^s|K009t)8rrlV1PRDl$&UB_D9QOr~WHw(|gI-ANBI{?&_re>VEh zOZ@xtPqA?11i>b~gY@oS1M=+TU_o~pxqJ`a9j#0KQM+0>-<<}^(Q_Z%u418l!O!;p zTFMq%00hb~Ca4}hYyDk?6d0mvIiMB{fe^k5oPjU_7^nXNDI%VOs0p#L9V1AlXWi*% z4)y>GhcYBh!1}0P8D4q9K!h<>|K_};loT*elaP`+h=INYKTpuR%vJ~VkRa;pxc_zO z$B!TF?efbf1U(oZ{OjxML0c1tqr0{d6EZrHTdP3JXliOABE~#v7vq{Foq>cr!Joc8 zBP0%Z`C=WAknuzL)YReSM%NW54|SHQbKA@%arlm15hxn&Z(h+GFd=X$t4< zkZg}IWwj;rs(RVeUH!lum&WG`Bu#sJ-pRR%D%ble zisMv5TQAfV&@Yc-7!F7VC>L9ldtJO*_UXgU)Cc<3aiG!2TpsCrO|l#}&SsosjJ94K=b9QJ){3Tjr3=0kXM#iOB)cubLjI}d$iWMp6g zC#Ygs_Oqnjmyez2yTM1Rs`?442LIsfaU?F%uXoAZDvV%6ey#>)jMs2 zA@UQet&3p%de8@56qEnrk3>32^(HDYRQvB3`u{w&NoE9V;b>pu_=rwVCFR0dP$$+= zS+EO%!1(tzhB8CI$ApG)fZCseX-mId$BviOD6#yUDBE#HUEDW{JJ0tf$)(xHv952v zo{WZ{pSxd$EB$rxoxO!zG2z`75}JC3OWAF2dB2Y2LJ`X$n1`w=fg>u612bq0G(`nj z*@yRuG1Beo*h|czK&jF=L=S4UUHkR5ta=E*Nexak#zv0+Vq$;Nc*xa6yB&HfPkYiv zZMgar*FbU!Pz-q0NMt|^jgoJcbMY6mlF(dvbF~E)BR@YsWcpmn2AzQQeo27vSqx!b zyPqPWqNkC!5~x#+;HIGD*R*}3oc2yLYx2?NR77qP#3Ublqdg?4t#Ha5(Rs*d0yUSQ(gLgq z&{1~eFn+&2I5?1^_Poo-7X?yDy~OCAG)T~qWEo=Pl`PC;Xv&$A1#n4s9NYN5xrjm_z2tgxg($bn!B_=s>~>m*`h z)ZNx@cwJeON2dGasXqaR1-7aD9YMiZX!n}-RK2m73np%YX#vzn1I6zqRGQ_n_o+Mu zOz%Ija4{?{fkwGBQpt^MR@FKNre1zGE{kLsC64L0%!2!yJ$bM3zDZ3?IJfSuTeM!9 zvU_6l*?h8U+n}$i+~T{uFxhT5-ArP;d|yuV=noDqO~L)_iYMPbT{sE1I(z&bZq3p) zzpolHdceV_EBveJwh*}dZ}uLr#knr#_hfjqdG_IBBuX0?7yuz6(iMw3Sp{dI57SUo z^~vqPM0x-(Z8-J6Frh;4{K;DXG0!w=DPtjdNe9_`t7%mN#pa>mG=vl>{gSg^PS%3Rx#%t9Ia{34C4n7&u&c&u2))Z>Y!`97^!iA6DAD&}ns zVGQ4^KzHlJa}`GYS~^x%f=FMU?W(qkn%dsBs|gp?>w`hd+z(|uX;adjRDybQUBY<_ zgcm8hf0+n<|4Nj>h4{t1U)s{$;K`!A+Z-lj^^t(7J%8@md*m4OO*SHbAO!v_@A~}U0e#eR z?HI4h4PNRee~F@cr#3a#T+mU?!J@`)0r1S5Z7FJ%1ZlIVpxS-7HqYIHOl3YJCPu~! zAYtD5`%iS8$)ra5zJUoNB6Vmb>bvY^?2B#aJrcGS8oz@__*TxXr@GR8mZl3N_y%8& z+d9NIKtgm>f~Q)|+pYl41z3IIE3%_+F)-MwWyJ5Px8^tIKEnJJX{p=+Vv~EssRwp# z`4tQTSs%JLH01unN^wQnlgrE8^J}!*7b!O^R;vUcL6Q{y08V!YvM)q%0Q|@YA3C#H zrRzc-HH(4Ow^!h%JG~WvCphQ4k>1JS;d0)GPGPC%OYz<)WE9WmQN4Sqe{Zwl&3V}` z1|?SHu4gbrRZ%!Nknx-OKn%G1Mv3c;qtD{k@p&64-7B^-&sTV%&JM}wJ_bFh?u_S* zg|T*BTZsCKv*N_w`l_%i`pw(bt|SX01hcauW)^&=Wp}Vqr%(L{R zkslu2tt`pFjY4}U$n`K@^%$#d@+(a>|3P7wt{KE>wKjKWIQw%0N_j|quS&`T1%DW9 zjb7zCK1a!yud~eJR@=9J`STt2(kr2*jrq)wtzdkS+Sxkxjpe2*%l}1;b z5Qn%_R?*!>oG(5+w9}~6)BK1sBr0v&$Z|1 z-?>~Kx{G%?;Z{0}o-(`Lm~%vzHN{OuCl&``w{D0^9Rme%$mmO&>tfEc2{tu49?_*n zu#S|>_pGNxdtU<}ZvPOD`I{a@E^ zns@$|I{TP}I**j4>X)3)vj6ZE?(c~WuA>=8wMHr_?2COY@A|+DITA=%1GOFK?$Pgi z8sj#rq`dNx914mN!0;nE-Ba}e=Y{rW5!b^o0h)=X^)uv(O#J!<1(US5&Ve0(A>sxO zI2l=4Fcr{WF>W(}vl*@FfMJ1g8c8mxbea=IZzYUp7$@5Ze)tfa3dFu0>)~OnwY^=6B& z$5&j)OVvd^EiSdx0>)vjOWg5O(6$#dJ%pELoODtzk!9_!$2uCHM4n7SLp4-GfTmG+ z+Oi%qt*g9?t0jo-a3;+t@+#k>?e_h^E6#%P-ioim+QzsxJ5XkC44(57tB|sNY2$s^ z!Ba||X#RdPFORxR{rf3tjE>JEU0btt=Ak6g_ni4*UO1mGj_2r3wUd*khO=s!bO^4ct;oyVNwE`4rERw%>g0wK?$Mg7U$J(gr{5}_wEpt0GX?-k z4v@cJF%yS@``>qFw>LsSge`igoHA&vGgHY%T~xB}zXT$zXa z%$b|bR#izb#Vq-NjOjrWUiCA<>teA&Iq#@6{K>Kg3IT}dN4$9x_h_Yk zzDA*;-*Cb4WSf&*f?zF#67jXDz2#kjL>c26b~+W?J;<{y{_eDx4!PJl_ZO~>NKy0R z`_0~<%SgXneQIuJbw0Ne&VK~$=UMR>~Bes?3Ov*0m*uMer*vfy!a5C z++FYE?4O-!cUsO4XK@Y-5o4`N$gS1{id^>%V*vd!(S`qQWBE$f%1m#8Iv5QrkOS)8!k!t`Fn}nO(kpSo++yt*WS84 zua*_A8IIO+NB8tDeKGpZR6Vu#J!g;H$3Ys;s*hnNvFL8BFiaoeJNs$q~)`a%ybj9+Qrl+T;>er+z1ySXeg5Ded+;O*Lb}j{NH+w;GI^2oK zj=4FiQk5%xmU-o>zkY43yR;V`tVMuzxED;p)xKFGcCOeea)cb8vYT}0GuehFvVuk7 zpyzR#{R-L6eeL|$<~eL)7M1|-XrdUeS?R({J#!1ub~_PmH+_9^IK%LYqvB1lksG^& z98;1JprB){34ods4XS%L=5kB*D=qR1d5!J!f%1@2K1M(XxONuuJJ5q^7aSWa2sKh6 zUka(Zb93yRO8hx0@Lf~#Z{U8rX&WF$)uj&*GBr?`4L4U-`26=(-3hbFIRvoUrUPCX z%MAS!QRYJAQ-TFVt>htEkWS_!rLxuCpQk1gv?jwZe^_!JkB7WWhmXb(Rh6UQc~jO~&=SSV#Vygah!|JDH; zy`J5{n1m8-BJyAFqa*`W0lojgVo-gh@Mp~#WQSUM{+H-88pnZ~3=R-MkBSoWC$ltt z1u2_c7ml#Q^EhNf!0%#D$-8xpoQz&{Ea3EM)RVxN|mXw-9O<^}>2Kpc)a(CB>KvHrC$3D;PIY<4nzgRCIn-=HUN2AaAWAj#6zW92u;H4oU zdM;mmH0r~FBUcOidyOaG>Q@@A5}2oJ+RVDzk9L;%IUQ3iy2U94nr+V`4PrcK!h|x! z(({eXC(dtE66X=mmfJ`pu6}UgB5d`DICo>qnC_wA((Xt8WTE zqY<^|jFwog@6%eKXF)F&#BU6>Bh5@q?711v8Arc5b!G#z?VwT&^|JuSj-%0Q}yReCg4)-$$pJwI$!&5zzBIV5_A?q)e+U+P7c<$+7P4d~9{H z)Bm5E>Uab~7$~Bn18FY}=o@KuFu&-QCIQF6jQqmD3_8_7U>($a66iV|apN zVixUqkWRprlir}4{Bz|$ow(Ef@h5O4$%4iWQtR2;!$wSEBBF~|dH#Zf9A+><56Y>4 z%>*@fUBI1)bRwP5M?+m5VwT+nCT);#k5m|ef9X-%Nz{kZx|~qWx2=?<5g1RVn1m0; z@!P)EI~W*YL-2yl+r8fh3m4wo%S(tF9;KXv$$0&G()Z)>)s!5lZ7TN=nAV^ialnWD zmD(}yY7x|dZOByM1(mwi&Q73Sp=V+$YAFQE8W>X$+4c9-Rv(AtwL_YHaaBImXKR}eho3cXA{B%@ z=*S?_E^sp+nMn}^!fepYn*Vyb*{4M<@Z_uZV(1Cb@#E!s% z+8?rFmyqhUP0SV$r{BLpiY(Ohdp@8cG8&IVmj9$Pc<=)UQdQ4uLd^Vu>G8pgaNq~% zgj->L!v@U8>@_VP64?R={3n*(OxR$H|F&;bMYP`&$pyBiOoYrX}zdN7m=G36xP zjj!FksiK*!O_P^bF`tPf!h6>_7+zoLRPx?abNopy^t3FG>!9oSASl=y>H|r(LgxGw zumU=d&^2T~fJoe03Bj*cT_cw3@cI{uH%U#=2L$y-` zoZVs74#!#=vZB`pNyVZ1>LR7Z=yRfl{JS^W+D{QOwfX1V_1dl+D_1fkhhTnMV>BMT zqbKIV;y_{3(}t$ND;Pz>1V)X2O`zW{OE;ga=XgUK~Hu%XrhQgpqVqAE5Q@kJsM(?FM_R~I4ylpu@ z8fcL!ltQ22e&^^CxK97jw7k6B^b;E~oM0aG&eQFYfSgeLcxQYH3Va+bsTpODklOP? zsU8H{l9gN0>JjW=1hJkDs>_B~Hj0}^YN&#^emp9v$`AJ+X0}xi!`1AH6TEWwu@Ntg zF4WK67t9CA1o+Pp_k|qx9jMXtSuj66{Wj3;bptb_Exk7IH7o%oz&9|LjTi6M5V`MFes_lge^K&WLmg1Cl56aM{+ z-}m32!A^pIAoJnh$>)=w6EJzZUHa=B-2u(1oo8Jd@DO!BBwYUVJgMt>01zVHO*m0N z@k4M*NaXyVOP>{x=OIyNc;MgBess+mBA_KI1}?oXYhQr~11gKE(e-=5bb`c*0ZJ{R zPLkhkMdc|sZL3zBpz=C`)dmEf`L|D>L9n*!bO~+`uS2=9@Dg_}5v`?kqvm4HTIEQJDILx51(00!~NG48RaYO1n~+Tom) z>%gto_v6zX6x~8t{D7ehPM>gy#<`ghAZQS$Io0T3`Of`Oj}^E60}vm%_e+<76coUo zZfSJ_tB_M=kf;XPweh;COo0PD(&wum-fQzuhLgV6W-np2b1wf-AgZEm?trVG2 zjTRMhwUi7hEQ$ViS9xhu5U0(&4_z~>gpE1qlR!^50Gn|&1R`L&Q&d1k1y$g7fQ-pz z*WJtpd z5m3_=mR^u9f%_br0?3=hf%p#isewm^otb%w`grHH$t#epG8(rOV}9q#p7k(K%!?ZH zSP?9zUC-Hx3GfLq>_j*i%es`RV7WB)M(_+k;oJ_#0ln07zsS+tYeEb!zSqCR#itT@ zOr=}ll=0!P&G3jFus%>&vvz;Ea@r}@O%7-{SzZsDNZ3pO&6Z8o(RS_eA=LiW$T0%s zoHS-z1fA2;VBaAb&1a;*=isxd@DCuXKY#ghKeai}6gVZDh5^hUJM}ISTzR3PAtAaD z+@Pcsd{B=$K?>fy^H4VMw;^hHKP+Y$#;v#=1+cV?84$nQH9tTme`#u}NGpPJf9~&A z-N7T-jZlp0xF-d#fdFbMlV7pq_H15zyIgzhHIb7(%IdX>TgWC!2mFz?AY-P!g1GOX znu4;p%bCwNx}oknEwZsSvW+}}@d)9Mb8Vaf;?IcWD-D^IMt+se&ycFVAg>h-4OQKT!9o=3ziqfV<=(wGb0zD0fk<(C3$q zy2}-iUk~pvasLgRs>18-9Kcm5aqnIoR7Ctdju7jrp@1?k9~6ydh44cETt7uO?C(%R zoz4@^eaVU#MT|n8NXHAD2J{->HJYP0sk%5^7U@maAtNP_#3J!ru@q#+J z4&$&jiEQt9<%Dp6-Ty(0FXXEAaM>zQtLks?>2TygJbQs4vrVW>83 z&e9bUY!0S9z*v@R=*}d%#)hcHs7gLQW>tGD-3dxyB)#`e19iI^5S%%xyt-7g+H_?! z6d2+dHM8#+APk-kebYt00Z9(jjiU?-D1rCU*LYK2nh<9rWpmB8qzTD_0s<~Ea;i1r zfpWj344@~Xi#TDr&z=vzZ?k*2;=K|_OPK)& z6zx|B=Cxlx>s~}gMk0L5Vh>xEPl*RvZhKz{R|mb)WWw3UOHGFra%T~daXBuSrMYbO*kpQo+8vV(?y+{vmB3s8azJkU7Qqj% z+|W9GvUO8T{7&RBL7{XQE`U%xZlvj{4`2d`Pfdu@1=i76YF+S>Ow1~Q$X zG)3`BP>NqW%?Oru(eKXR2$yLD+~Mbu-k%JGajATqXTQ>zuY)or3{ek=sZyvV0>3q( zf??GYdl=95$U8xnrDWNl_CnP{{ZG%D1PSphBV^m>*d72%$D#S-&%2C8dRR-oLE#zR z!24k2w*|Ra2yd%v-I{jrQZOyIa@08U_9*We7H}8%!Fjq7M#IOaTk+l%@(wS673v{K z#P2kF2g<<$LGvv-I&yk$AQA;^$&IxA4`E+OAmMbQ0RqQq$;np*-PI4|kFxCgG_=Uy zF@_2J{_D&Z_{Fp?1P~v&%Zxb08HtRougKLEHv-r`mHH+#N z%g4w_UbD`t_EJe5N&>5mK!2&UA}g*pc{YmUQIlRofc(^XLcsERJK+=V(Q6-rpq>G` zSbA76%bRi)`zD$-M&v9H3!4Rngbc)opI#v&o2i9d6IT|{3R$Yuedv}Vp*E}==ECps z1}^{MmmoR@*V^3Igr8F%{5(hH2Qay+sqp)-35}a#so}H+#B9;yZ}s}}P;O`9t+?l8 zsqxujz%!dG(n4}D@aQh7BzEX?Y6Em0eBrzNNK_0WichS1ego&(8{OT()VkxoF)kmv zW2vsLt})>84iI6jJt!cTTm?3?pUlTWr7=4BfN_ZlcZ}Zy#r6N2dUzxAY;%Z17utA!&&hgSAf$M*$oiUNGaovX^~Km>eQ1t(hFS z)9^@tRg?f1 z#UUKjOGk9~wo}yEr4e%YVaMhxym{q#Z65P-XQj#aJ6M6YtonC7?!!CmJT1i^U+93L z8O#z=Ce zj6Wu0B5R`SenCMIhN1f>@s@~<#ep7E{{kRubz^EEIu+*AbehT0RK1f!Ht~q493oc6e^Pj*h!y4A;79_ zs89Z`4#5;7+JE+ZdykT^pm7tPV~Q|hDIm+5)c4UBG@2Iu1&z=AQy22j-@%-K+fVSs zh`I7N1o_|g1OR{0RtBiNe|`w5yovTjr0SMv)G0Vgj!j?gS7 zXe;c0d(62qhvTVerPcdz`OR*ZwR&#JTJq(GG=urL2+g7C*<F8vllqrRCg?2;^4`$|neSS~)H^K#s z7(lpW2zhRxO_;5VSD*)3|nSXOmzcOe9o~VQequGVinfa9d7qK&ZMn)?Gd)_dAKHZ+Jg>p( zT}F3R<@WL>BM~?|s)A4M@Kz@j8_NO+qY+A?0a|9{G87lgzB{2jR~4NR(Z9NEnm1Ze z_uC$LKI?v^Pqw*eG!=ma>NTSfM!a1;G28nZdKb5lrvEy4_cVDHGvvvQ0A@Dw(rZ6#R$WTIuHYMUdBD8Id!HWyZUiakd zNs?eRuP}SK2BO{D1s`>`B+h9RjX&$Ys;)1J5B(R$;KVb(;=jtsbWuZlxDrk5Y?ehEJH9hu9BFQ0xSTur z{hhcrnT5rW3U38pU&?lsZ1M{pat)#Bf*&tv69-{#%e;>^6n2UM?H_3p*8$t=Z9^sQ z1GPorE!*@AGp^i#i49%D9o_)JShmCAo62AfEmHF zdzqJPPz0L=Fmk_*eFblZZ+INeRQ8WQV4C|Q%KHdNumuv9Lvm>kuTDO{PhWEIY-;Di zO&1GQjQ?)kJ5XQy#nZ^yEALa{ENxQbP!S|@*$=!# zRMvDsFmuW^zvUXbnEkwK@-|De1N$P$1aw@Cpd*WFmB#4fr$Fm4T1^01&@WSDwOATW z($Y8zy+#5ZN}ug}BZO&?l=dS7FO9n)2jWj~XQcPw3QfzxXY9emoobtnNl|n4lf;vLt0=O5GBwnakdD->9bVBtHe}Kl4BhpxsdcrG|<6Og^l~wAj z)j%Qmlt)!x&YDhY>!bYfVcNEFnG#j*XO|8D&;})Hj*f;+-54d)?2hIF(;!jL#68k} z%y&T6&g}D@#+MMUr?mY1faEE}Ni(6P8lBJ%4TTbrM5BTONZ%v3qwY+%j{GHKLx&1K zNx=lcIY7Mv3E-M;-`wt90yI8;+sB!cV>;&YiMza$tuyORC&zW{lpX@;uVWGBA#lZcyvHIi1dE~QRlKXLl_6g?XK*Y19`Bz7Yfin%J z*ueN+Kd=HB0Mbqjs~Xig|~{mPr> zmj@aO$E?D5LC|AnX_)_>LthTPWGNG{Z;a05fKdfVN}hZ?rE1}Lkq1By-?K$M!rUNP zTdF{^mlq?|!YL4G&A-e|BSFH*cf{p8E#lL8e@4yqT#f6uF$4tp2nzx6qS!q_G3%-T z-|Y7Co3^f@t;H)WqWmrDqqI4dD@&(6gMFS%yZTqTAu*j7m9Dw%dGmo@WKu0LTfg`1 z2qVzN#lj_%n|06vLshsOFc8%#fPqK=h*JC%y-Ywtf67e!PzNB$^c);_n_FrUuhkE6 z6P@}Ax3upCL+QT#r*Qn_QHqjO!b{5F{`S`)6i+Bn6ea|MXimTzlQ!=gR7=1crtpwW zcUxRGbxqpxwQ))4dzJV_DVbS$+;qT9m z6ZGuLtuVlg;5%#@6;kn#U%YAiBx5i(m*>P@z>NXGchY~h#EFV)yZ+~Gig#6anx3Ln zsgiAks$ZQjJY8%U^a5>-4zq919tvpd|2>Bu4Np+KgcM8RKrRRge)rCf$#^cYI;&VO zh+GbE)HbJEvEUJtRpSSUrNl>(z6(gMEn;}Lx_a2Ridwx{QPVm7Q~BQqtgF`4kaIiYl{pF6Mr+v9_M*ah?mxH#~B4t9d2G*P{UsCpzo8b4h^r z-^gjy`RK!<1z)pO8#38TQW*MEJ`e;e|-^~e&}7^W!j$tA#UJI76cEkXCMzQPYu1%I*LWWMNv zkhtgtQ~BRS9IG8RE4F%XZT27mbayf5nYsEMU AtN;K2 literal 0 HcmV?d00001 From 4669665abadd4b99b8b53516ccf10c961392460f Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Tue, 8 Jan 2019 15:57:24 -0800 Subject: [PATCH 13/14] Online deletion typo fix --- .../the-rippled-server/ledger-history/online-deletion.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/concepts/the-rippled-server/ledger-history/online-deletion.md b/content/concepts/the-rippled-server/ledger-history/online-deletion.md index 880862fa63..c543308aed 100644 --- a/content/concepts/the-rippled-server/ledger-history/online-deletion.md +++ b/content/concepts/the-rippled-server/ledger-history/online-deletion.md @@ -25,7 +25,7 @@ The online deletion settings configure how many ledger versions the `rippled` se - The server may store up to twice the configured number of ledger versions if online deletion is set to run automatically. (Each time it runs, it reduces the number of stored ledger versions to approximately the configured number.) - If advisory deletion is enabled, the server stores all the ledger versions that it has acquired and built until its administrator calls the [can_delete method][]. - The amount of data the server stores depends on how often you call [can_delete][can_delete method] and the how big an interval of time your `online_delete` setting represents: + The amount of data the server stores depends on how often you call [can_delete][can_delete method] and how big an interval of time your `online_delete` setting represents: - If you call `can_delete` _more often_ than your `online_delete` interval, the server stores at most a number of ledger versions approximately equal to **twice the `online_delete` value**. (After deletion, this is reduced to approximately the `online_delete` value.) From 9d88381c899aa6ae82f0c5e93a7ee4471f408845 Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Mon, 14 Jan 2019 15:43:46 -0800 Subject: [PATCH 14/14] Online deletion edits per review - Cleaned up can_delete method fields, errors - Changed commands to delete old db files in full history tutorial - Clarified RocksDB w/ rotational disks - Clarified how many ledgers may be stored w/ online deletion --- .../ledger-history/online-deletion.md | 5 ++++- .../can_delete.md | 19 +++++++++---------- .../configuration/configure-full-history.md | 6 +++--- .../installation/capacity-planning.md | 4 +++- 4 files changed, 19 insertions(+), 15 deletions(-) diff --git a/content/concepts/the-rippled-server/ledger-history/online-deletion.md b/content/concepts/the-rippled-server/ledger-history/online-deletion.md index c543308aed..5ec0f0939d 100644 --- a/content/concepts/the-rippled-server/ledger-history/online-deletion.md +++ b/content/concepts/the-rippled-server/ledger-history/online-deletion.md @@ -22,7 +22,10 @@ Inside the ledger store, ledger data is "de-duplicated". In other words, data th The online deletion settings configure how many ledger versions the `rippled` server should keep available in the ledger store at a time. However, the specified number is a guideline, not a hard rule: - The server never deletes data more recent than the configured number of ledger versions, but it may have less than that amount available if it has not been running for long enough or if it lost sync with the network at any time. (The server attempts to backfill at least some history; see [fetching history](ledger-history.html#fetching-history) for details.) -- The server may store up to twice the configured number of ledger versions if online deletion is set to run automatically. (Each time it runs, it reduces the number of stored ledger versions to approximately the configured number.) +- The server may store up to just over twice the configured number of ledger versions if online deletion is set to run automatically. (Each time it runs, it reduces the number of stored ledger versions to approximately the configured number.) + + If online deletion is delayed because the server is busy, ledger versions can continue to accumulate. When functioning normally, online deletion begins when the server has twice the configured number of ledger versions, but it may not complete until after several more ledger versions have accumulated. + - If advisory deletion is enabled, the server stores all the ledger versions that it has acquired and built until its administrator calls the [can_delete method][]. The amount of data the server stores depends on how often you call [can_delete][can_delete method] and how big an interval of time your `online_delete` setting represents: diff --git a/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/can_delete.md b/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/can_delete.md index db8f1ea407..ca647a42bb 100644 --- a/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/can_delete.md +++ b/content/references/rippled-api/admin-rippled-methods/logging-and-data-management-methods/can_delete.md @@ -43,16 +43,15 @@ rippled can_delete 11320417 -The request includes the following optional parameter: +The request accepts the following parameter: | `Field` | Type | Description | |:-------------|:------------------|:------------------------------------------| -| `can_delete` | String or Integer | The [Ledger Index][] of the maximum ledger version to allow to be deleted. The special case `never` disables online deletion. The special case `always` enables automatic online deletion as if advisory deletion was disabled. The special case `now` allows online deletion one time at the next validated ledger that meets or exceeds the configured `online_delete` value. | +| `can_delete` | String or Integer | _(Optional)_ The [Ledger Index][] of the maximum ledger version to allow to be deleted. The special case `never` disables online deletion. The special case `always` enables automatic online deletion as if advisory deletion was disabled. The special case `now` allows online deletion one time at the next validated ledger that meets or exceeds the configured `online_delete` value. If omitted, the server makes no changes (but still replies with the current `can_delete` value). | -If no parameter is specified, no change is made. +### Response Format -The response follows the [standard format][], with -a successful result containing the following fields: +The response follows the [standard format][], with a successful result containing the following fields: | `Field` | Type | Description | |:-------------|:--------|:----------------------------------------------------| @@ -62,11 +61,11 @@ Use this command with no parameter to query the existing `can_delete` setting. ### Possible Errors -* Any of the [universal error types][]. -* `notEnabled` - Not enabled in configuration. -* `notReady` - Not ready to handle this request. -* `lgrNotFound` - Ledger not found. -* `invalidParams` - Invalid parameters. +- Any of the [universal error types][]. +- `invalidParams` - One or more fields are specified incorrectly, or one or more required fields are missing. +- `lgrNotFound` - The ledger specified by the `can_delete` field of the request does not exist, or it does exist but the server does not have it. +- `notEnabled` - If either online deletion or advisory deletion are not enabled in the server's configuration. +- `notReady` - The server is not ready to run online deletion at the moment. This usually means the server has just started up and has not yet acquired a validated ledger. ## See Also diff --git a/content/tutorials/manage-the-rippled-server/configuration/configure-full-history.md b/content/tutorials/manage-the-rippled-server/configuration/configure-full-history.md index fd7a96d5a1..d859cf08c6 100644 --- a/content/tutorials/manage-the-rippled-server/configuration/configure-full-history.md +++ b/content/tutorials/manage-the-rippled-server/configuration/configure-full-history.md @@ -59,9 +59,9 @@ To configure your server to acquire and store full history, complete the followi After disabling online deletion, the server ignores any data that was downloaded while online deletion was enabled, so you may as well clear up the disk space. For example: - rm -r /var/lib/rippled/db/rocksdb/* - rm /var/lib/rippled/db/*.db - rm /var/lib/rippled/db/*.sqlite + rm -r /var/lib/rippled/db/* + + **Warning:** Be sure that you have not put any files you want to keep in the folder before you delete it. It is generally safe to delete all of a `rippled` server's database files, but you should only do this if the configured database folder is not used for anything other than `rippled`'s databases. 0. Start the `rippled` server, importing the database dump if you have one available: diff --git a/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md b/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md index 7296e9a286..9bdaec4e07 100644 --- a/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md +++ b/content/tutorials/manage-the-rippled-server/installation/capacity-planning.md @@ -53,7 +53,9 @@ The example `rippled-example.cfg` file has the `type` field in the `[node_db]` s #### More About Using RocksDB -[RocksDB](https://rocksdb.org/docs/getting-started.html) is an embeddable persistent key-value store that works well with both rotational disks and solid state disks. +[RocksDB](https://rocksdb.org/docs/getting-started.html) is an embeddable persistent key-value store. + +RocksDB works well on solid-state disks. RocksDB performs better than NuDB when used with rotational disks, but you may still encounter performance problems unless you use solid-state disks. RocksDB requires approximately one-third less [disk storage](#disk-space) than NuDB and provides better I/O latency. However, the better I/O latency comes as result of the large amount of RAM RocksDB requires to store data indexes.