From 45e7b2aca0333514abcf12baf0d6b6736e7b1346 Mon Sep 17 00:00:00 2001 From: mDuo13 Date: Tue, 4 Dec 2018 19:34:07 -0800 Subject: [PATCH] Online deletion: concept 1st draft w/ diagrams --- content/_img-sources/fetch_depth.uxf | 87 ++++++++++++ .../online_delete-vs-ledger_history.uxf | 107 ++++++++++++++ .../the-rippled-server/online-deletion.md | 131 ++++++++++++++++++ dactyl-config.yml | 9 ++ img/fetch_depth.png | Bin 0 -> 10064 bytes img/online_delete-vs-ledger_history.png | Bin 0 -> 8121 bytes 6 files changed, 334 insertions(+) create mode 100644 content/_img-sources/fetch_depth.uxf create mode 100644 content/_img-sources/online_delete-vs-ledger_history.uxf create mode 100644 content/concepts/the-rippled-server/online-deletion.md create mode 100644 img/fetch_depth.png create mode 100644 img/online_delete-vs-ledger_history.png diff --git a/content/_img-sources/fetch_depth.uxf b/content/_img-sources/fetch_depth.uxf new file mode 100644 index 0000000000..97480a876a --- /dev/null +++ b/content/_img-sources/fetch_depth.uxf @@ -0,0 +1,87 @@ + + + 10 + + UMLClass + + 30 + 70 + 210 + 40 + + Ledgers not stored +bg=#e1e4e8 +fg=#23292f +lt=.. + + + + Relation + + 20 + 110 + 740 + 50 + + lt=->> + +Ledger versions +m1=oldest +m2=newest + 10.0;20.0;720.0;20.0 + + + UMLClass + + 240 + 70 + 240 + 40 + + Ledgers available locally but not served to peers +style=wordwrap +bg=#999da2 +fg=#f5f7f9 +transparency=0 + + + + UMLClass + + 480 + 70 + 260 + 40 + + Ledgers served to peers +when requested +style=wordwrap +bg=#1db4ff +transparency=0 +fg=#f5f7f9 + + + + Text + + 270 + 30 + 170 + 30 + + fetch_depth setting +style=wordwrap + + + + Relation + + 410 + 30 + 90 + 60 + + lt=<<<- + 70.0;40.0;70.0;10.0;10.0;10.0 + + diff --git a/content/_img-sources/online_delete-vs-ledger_history.uxf b/content/_img-sources/online_delete-vs-ledger_history.uxf new file mode 100644 index 0000000000..051c9efe45 --- /dev/null +++ b/content/_img-sources/online_delete-vs-ledger_history.uxf @@ -0,0 +1,107 @@ + + + 10 + + UMLClass + + 30 + 70 + 330 + 40 + + Delete automatically +bg=#e1e4e8 +fg=#23292f +lt=.. + + + + Relation + + 20 + 110 + 740 + 50 + + lt=->> + +Ledger versions +m1=oldest +m2=newest + 10.0;20.0;720.0;20.0 + + + UMLClass + + 360 + 70 + 170 + 40 + + Keep if available +bg=#999da2 +fg=#f5f7f9 +transparency=0 + + + + UMLClass + + 530 + 70 + 210 + 40 + + Backfill if possible +bg=#1db4ff +transparency=0 +fg=#f5f7f9 + + + + Relation + + 520 + 30 + 70 + 60 + + lt=<<<- + 10.0;40.0;10.0;10.0;50.0;10.0 + + + Text + + 570 + 30 + 210 + 30 + + ledger_history setting +style=wordwrap + + + + Text + + 150 + 30 + 170 + 30 + + online_delete setting +style=wordwrap + + + + Relation + + 300 + 30 + 80 + 60 + + lt=<<<- + 60.0;40.0;60.0;10.0;10.0;10.0 + + diff --git a/content/concepts/the-rippled-server/online-deletion.md b/content/concepts/the-rippled-server/online-deletion.md new file mode 100644 index 0000000000..41afbdcbd8 --- /dev/null +++ b/content/concepts/the-rippled-server/online-deletion.md @@ -0,0 +1,131 @@ +# Online Deletion +[[Source]
](https://github.com/ripple/rippled/blob/master/src/ripple/app/misc/SHAMapStoreImp.cpp "Source") + +The online deletion feature lets the `rippled` server delete the server's local copy of old ledger versions to keep disk usage from rapidly growing over time. Online deletion happens automatically by default, but can be configured to run only when prompted. [New in: rippled 0.27.0][] + +The server always keeps the complete _current_ state of the ledger, with all the balances and settings it contains. The deleted data includes older transactions and versions of the ledger state that are older than the stored history. + +The default config file sets the `rippled` server to keep the most recent 2000 ledger versions and automatically delete older data. + +**Tip:** Even with online deletion, the amount of disk space required to store the same time span's worth of ledger data increases over time, because the size of individual ledger versions tends to grow over time. This growth is very slow in comparison to the accumulation of data that occurs without deleting old ledgers. For more information on disk space needs, see [Capacity Planning](capacity-planning.html). + + +## Background + +The `rippled` server stores data from recent ledger versions in its _ledger store_. This data accumulates as the XRP Ledger network creates new ledger versions and validates them through the [consensus process](intro-to-consensus.html). + +The XRP Ledger stores the complete state of the current ledger, with all balances, settings, and other data, in each ledger version. Unlike many cryptocurrency systems, an XRP Ledger server does not require full history to sync with the network, know the full state, and contribute to making forward progress. + +Inside the ledger store, ledger data is "de-duplicated", with data that doesn't change from version to version only stored once. The records themselves in the ledger store do not indicate which ledger version(s) contain them; part of the work of online deletion is identifying which records are only used by outdated ledger versions. This process is time consuming and affects the disk I/O and application cache, so it is not feasible to delete old data on every ledger close. + +### Fetching History + +When it starts, a `rippled` server's first priority is to get a complete copy of the latest validated ledger. From there, it keeps up with advances in the ledger progress. If configured to do so, the server also backfills ledger history up to a configured amount, which must be equal or less than the cutoff beyond which online deletion is configured to delete. + +The server can backfill history from before it became synced, as well as filling in any gaps in the history it has collected after syncing. (Gaps in ledger history can occur if a server temporarily becomes too busy to keep up with the network, loses its network connection, or suffers other temporary issues.) To backfill history, the server requests data from its peer `rippled` servers. The amount the server tries to backfill is defined by the `[ledger_history]` setting. + +The XRP Ledger identifies data (on several different levels) by a unique hash of its contents. The XRP Ledger's state data contains a short summary of the ledger's history, in the form of the [LedgerHashes object type](ledgerhashes.html). Servers use the LedgerHashes objects to know which ledger versions to fetch, and to confirm that the ledger data they receive is correct and complete. + +Backfilling history is one of the server's lowest priorities, so it may take a long time to fill missing history, especially if the server is busy or has less than sufficient hardware and network specs. For recommendations on hardware specs, see [Capacity Planning](capacity-planning.html). Backfilling history also requires that at least one of the server's direct peers has the history in question. + +### Full History + +Some servers in the XRP Ledger network are configured as "full-history" servers. These servers, which require significantly more disk space than other tracking servers, collect all available XRP Ledger history and **do not use online deletion**. + + + +### History Sharding + +An alternative to storing the full history of the XRP Ledger on a single expensive machine is to configure many servers to each store a portion of ledger history. The [History Sharding](history-sharding.html) feature makes this possible, storing ranges of ledger history in a separate storage area called the _shard store_. When a peer server asks for specific data (as described in [fetching history](#fetching-history) above), a server can answer using data from either its ledger store or shard store. + +Online deletion **does not** delete from the shard store. However, if you configure online deletion to keep at least 32768 ledger versions in your server's ledger store, your server can copy full shards from the ledger store to the shard store before automatically deleting them from the ledger store. + +For more information, see [Configure History Sharding](configure-history-sharding.html). + + +## Online Deletion Behavior + +The online deletion settings configure how many ledger versions the `rippled` server should keep available in the ledger store at a time. However, the specified number is a guideline, not a hard rule: + +- The server may have less than the configured number of ledger versions if it has not been running for long enough or if it lost sync with the network at any time. (The server attempts to backfill at least some history; see [fetching history](#fetching-history) above for details.) +- The server may store up to twice the configured number of ledger versions if online deletion is set to run automatically. (Each time it runs, it reduces the number of stored ledger versions to the configured number.) +- If advisory delete is enabled, the server stores all the ledger versions that it has acquired and built until someone calls the [can_delete method][]. + + For example, if you call `can_delete` with a value of `now` once per day and an `online_delete` value of 2000, the server typically stores 2000 ledger versions at minimum and just over a full day's worth of ledger versions at maximum. + +With online deletion enabled and running automatically (that is, with advisory delete disabled), the total amount of ledger data stored should remain at minimum equal to the number of ledger versions the server is configured to keep, with the maximum being roughly twice that many. + +When online deletion runs, it does not reduce the size of the database files on disk; it only makes space within those files available to be reused for new data. To actually reduce the disk space allocated to the database files, you must restart `rippled`. + + +## Configuration + +The following settings relate to online deletion: + +- **`online_delete`** - Specify a number of validated ledger versions to keep. The server periodically deletes any ledger versions that are older than this number. If not specified, no ledgers are deleted. + + The default config file specifies 2000 for this value. This cannot be less than 256, because some events like [Fee Voting](fee-voting.html) and the [Amendment Process](amendments.html#amendment-process) update only every 256 ledgers. + + **Caution:** If you run `rippled` with `online_delete` disabled, then later enable `online_delete` and restart the server, the server disregards but does not delete existing ledger history that your server already downloaded while `online_delete` was disabled. To save disk space, delete your existing history before re-starting the server with `online_delete` back on. + +- **`[ledger_history]`** - Specify a number of validated ledgers, equal to or less than `online_delete`. If the server does not have at least this many validated ledger versions, it attempts to backfill them by fetching the data from peers. + + The default for this setting is 256 ledgers. + + The following diagram shows the relationship between `online_delete` and `ledger_history` settings: + + ![Ledgers older than `online_delete` are automatically deleted. Ledgers newer than `ledger_history` are backfilled. Ledgers in between are kept if available but not backfilled](img/online_delete-vs-ledger_history.png) + +- **`advisory_delete`** - If enabled, online deletion is not scheduled automatically. Instead, an administrator must manually trigger online deletion. Use the value `0` for disabled or `1` for enabled. + + This setting is disabled by default. + +- **`[fetch_depth]`** - Specify a number of ledger versions. The server does not accept fetch requests from peers for historical data that is older than the specified number of ledger versions. Specify the value `full` to serve any available data to peers. + + The default for `fetch_depth` is `full` (serve all available data). + + The `fetch_depth` setting cannot be higher than `online_delete` if both are specified. If `fetch_depth` is set higher, the server treats it as equal to `online_delete` instead. + + The following diagram shows how fetch_depth works: + + ![Ledger versions older than fetch_depth are not served to peers](img/fetch_depth.png) + + +### Relation to Real Time + +Online deletion settings must be configured in terms of the number of ledger versions, not in terms of real time. You can estimate the amount of time represented by a number of ledger versions using the rate of ledger progress. Ledger versions typically close 3-4 seconds apart, with the overall rate of ledger closures ranging from about 20,000 to 27,000 per day throughout the years 2017-2018. There are hard limits of at least 2 seconds and no more than 20 seconds between ledgers as long as consensus is operating properly. + +The following table summarizes some + +| Real Time Amount | Approximate Number of Ledgers | +|:-----------------|:------------------------------| +| 1 day | 25,000 | +| 14 days | 350,000 | +| 30 days | 750,000 | +| 90 days | 2,250,000 | +| 1 year | 10,000,000 | +| 2 years | 20,000,000 | + +These numbers are estimates and depend on several factors: + +- The rate of agreement among trusted validators. More disagreement slows the rate of new ledger creation. Disagreement can occur because of random chance, network latency, or validators running different software versions. +- The volume of transactions in the network. A high rate of transactions can increase the work needed to close a new ledger version, reducing the number of separate ledgers closed. An extremely low rate of transactions can also cause ledgers to close further apart (because there is nothing to be done). +- Changes and optimizations to the XRP Ledger's consensus algorithm can affect the speed of consensus and the rate of disagreement either positively or negatively. + +### Advisory Delete + +By default, online deletion happens automatically and periodically. If the advisory delete setting is enabled, online deletion only happens when an administrator triggers it using the [can_delete method][]. (You can trigger this command with a `cron` job to run automatic deletion based on actual time instead of the number of ledger versions closed.) + +If your server is heavily used, the extra load from online deletion can cause your server to fall behind and temporarily de-sync from the consensus network. If this is the case, you can use advisory delete and schedule online deletion to happen only during off-peak times. + +You can also use advisory delete for other reasons, such as if you want to manually confirm that the transaction data is backed up to a separate server before deleting it. + +The `can_delete` API method can enable advisory delete (with the value `never`) or disable advisory delete (with the value `always`). These setting changes persist until you restart the `rippled` server, at which point the settings in the config file override them. + + + + + +{% include '_snippets/rippled-api-links.md' %} +{% include '_snippets/tx-type-links.md' %} +{% include '_snippets/rippled_versions.md' %} diff --git a/dactyl-config.yml b/dactyl-config.yml index a67e7f9e90..cdf9f06032 100644 --- a/dactyl-config.yml +++ b/dactyl-config.yml @@ -545,6 +545,15 @@ pages: targets: - local + - md: concepts/the-rippled-server/online-deletion.md + html: online-deletion.html + funnel: Docs + doc_type: Concepts + category: The rippled Server + blurb: Online deletion purges outdated transaction and state history. + targets: + - local + - md: concepts/the-rippled-server/peer-protocol.md html: peer-protocol.html funnel: Docs diff --git a/img/fetch_depth.png b/img/fetch_depth.png new file mode 100644 index 0000000000000000000000000000000000000000..f760d6d91e35aaf5ed0d538ddeb4838fecba84c3 GIT binary patch literal 10064 zcmbVyWmr_**Eblqaid5p5()!I3(_qi9FQ()Q5uG30EbXQN{K^B4G2S*Fm!s;-93cF zh%kh7!+Y@lU)TFQpWf$r=fgfTd-mCTt-aP>_1ht8D)M)3Q{N^aAh-i4$Z8M}TqT0O z=iR&lwGrP;qzMREr~ujLS{_sDDYrbx<)+>f8reVhHoN|uisV(RQIXu%HDWDtl2fBS z-HAZ@fA*v&gR64>W+(~`#s@mj$zPEzA|NEad5_dNG9Wo1GN5~Ue2Fg_f5Chw#loV7 z_mN`wfpjEws#*2tQA(y<}HF|x*aB};2L7d!x38Qa-1N>vVUW) z@j}ju5Jbm8+{13A(e3m+>*o`0=d7G&P-0_eU+zg@Of@L8a>BwZISs4S3|xA?@F)|t zw6=C8n)n=b*YPpFxN)Nf&#aRC%GkK*i3|XQ6!?y6 z>&`ef-yvgud7tM^05f30G>CYN7dx1Cp%rnh+Mlw`ewm=5u0CHDSwz)_#TuBExZ%fU zYTZ-$ZR08u!{pyCbwnXf3Pl=OIt%X0$;rKK4~Jtb3XNv8scMgJEmISK*VHZ!cZXFk zw&NRrnOOFG8O+x$PD^`a-FS9-+9EwxWP~rnEcp{lMDF_xXDL&LF$oDB#_Adm2Bzs@ zER^E8di(q1y{5`-D?PT}$Pq%@4x=Qcq>PJ=Wja#w&kONOhn4=Vr>w;+D5wjf@2B%x zhyz9BS!ro$+W9(jJ|~51?83sHt0TGd^Yc`$d?U?pi`Gbr-J06!>J>K!D=I5UxqvG< zo1m74#!!}W62#lX6zh)2$KiC3h92qgILwN*6Up5~`5arH;lsEagb-`5F!fYaRF!s< zL(R=HC;B=%W7RH}B-h;lGb;CKryvv`a^G&U>m_K&`tr@2biO4PRa6;%!FkK7iUA03 zjeI*%=eYyzqV?b1-Q5m#TQ49AB4O`qo&ct%rb8$rb!Ci2Z=wL4&7{^1|MM4N5HWom zdSc>LHppk*MShka?DxxWayBC)gNfdW+`BVcRf07=jVokrVN};-RB&r&Ct544bwTTL zd2@5zVuYkm^Buve>T38*jrPd8U-(CMaOo9MdkrzY1PyeiAO+)6)UdM3DYN9^;ZYgq zd-uK~OW>$Y9sJ~yX%vMHx@I}J=T`gjLtU&K=$t!50t+Ru<{$ycI zrFZzk%8ka0b3eZ;>v-g5{rFdzAWnmdrRC+IfPnif_9l_>3KdgJxV1IA#TK|6up5H) z^z@u+fp5XG16@m=e)#Z#yYbXnR#p}O01pq3_HbHrcSuFn8ld&3g6PEGb)h0tno6!GLO)<^9#se9xVqMzsj<#m|C6!^7n`=>3CPqyv7ea8I7V>-=bWrpC>7 zvK06I!`06b1XYg!HZCqYL3=}N1zgl^mCxtYW3m57uOdTjq%z8TU$3Ohwy65WuJnaz zqtRm{j!$+HzUQYB-p6G&ql)|d@F~9ageyh#cWR3vIjO4;e&Kn$TO+wY@wUxW_Rnq; zR6XTd?0(@RE-QlEU(0n5c?=M|hY?CG#==ZY<{*+S%*@bPSvA$wT8$8z(NH6)h zlN=CmEt~jF9PYCo7#bEf=e5(@mm+H7y*KKAgZS?4+csbRb2|Q5;7jxS=dAocA3|3N z3AiM0-MXb~=p;u*Wd*TqW5Xs-bZLjJIHsXx%BeN_QR0#(YtOyz>!={7H)3taMC~He+7q!3nV4DLK3)3D-t7f z4dR!8PvfsU)Jt06RHXww6;=Bd;ia>nW`zjh)Gzz?481W^tBMj!@W1+Juno#<8_0Of z?=V}ZnXOV?T}|`glD{NFUT5gTu!aIaC%>uiSDnXA{N4(gWb zpm%w#_WQH3_O-!|3{ypLdT!W`Oo8;dVHpeGo!-**p<|hS0O&Uvx z6nOIr@TThXqe$2lfZ--A1O@YZUG@3T-%{YJ+RNnKaBMC8>xX;01je%{^p~o)^687K zZuwq(k6G6gJ}p{e6kbZudL1eh>KV1na=*G`6g27|*vo)iT6&Q%*e&R#Nc6`?Ky{p7^$z77y6qOY_s0G0 zL(OiTqQSUqON+*{V&97+Zg*mu-QP>O7jy>^g1Z6|oX>2BL^Ur)F1XrZm}!{t;fYe| zRK7zEfa?ED$F77}+ol!RuEw6+_4*OT(tr|`6cwB+vMUXVF|3UeGaz@J{*X<+HMjyUo0QZAoo{B>u?nk?*io8CU7fNn(fLxyzcs8VM6ck-}i-Ch^TY z^`*mW>y59@=RQZ=GOqQ}Y>q=G)#YW$#kJ#Jxr8ZZlor^VQcP`cVDsac;>^ion3wh9 z>A11Zd2JN$Hgfc1m&710Z_TCJf)?;jo!S6O~* z|EOGgI`{0H@s6*W!bImVE4|kteaVgbBtk0?I)&+IlxYxwGa&E{_|;PN$wpW*UP zvH5c;J5=29{BIY@BvAY;LVVNe;JfKr2V9~XtMvoWa+A=V+ho0nsrRtI07>VuEA{=c zHL+PX!;=#Eh|wudvkJ{xW%2~<=)%I?*ZxuH_i4?n)(NPjijZTwsa4K(oC>|99vw9a z)18ZB`bsW?6(=DveMDhWWLe8Q4FwPMe^iCP)gi9n6{88xM9H)y)%Sft_0-HWA-*>$ z%Q~1QXnhhA3U*=JH)XDwwZB@&Xlq@q$!31f*6Wspu%~b#lB}*%A`MEL>T9@;7-Yhe z*z@d|fpo?PlhvrCNWYUU z;!{nT?BXzAZU_2o&`js~Gr-1997t`Ak+;wbS}!Kw^ZfZbvuwIJU}EaWX2hgEPX3JT zq7Jhfbm*KLym9+)QX8;ARPs{sg}FIcDV(ZDvHRNRy<8kA@6d*0b=}$9jZQf5Ysu?S zik|&9m0{TW{1#nfA~A#ak{dasZeQ*he!bL*RTSD`p*vz(KWMW!=BzK2lj>_VwsI&J z2B6Zp>+9gP6454XSI|Y{Ok#No8qI5fVt-FV{hX1rk$}i>*jr=8{*H~&gT6gU`$0;J zW&I6mFOYQKL#vB`R&vDri&=4oCqHFIAWpGFUKw5fVFnbYFW|@ zi!Y3e(;mNT4L)<(F0;^>dQCzuZv77!J2YzxSX8elZ*QTa-wja{tLFBx9o*rHGvVdt zIG=d8IW+umxM>VC8lzsSfwQUeq+YPKn(zrUH__ZB03?tE`eKv#D%7si<2U6@PC^rid{Z0~V$$E`$^>EVqwx z3);i;lGBIc{rkBZJq@%qP}(lnt-V{tn3I)OYYG*6q?CBOl^y!rQ*ui_BfZ#hKZtgD z;7z?cYHO|bRHOzfv4zSKwM%;gp`ZLTCJm$f(-&+2P^>l7jOrL1c$q)ECB(Y)bvN-M zcqIx~`nOy?*SfkKNSy_{9vBe9w|lB78A4ZifS=7T_M+um3i%fNR+9-M4BGhwm;8aWmnf6mszl;vv zMFbxd)q5VvF;I>kC4Ukd4`3JIiH2by@w7E0D;v}^S$Bq(xAl%{BiMkAhXUHfTJVu* zGw$|i6>Gi1nQtq8;r_#l@QGhcfqEqtDHg9S$RfxoWI-6+9;Q6^sI}tB6E4mIuHS<+ zc>mGvO&-?(J5g`MNGDGxe)@fkp8Y!)iGx?wPG20_6H1rF@a|hUUs0{chy|g$zg(}k z8m}Q9^GHlEzYx!_+>_9(NUv~aV=Ib7-`JlLpef6?Xl}h_UhB)|!Ulwq+Qdn^3{=b) z&SquSl*v;7HeZ_IAO86|6cXPSXSS(bPa57i#crR-K^H<`4D$CPdBf4h%@BHj293b{b&n7n5g=C|^(FtwIDd zK=%{UeAcX6O3gIn)aROtH51_C#|L`VL!rgabKe|S@~$iCP?E>m&8_6S=Pc>nc0W4g zG{LE>?P4Sfg-tGQBZ@Y`S1P1j)_c#wH?G#7EVXY&dG$mJU zeMKauwYaIpAaH|+JQ~P>t`K~mV zE0jcM)+}9a-+OVuIzg}gc;;w^`0w?gd0smi6)KTwi@2l&*n_u+7h;lUi~WW%Fu1H; zuZXJnN=Q`yYTV8cJK0>Sz;Nu_^v#i~5D}MJ2BQ{2yQj_YseAqB{?GTjxoxU?IRtfm zl-yRcM+SZ?EP}DN)-t_8lhf8&}P z0M^rlTakk!E-m9rSue(y}nyG4b z8qbP7ar$s$%xv|W9z}KY(U$cB!+vFKT!AC;O_2iCX-e+SPK+IP1vBx~>4bqzTbz5= zsFWgQZxSg5c@zpUa?&_dECzAL42XCX zTzsv(Fu%A-(5b8ndY$iW67SIp_x7rDm%8l4bC{eRYWDB0x@Q*%+xCq+r??q`ycm~k z-_yyI1BqXU?A|1amgnxf>)tm+B(8k_k3^^F5npe7lTZY=+^b2C`Bv+1T#@5PKaWN5 za4=XLu@M(OV;vGO);klu_$4$fPaxY4NVKUx%!Wid` zW2YiTUniaX>rudX%YK1vu3Mj~LgjV#nv3K-zMpLg0_rY40WkEF2 z9cTiCWqY5C|MSfkTEQ|)6tQ&O<$aPmvzC^bW7(zY$ML^eIL6FF_9F;hrD#RNj_$Wd zB>((5k?cOBvxo3#ohPzIe1-S+zqR$bwb^xB!`@Jetvw# z^-=G>5^|A|pdpM~64%5?0Q2o%l_x+`#)O(9W^|CbT%(X1_Ha*HAx{mWLvCZ~DSt1J z%+4T4q7qT&$&Jz@Q9>x!JTA!#3{PF{I&*Q}FFX9M0$u(uo17aR9$#SKF1B?;{DhKA zFTrcOXKN{X*urNDik4SRw#nkfxi)LDeM#_jT>XL6k5X;CPJuBMDV;Sd2jVRLw8dfH z*9@ug{%gb?;26G8e1m=VbIvaeCU35&YjdF+^EZCMp)U#s%Qd}QY1E|9-*Nu>xZV8@ z-OxceGL@FU?LzG-r5~*NJ``3-(2%4gM90UQM7nO5EoZ?MfRxbHi^Qg@g#WlzN8Zr} z->u~&y7%n6rt)DmQE-f-8vUPK*s87>e(eK{whgCa;nc<5uv?4jfW1z5JLOpRTkdh@V_7-Pxw%Dx73HwvzUe zE@il}p7IzBS$d6BvCt~mB1yD}z>O$wl!aT!uT1Ye+<}bIh(33#b1hpwF*Uer(3T^G zt$piI{p-FIQALnk~PxpRI7sL_#+Eobs-mZx2XOL6vZHbc4gV!J@p@($T6#q`y#KLuz0? zrtIZpAWXaQWpn9pyQ**J$Ev9PHloS+!!9TpvYFM;RYTW1_i@|^83=x|ZqCt7W89X$ z0r2K$tRzY4RM|RQ5q8|Dcd*FRh#4#D3iba8J8N^wcaP1ia`o8TyqLRNd@m`aPvT^X z?1B&P+gHzmPOIAOZLT=5Cc>Jq>OWmce&R)`Nta-b^pfhQt*CTK?&J1mLQ7ZDV+z+` zx!=&P&K`m1_V?sabM&GorBRZbWxkWaPzZx#uI>Ie1&W))X&%j0ZOiAU;?^gk&Eovi zpF-LsdvC9bWyMN5-6@TY?Wa}miQ&PTU<0T2GonCP;nwe%E)j_g4QI0P$vSv;gDedL z&rysH!I>QUv8_n@;zD_Ut=B;}PfRa`@DpQvejH4W*48__U`<+`Th5v`-wrVliS8!- zurih~%Dd#0Zm09oQ$^kS!bsrpJx^+VQZpxdCKI zi=LUcH85brkZU5Q8F?YjD#rWMV<=bOavugmb##>k1jxVhJPB^F`g{DMO52&8<6Q6W!mrBRJ%hLJ zQInYW{DT}fKEs*c3SnW)cax9uuU)n{s%sm#>bmAp$_}D)G8WZ`j17?c)e{Ew=FLp( zV~Mef>iXH*hu6!_OV!#ND{xl^IzL_MI>}r0D5t3E2I#gv{ z8TuU-JV9=m5xtwSY+M|EPzH@3G78oDylep&vBF$;6@xuwA#{4ZIMi2G_$6f9bU zRS{p-r2-vh*Xk@LbILf0Z{q?CV{Qhc%OyCai1-yUki6!ZYpOCi-GsqPm#ae{?18T4 zQm$v`Oc4Ms0|sDCzyi=uQ;Wlb%{p;v8W zB?cp-UfMK&Gv0?0@h9Z$z-Wfu4m+SHavr?Tkg+IhwSRQO!eZGP^MyjVN@ChV*BR0< zT6?ls^%!XDF>IoAe6ST>6(#)?EoPsz`e9B1Y@Vuu#xE7)!1xW6D*sx{NO4L9Yp~JiFPFsqPFeBCy5r;d4GdKtwcGn0?%3xPpm& z`l{bud~~85BNV<4GSgxM|8Kl@Uef1ct5amYc}~E|`c&v;cpDdeBcv5Y&U25tulh+V z*J{_*{T=F~0(A4;y`G3mqX3es>a~QOuq8_tZhR?K_0EL?P6lFt5u2R#dR~E$mBJ7% zF)qf$FmJELb! zxBX#*(X#jMz^m5R`WKrSF)lmnK6KpFkMm4fpE!G6`~F|Um}FF}%M)17q3u2t2el$$ zg23&tV7-Q}=9f@x(6c7iIoIrhajG7sDeFUb{Hm%uWxz5Lq%S5z@sxFE@9Hm)7zSH9 zLbZF;lzO)H|iO0D*GoFI!47}Zmx>{`Eyce%Lp>(KG z4J7y6pHMGPKh{0?6Syp&tr`i40tFT+h^rJ+a}ajCXu*sa-XO|{tj_KY|Gtp~$kAy8fW zk%#-k50~{~(o5$J)xIwu&2XYh$n`#dboo>G5og>r`orG_K2IB9;G>&TMG!6$0|)y#O~Joi>Qo;NKJyxZqXFLkyK zkk-}J)wpj;I4^ZT81=`o=xYS=DLil6i0H)b16ST3GXYLcPBKAv>-WYCyuG}*EGMaZ z%8OCPU4(y7Xjf+!7YMWTnsFP4upnJ?bMyFWI|cBm-Nd)b*>_=$T2^A9mY$xlWnZ#^ zeRU=TkyOqguuXw~@1a5P5^Xkd8`Tz^c5Du~MYi;5xW6DhJ-y^@J4E~#7Z8PQCYI#D zi(Q=Vfb&<0#a?qXLvX0i#mU@5Za?n2-%XmT;+b4u-+r8(ojp4_aDbq+gN+%>-b8I< zV|(|nPdGVaZS5hbE=kZq;`Mg|6BCoSTj~Wh%xOUu?t-x+`~g;edjaT| zS*gc!87&p`^z;;vT?v9zX5c21b=#*0_3!p>-@Z+{3E`#qfv?RFz+8?7qM1Vy0|Emf zjypXck8_{(fSRhR&~6MzlUIiX^1-LHymHdgz84V*3k$#D6h*8JG*D7fhDH?_RH_(B z*An68bPDyoPu43-^_`u!A;_w^xp|$7e6wz+@3Y|S{rmTyAd@A$YhLMUXn@9b5X#GG zRKq7JX|g{GiM2J1ioNvudlLX$B8o+RIoYV?hQZJqo0 z=>6r1_SV)d!8>;hz@=&H;jCT=-1^+dXV!7=NX^i2y3|~Wn3x#usi&v6+?P`ROv&|m zpS;nVrD>9#NSK^`VxZXZuZ&P6YA{3IHdVLCkTpF0Y(O_E+vsCRNQmzMny=TAC6m@;@Z}r zYYL8Qb}zu>jE0@zBJa=>qgL^p$(hICwa<#6w2TZ+_8!vF(B#?UGgQkHKwVv31yUUA z_!a1Wh^|v5MX5)p==4`#3ZCRl!Fl1vEgT7q+ zok@(5eGM$Z{BN~JSf{moZq30#pXlc4QnYH>p5*Co8EYF&4UHD0&q3|n(aFgLHRl@c z*wfcL7Pgm?idfg2-emi;!BH?PEf6*?T0MWV5XMb29v>}jug+yvr5t5Y2*N%+6jV|9 z?$B_Q1<8nlf}%W5EoJ^a;ni86lXbSB`HEO}-BtW9e!O33+JYHS!CcQV0hh`$v}R7& zldRlHoHuSS1R_&XQr3l>YULy7pt~9%pXaujIB-eh5+0Al z#p*)9d$$L_?vi!9>Fd`&y15%^NJt*@!mi^dN-_%ca8T`p*!R%iaKpvLMe_I;<7Kr7 zioy&o&7I0c-Du&`{yG+0X3d!Ee5I>?pv-!h`Q?{1RRtS6PP-Xe`)9Q~!$DZ`gWAv& z{cQIjM)`1vJ_bv2J_Vo@d3y0SLOhG9H$_x;1pVK@|Ghr6(_&jWbms*Ek)eAwmB?%F z&2m@(o#OiaM3G8;rd-JQK|8DQt)oMs7MAS^DJa#}(a!S0%CLOjJ3VEtU;6GbGmHdF zFB>5PmU8Kd+26!*N?nbFxv}ReD}m3>LRe6d#SpT;GFx8@x!;oVZ*J}j6_Y8cK9@pH zobgYbq3!y4Fz4$#lg$5@nqZVc8%iA6hZ0YcA%s*XS8?*hKwF%1C6v8rpwGS#E8TE> zaw6D7S+N1Fw$`T_4{JRun!HrxulFbF4`#8D6zK>F(xm!ig)zLmyc(LCGt<*cVhi%y z1bgaWUJugu`lO=1%$cVgbWvc88OGm48RuR}_&&cQr8CK)VG_D>0Qo`vqNL}~pVN>L uSbhKB`PBcx=>Nfx|D8+!pH8U8a^x$#?v_j|<4Jevm;pHz*&^xJe*Xtr2}7U& literal 0 HcmV?d00001 diff --git a/img/online_delete-vs-ledger_history.png b/img/online_delete-vs-ledger_history.png new file mode 100644 index 0000000000000000000000000000000000000000..3268afa3fc12aac1588bbfe67ee889ba5e128cb3 GIT binary patch literal 8121 zcmbuEcT`hN*XU8iXcT=5F|*E-a?NcAiZ}35(R0}J7N?9h_uj~fD!4v zi1bJ^K?>?Zo@lBiOMq^9f^l|VL zw_2I~zD^RWj>gX$JAuz{>Gi#p^6XFFe|s-_KM5F`=s}$a4fwq90(t>>Ux%ER$(YXH z|185v{!jhy@*i=cf0X|z{?GD%-1OfyfRp%&T>9(sRY{k(Yum2v10NnfieX>&D|uUZ z`NI1lL+MAXh)L`r7J-oVUNIjs93b3){LFxVuW`jlVxI5R^r6;_FFQW7BBCwwpeu^B zAAWX1E9lmH`Wi>h1bkf#nIX@gx?20~ogEkq#wPYmyVyirSh)1(Pbf}W-eKh9la{dW zrIu%CQqH$`pD60+=tLZnrHr4#TdmS%eAjbgK_-m>c*9~7A@h3utinR0QZ?OuOmJ|H z@*Cw?4k5e2$@>+^B2TO8{>|BzjWVuVx7yIrBqStNjz3C(R_p8Qr>CbYxN^CuHCEDY zAz|VE$;rvy-rj2VRinW!3dXf{n3H~nWfr%_tL!ZuePytNZi7vY!ppx@;pPoK24pG5 z-0$rsT?C#^R)5U&!u1dn6ORB&ynp`yGn;H=;lI_&oK=Z`hGm9zISp* zeKH&DGCq%C2O3lf0$mz=gWc%%_V#-Aw>Ua=a6=|ZA$<&#kH5)yKIda}DKPa?L!?4Tor>{&~ zA&{|Jck5yYzJwvqY*3cBf$v@Gt*tHHntp`5t7(AkYmCzs>zXO#cmilCjEb!iKU9|J za|L~I{nVKvCHq%~I4H8~ql+@qXMbrhr{{CJ_{D(gUXvS>5*pJ;Hqm>wJy)g7Oy5bm zTC{{w;V;}Ykte2fiCS8ERv=qeRz}Duvu^j9YyYWX{ zU!>QEh5lc?nVF0H{QPP@agoKP%RfpHhE)uz%ThXszWPrxX(CarLk+$g%5PXzh1KeP zhlo@C4|*j;e7b=ZMA+YmgnFoXO>e+xVX4Jc{iyM*hO+ zm+U@qc0}vm%*86fq;a3t{o0Vp;Iud&6J4mS&1y938Koq+ppy$#fVJpQyCo8GOWfO~$9k?RqgMoTMgH z;vM_ds~=14C##F`ejasXh87kUy&2-zp7eQW2{Q3!UAYYSql#L81i|YPDK+4u@apFk zEW?}I+W|*wHQ!&GB3jo0&w9hB_yE;qj$#o1QFQJ{-uHB4;5IeqpD+0x>DYsiZ?@@9 znQxEX7%SsGA9=BJp)-*)=wKFwLPbYM8&}w}%!CUvqv?;2j{z^K0M>a?QIRLMPsprx z^JE`mw4i1}yoPV9tQ2ceU$<2?#wxi>s}}3!YZ55)0oM={^Xf?xX^-VhZBr-sQ?ZK0 zO^>{0>$7TTLwh7HG6|V|G!$CnyeyL-vMeiP@wrwbQ)0+E z5tgTgurM<#0>z&PQFcMeqRqi|EXsm z5ULmJJv=>~B?p0^OjHuvYGprOzPLJG;s5QTFQN+)!9=O z_Jy0N_UJs>ZHAKoHUAR&_Zpzti=8XX%)T3q`!n-IE-o$+5fMpOeL0C}1&MewA9462 zd$5k(DMIq62i9GY4K(Ln+*eXjQAy@EVCesWIXm6&&6I2o!1LR*PLmQ5QxdVa4iD>X zg*%U-W>bVrd3pY5Kz@(?e$18NY?Oz~5UwmPD z^Uj?TpS1}B_uhxs946IHEUjPvXyg(Fq?PYQ#(M8c@hjhf{B(`vU;m`m*4Ea0;~XsI zDX6G&AN(2Qfz;+oVQCk)&lu_n0|{V@^%qd8Am?vW{mlnjK_SPfBuo&u^iao^t2A$= zd~a+xfZS`qN+YW4Af2rc=n3->t(`5Fo_N-3>yX(pEbN*EJ1Vl4@*#(}$9|>@-;Hnz zO5*PDU0;6&x>xEHDV4q-+sp+Z^s?LJ#8GsD@atVtNFQzp8I!{G^A~M2l#Yxk1S$en zp(TccuS1A|G8|0%n2!FBNXkxY_>cDHd$3!J+IL8-V1guplCjXNNSP}^TqdR#<+7Vf z`j3?jW6Y*&&L(s(*i~8>DjknUy$+Vvv2`FDEe3Dqe(l#{94`s>bL{Lo`1IdV2tDb9OOW@BlqBa)%WSq9!( za`DG=y(e}`C>Nn7klx2&4%H&?=MVis>UET(n6HsM?xp5TRr*p~kT4{zAYh;(j2i0H zcgf9IrN8I&aP?3e^ZZAXV{9sEb4pS|7}Q{_j<%>v0{lz}GU>bC+1I!|{~?D_Hlykc zI@bEC+t=avGP<4*J&~#XUMO(AvPj-2EiMuB+*FD(#ZqZ7egQ7({f$ms=cWNm+X^sV z_jh9T*rl{FL+5wL&(T(ETFVcGA(chyHK6ix(+%b(3Nvfm1>2*g7HbpssBnqNVOqTz z{4llszE}P%ce}C@Vn`AnG@E;9d1vtK_(+KpQLr&CXQXD*7EN=ESU~y5E}z z?kzt8me_(4q96y8cJTD5Hm` zC$sISZa#Y}^N3VE_(STXI@Ra}CH(WM=_7qQH9>{3$MGPBL`>@El;?Ev=d9-Y_^CzM(4v!;FsZ-Lx(rE$6e-`-<(bq| zq}S>|pvq$`Zr;?&eDV&{3XXF^cRToYWcDz{(9Sju594K7XxaA|BO1or*o9abljG@7 zwwF&umPu;v!EG{=6aoy}n1ceFHm!W@!+37fxHdMaCb?VS3DH)l;qQy|#r#cnIc>+}25L;~dj0S~6fKvQZac*IG(W)=EKVL;#MGhE~s9P&1YMD7OZLVWY-w<57_YFc( zZaQ@Qo#QY0<4NDOtrqk{<~uxH{@`WJchbd_hTZbj<`qw@t#rG7t;tNw$+lKXI8EIQ zb~z66jJiGH(1+}Z2#DOw8AqHhkN0I(J2fYz9b1pZ7Yy5e6z(d4nrmtvBIM6x4X?x1cIjaLYj8;*SXGE-jNu{_&2kPRfZY zBX^Z_y;or;-&3u#;MwoZAgGB|RFRx1uf%4D;k?T`86kn7JJBkQP4=3P2ZhHEXP&gi zk|yttyOM_sh>@lEC66WFPpJ+vd^-4~@Jaq;I+xf@8(Kj=t*sle*G0fl3)&0HlVZZv zOW0`m>5-5Yaq@uhQ+J#8iWv=X* zVZj8wVQ^T?vDa}Pv3ehm^fsR0>UtSQ;5usu|mKB8uq?=(8AnD>-3DzPZ+ zi%>*;yCydQVnlgV%`@9bfO$D?-5yF|CV7s3oG+hm^K>VtIE_zHgi+O}z2wXTZes}( z>MFO#Q9EKHi^RLNYO-XoXd%o_`;xo{b2L*>Evp-&A!=a zL#Mx<$Bw0x-~8t}(*|KiE(lr570S`XwT3RmvG`5LjpmxR90qyDIhiy$tO-$$*1yRFLQ0;U?J^BsNTH!h zLqTTW))q@rWX9d?zJ=w^wi%PGcjdWXF!DkzjM|76EnWIjtz0RF+D=c+ncyZ2JreZoF^VBy+pLcYEh_|d!BMvyJC&Bbxo|Q5Q^}8#?sc`7P$408R3&_+{>NAK^=N1v9GWHTZ4%sR zp1DSt>&~S>>#pY1O;j##ICihd;qRa9u*64nB$iDeCSJs?bCq7+IemS0d-O_hJ=7SX zp2=la@g%3o4iR$ZCnpqMLVLBA(gq}cPQ{m@W zYT!7!8^Yd&y8UWxNoLe@9EpIZQy8BdgQZl`qCq+O0HL8+D|~% zt$5F{R;3eg;~~bvd*vR~oN{;QD}v_pEa`>Wh;c_k3R3=ObbPY(LD@YIS1D2mfESl( zYUpU+kq4*2c+NsO)pA$ji+LiX4iRk(a@CCshqO^D5w{@a%ZX{(QR9cD=7F7!`yd(N zIyUw~xTS|H9v6iTG8C?Bw@GSpTBp_oYDt!2{ zpJOC8@!GrPAr0z%j(eKSf-QBrL!!r^gM$L<{_&H?FMrR%nl}q*_^#`0LfaE4Cn6tdqm~}dmh2xs z&hX(peu{j)Rm#h21nV@{{|VJaZ=%?^2$pK z8+Y$fk!pa32fRxct$fg59X&&V?MdMrrKH7?7yRSw@fZ$aGyodj1Un6qWcG`7P`h#p zobuIO7m_^lQbx;br35*n*VRnXqB5zo{COPB7GVh^bHiX~9|*0WGKwsl-D+u5r(TJ^ z@8FC}|9N`i&*0>FhO-=wi-5b38ASnG;=$&mAd^>t?tbL%-MZ z=mgIAq>%9Yh(yK&Z#_Rc{XQdwKyN^Z74H7)==A#-6$;(zxcz%rI5aZu0g4Rhi;E0S zCwLL~A&%lxgrC<%UH|`xoF7E~YnePU^8c!v|67Dl@wX!X(1upXU){mpWpBTetFzN-bgmNU=WM zkcF=|%bWHZY9~4eA7m+-%F1D-r5i74KnLS?2u3C*=`HUY5JgNA3_$4`L5pttWGVkI z=#B9LXn{_S@^tXo2|!j^bZ4m#W)>VBi+2E+t z&DZh}$@ecwX@X`$nMb55HSY5AegvR4z$`ofnz6qx%+D_>#&CFeIAkIDY~Z~~l>@;3 z&}j7Ncw*Ew2`9)20iUaY0EnMYt#LV#_PQ3t5e)zhV*pYk4fBx5`;GU3ogLFuC|un0 z*JB!*SQP56dtO{zTmXKh*u24TQp|qn=3vI==H|$VegX5)-mkkDOct4+AGPI#X)-^$Cn;&F_ z@L}j|d8}+cp8Jv6{8xbjC||(afQEu4I|0IQb)&BTMdLHmu(Y(RE0=1kp{mpWLfWuG z0BM)IEp{cdiaQipHd~ll{!O%PBOFvNUFS4XjnWX=^qfV(%XpKn~M@&$qc*9GPkx?titpq zLi_SSh8$0g^W?8zzw+J)AV4q^2lQB(b(O|O(@}{bY4-`a0FEb7BSXx%F`!ByIhco0 z5FD%LD#>`_RMAvwT$)5DSX)S}?(k+LHLz3%m}h>2PaOyVjHPM-7c1q00UtJ|0B%)m zJMOZAX>LB(Tzt#55uakb*L3)c&AvAm)a6lX(Wo4KRmYZ^id@YU)8Om6Y;$;U;575u zn0RnNTj|lGZUC;A5pHaMS(<5Vc-P3Mm$#I;Kr5vkFZjZfJZ1WV30`lz>1aLib*Z4R zak*0EYFRr6z#=eBC#0OuH%(=q5U*?z_l@Rc|Gi(uI&Cnqev*)oCg9&AS|<_y^+&BU0u??>r-DtDKWukFz3V(66b9>A)($qo_zp! zpI;0~91;jCnG6NVd=S8`_osakuV25$Cq9VJyMZ1U7!a{;W0wyyQ&JNheXnq7yyDw0 zp8eOHK@bi`78Vu&z>9ufa4|ylB>dL)Xva|gd~F&w_&AUAGinobhV!*Jfnaockx2`J zWfJs`)kARU?^ZCL*f%r|gOvyUM~R@pXvAW!rc-$ngr5)$cQT<{>ym1-4 z;sXC0wXe`|iW0dgqxnTRy_ptx{aIHqIV(fCQG4fhGXhfM$5$$)wsYSi-mpmq{R|;R z_EJ$&<~;t7vgV(}`rk^p|AVIgo~!?ZH18&KDu02~1>+hTVFBb_B2}=aQi&oI{=We0 C?MYbx literal 0 HcmV?d00001