This adds codecs for snappy and lz4, and a new nodeobject codec. The
nodeobject codec provides a highly efficient custom compression scheme
for inner nodes, which make up the majority of nodestore databases.
Non inner node objects are compressed using lz4.
The NuDB backend is modified to use the nodeobject codec. This change
is not backward compatible - older NuDB databases cannot be opened or
imported.
This introduces changes in nudb to improve speed, reduce database size,
and enhance correctness. The most significant change is to store hashes
rather than entire keys in the key file. The output of the hash function
is reduced to 48 bits, and stored directly in buckets.
The API is also modified to introduce a Codec parameter allowing for
compression and decompression to be supported in the implementation
itself rather than callers.
THe data file no longer contains a salt, as the salt is applicable
only to the key and log files. This allows a data file to have multiple
key files with different salt values. To distinguish physical files
belonging to the same logical database, a new field UID is introduced.
The UID is a 64-bit random value generated once on creation and stored
in all three files.
Buckets are zero filled to the end of each block, this is a security
measure to prevent unintended contents of memory getting stored to
disk. NuDB offers the varint integer type, this is identical to
the varint described by Google.
* Add varint
* Add Codec template argument
* Add "api" convenience traits
* Store hash in buckets
* istream can throw short read errors
* Support std::uint8_t format in streams
* Make file classes part of the public interface
* Remove buffers pessimization, replace with buffer
* Consolidate creation utility functions to the same header
* Zero fill unused areas of buckets on disk
* More coverage and improvements to the recover test
* Fix file read/write to loop until all bytes processed
* Add verify_fast, faster verify for large databases
The database version number is incremented to 2; older databases can
no longer be opened and should be deleted.
This reverts the change that makes RocksDBQuick the default settings for
node_db "type=rocksdb". The quick settings can be obtained by setting
"type=rocksdbquick".
RocksDBQuick settings are implicated in memory over-utilization problems
seen recently.
This adds support for a cgi /crawl request, issued over HTTPS to the configured
peer protocol port. The response to the request is a JSON object containing
the node public key, type, and IP address of each directly connected neighbor.
The IP address is suppressed unless the neighbor has requested its address
to be revealed by adding "Crawl: public" to its HTTP headers. This field is
currently set by the peer_private option in the rippled.cfg file.
Insert now blocks when the size of the memory pool exceeds a predefined
threshold. This solves the problem where sustained insertions cause the
memory pool to grow without bound.
* Remove unused members
* SerialIter holds only a pointer and offset now
* Use free functions for some Serializer members
* Use SerialIter in some places instead of Serializer
The LevelDB and HyperLevelDB are removed from the backend choices. Neither
were recommended for production environments. As RocksDB is not available
on Windows platforms yet, the recommended backend choice for Windows is NuDB.
Add metrics to record the number of RPC requests received. Record the number of
node store fetches performed per request. Additionally record the byte size of
each request response and measure the response time of each request in
milliseconds.
A new class, ScopedMetrics, uses the Boost Thead Local Storage mechanism to
efficiently record NodeStore metrics within the same thread.
* The rippled.cfg file has a new section called "amendments"
* Each line in this section contains two white-space separated items
** The first item is the ID of the amendment (a 256-bit hash)
** The second item is the friendly name
* Replaces config section name macros with variables
* Make addKnown arguments safer
* Added lock to addKnown
The PreviousTxnID field has been deprecated and should not be used for
transactions that use the field will now be rejected.
The AccountTxnID feature should be used instead by enabling transaction
tracking and specifying a transaction ID at submission. More details
are available at: https://ripple.com/build/transactions/#accounttxnid
When clearing out a message key the transactor would incorrectly
create an empty `sfMessageKey` field instead of simply deleting
the field.
Clarify logic by reordering checks.
Autobridging uses XRP as a natural bridge currency to allow IOU-to-IOU orders
to be satisfied not only from the direct IOU-to-IOU books but also over the
combined IOU-to-XRP and XRP-to-IOU books.
This commit addresses the following issues:
* RIPD-486: Refactoring the taker into a unit-testable architecture
* RIPD-659: Asset-aware offer crossing
* RIPD-491: Unit tests for IOU to XRP, XRP to IOU and IOU to IOU
* RIPD-441: Handle case when autobridging over same owner offers
* RIPD-665: Handle case when autobridging over own offers
* RIPD-273: Groom order books while crossing
The new interfaces take into account the different semantics of XRP, which
do not have an issuer or transfer fees, and IOUs which have issuers and
(optional) transfer fees.
For XRP, the new `LedgerEntrySet::transfer_xrp` will transfer the specified
amount of XRP between from a given source to a given destination.
For IOU, two new functions are introduced:
* `LedgerEntrySet::issue_iou` which transfers the specified amount of an
IOU from the IOU's issuer to an account.
* `LedgerEntrySet::redeem_iou` which transfers the specified amount of an
IOU from an account to the IOU's issuer.
A transfer from user-to-user (e.g. to fill an order during offer crossing)
requires the use of `redeem_iou` followed by `issue_iou`. This helps to
enforce the Ripple invariant that IOUs never flow directly from user to
user, but only through a gateway. Additionally, this allows for the
explicit calculation and application of transfer fees by varying the
amounts redeemed and issued.
The new interfaces promote type safety since you cannot use the issue
and redeem APIs with XRP or the transfer API with IOU, and the issuer
to be used is implied by the currency being issued or redeemed.
