NuDB is a high performance key/value database optimized for insert-only
workloads, with these features:
* Low memory footprint
* Values are immutable
* Value sizes from 1 2^48 bytes (281TB)
* All keys are the same size
* Performance independent of growth
* Optimized for concurrent fetch
* Key file can be rebuilt if needed
* Inserts are atomic and consistent
* Data file may be iterated, index rebuilt.
* Key and data files may be on different volumes
* Hardened against algorithmic complexity attacks
* Header-only, nothing to build or link
* Add Backend::verify API for doing consistency checks
* Add Database::close so caller can catch exceptions
* Improved Timing test for NodeStore creates a simulated workload
Unit tests that wish to spawn threads for testing concurrency may now do so
by using unit_test::thread as a replacement for std::thread. These threads
propagate unhandled exceptions to the unit test, and work with the abort on
failure feature.
This removes the old default configuration for the "rocksdb" backend and
replaces it with the configuration that was formerly available using
the experimental backend "rocksdbquick".
The new configuration setting improves the performance of the key/value
database by changing the compaction style and tuning the size parameters for
the typical rippled workload. Testing shows a decrease in I/O spikes for both
reading and writing.
An alternative to the unity build, the classic build compiles each
translation unit individually. This adds more modules to the classic build:
* Remove unity header app.h
* Add missing includes as needed
* Remove obsolete NodeStore backend code
* Add app/, core/, crypto/, json/, net/, overlay/, peerfinder/ to classic build
The SHAMap class is refactored into a separate module where each translation
unit compiles separate without errors. Dependencies on higher level business
logic are removed. SHAMap now depends only on basics, crypto, nodestore,
and protocol:
* Inject NodeStore::Database& to SHAMap
* Move sync filter instances to app/ledger/
* Move shamap to its own module
* Move FullBelowCache to shamap/
* Move private code to shamap/impl/
* Refactor SHAMap treatment of missing node handler
* Inject and use Journal for logging in SHAMap
The SConstruct is modified to provide a new family of targets, ending with
the suffix ".nounity", which compile individual translation units instead of
some of the unity translation units ("classic" builds). Two modules updated
for this treatment are ripple/basics/ and ripple/protocol/, with plans to
update more in the future. A consequence is longer build times in some cases.
A benefit of classic builds is that missing includes can be identified
through compiler errors.
Source files are split to place all unit test code into translation
units ending in .test.cpp with no other business logic in the same file,
and in directories named "test".
A new target is added to the SConstruct, invoked by:
scons count
This prints the total number of source code lines occupied by unit tests,
in rippled specific code and excluding library subtrees.
The synthetic field 'delivered_amount' can be used to determine the exact
amount delivered by a Payment without having to check the DeliveredAmount
field, if present, or the Amount field otherwise.
The field is only returned when metadata is available and the data is not
returned in binary format.
* Added new test APIs allowing easy ways to create ledgers, apply
transactions to them, close and advance them.
* Moved Ledger tests from Ledger.cpp to Ledger.test.cpp.
* Changed several TransactionEngine log priorities from lsINFO to lsDEBUG to
reduce unnecessary verbosity in the log messages while running these tests.
* Moved LedgerConsensus:applyTransactions from a private member function to a
free function so that it could be accessed externally, and without having to
reference a LedgerConsensus object. This was done to facilitate the new
testing API.
Source files are moved between modules, includes changed and added,
and some code rewritten, with the goal of reducing cross-module dependencies
and eliminating cycles in the dependency graph of classes.
* Remove RippleAddress dependency in CKey_test
* ByteOrder.h, Blob.h, and strHex.h are moved to basics/. This makes
the basics/ module fully independent of other ripple sources.
* types/ is merged into protocol/. The protocol module now contains
all primitive types specific to the Ripple protocol.
* Move ErrorCodes to protocol/
* Move base_uint to basics/
* Move Base58 to crypto/
* Remove dependence on Serializer in GenerateDeterministicKey
* Eliminate unity header json.h
* Remove obsolete unity headers
* Remove unnecessary includes
* Generic functions to add entries to both object models.
* Add Json::Value into JsonObjects.
* Write Json::Value to string incrementally.
* Get rid of ripple::RPC::New namespace
These identifiers were part of a failed set of classes to replace
the functionality combined into RippleAddress. They are not used
and therefore can be removed.
* Remove RippleAccountPrivateKey
* Remove RippleAccountPublicKey
* Remove RippleAccountID
* Remove RipplePrivateKey
* Remove RipplePublicKeyHash
* Remove RippleLedgerHash
* Remove unused withCheck argument
* Remove CryptoIdentifier
* Remove IdentifierStorage
* Remove IdentifierType
* Remove SimpleIdentifier
* Add missing include
This replaces the experimental validators module with foundational
code to implement a new system for tracking validators, validations and
the UNL. The code is turned off by default, in BeastConfig.h
* Remove obsolete public Manager interfaces
* Remove obsolete database methods
* Remove obsolete ChosenList concept
* Remove obsolete code
* Add missing includes
* Tidy up STValidation.h
* Move factory function to Validators::make_Manager
* Add Connection object for tracking STValidations
All of the logic for establishing an outbound peer connection including
the initial HTTP handshake exchange is moved into a separate class. This
allows PeerImp to have a strong invariant: All PeerImp objects that exist
represent active peer connections that have already gone through the
handshake process.
* Replace SYSTEM_NAME and other macros with C++ constructs
* Remove RIPPLE_ARRAYSIZE and use std::extent or ranged for loops
* Remove old-style, unused offer crossing unit test
* Make STAmount::saFromRate free and remove default argument
This replaces the stateful class parser with a stateless free function.
The protocol buffer message is parsed using a ZeroCopyInputStream.
* Invoke method is now a free function.
* Protocol handler doesn't need to derive from an abstract interface
* Only up to one message is processed at a time by the invoker.
* Remove error_code return from the handler's message processing functions.
* Add ZeroCopyInputStream implementation that wraps a BufferSequence.
* Free function parses up to one protocol message and calls the handler.
* Message type and size can be calculated from an iterator
range or a buffer sequence.
* Fix to_string conversion
* Fix assert on debug invariant checks
* Fix the treatment of the output position when the entire output is committed.
* Add unit test
The abstract_clock is now templated on a type meeting the requirements of
the Clock concept. It inherits the nested types of the Clock on which it
is based. This resolves a problem with the original design which broke the
type-safety of time_point from different abstract clocks.
Makes rippled configurable to support deletion of all data in its key-value
store (nodestore) and ledger and transaction SQLite databases based on
validated ledger sequence numbers. All records from a specified ledger
and forward shall remain available in the key-value store and SQLite, and
all data prior to that specific ledger may be deleted.
Additionally, the administrator may require that an RPC command be
executed to enable deletion. This is to align data deletion with local
policy.
This introduces a considerable change in the way that peers handshake. Instead
of sending the TMHello protocol message, the peer making the connection (client
role) sends an HTTP Upgrade request along with some special headers. The peer
acting in the server role sends an HTTP response completing the upgrade and
transition to RTXP (Ripple Transaction Protocol, a.k.a. peer protocol). If the
server has no available slots, then it sends a 503 Service Unavailable HTTP
response with a JSON content-body containing IP addresses of other servers to
try. The information that was previously contained in the TMHello message is
now communicated in the HTTP request and HTTP response including the secure
cookie to prevent man in the middle attacks. This information is documented
in the overlay README.md file.
To prevent disruption on the network, the handshake feature is rolled out in
two parts. This is part 1, where new servents acting in the client role will
send the old style TMHello handshake, and new servents acting in the server
role can automatically detect and accept both the old style TMHello handshake,
or the HTTP request accordingly. This detection happens in the Server module,
which supports the universal port. An experimental .cfg setting allows clients
to instead send HTTP handshakes when establishing peer connections. When this
code has reached a significant fraction of the network, these clients will be
able to establish a connection to the Ripple network using HTTP handshakes.
These changes clean up the handling of the socket for peers. It fixes a long
standing bug in the graceful close sequence, where remaining data such as the
IP addresses of other servers to try, did not get sent. Redundant state
variables for the peer are removed and the treatment of completion handlers is
streamlined. The treatment of SSL short reads and secure shutdown is also fixed.
Logging for the peers in the overlay module are divided into two partitions:
"Peer" and "Protocol". The Peer partition records activity taking place on the
socket while the Protocol partition informs about RTXP specific actions such as
transaction relay, fetch packs, and consensus rounds. The severity on the log
partitions may be adjusted independently to diagnose problems. Every log
message for peers is prefixed with a small, unique integer id in brackets,
to accurately associate log messages with peers.
HTTP handshaking is the first step in implementing the Hub and Spoke feature,
which transforms the network from a homogeneous network where all peers are
the same, into a structured network where peers with above average capabilities
in their ability to process ledgers and transactions self-assemble to form a
backbone of high powered machines which in turn serve a much larger number of
'leaves' with lower capacities with a goal to improve the number of
transactions that may be retired over time.
Split out and rename STValidation
Split out and rename STBlob
Split out and rename STAccount
Split out STPathSet
Split STVector256 and move UintTypes to protocol/
Rename to STBase
Rename to STLedgerEntry
Rename to SOTemplate
Rename to STTx
Remove obsolete AgedHistory
Remove types.h and add missing includes
Remove unnecessary includes in app.h
Remove unnecessary includes in app.h
Remove include app.h from app1.cpp
This transforms a ConstBufferSequence into a new ConstBufferSequence whose
data is encoded according to the Content transfer encoding rules of RFC2616.
The implementation does not copy any memory.
* Remove CKey dependency on RippleAddress
* Create RAII ec_key wrapper that hides EC_KEY and other OpenSSL details
* Move CKey member logic into free functions
* Delete CKey class
* Rename units that are no longer CKey-related
* Delete code that was unused
