The correct ledger age is necessary for checking health
status, and the previous behavior caused the online deletion process to
abort if the process took too long.
The tuning parameter added and the parameter whose default was modified both
minimize impact of SQL DELETE operations by decreasing the default batch size
for deletes and for increasing the backoff period between deletion batches.
These parameters decrease contention for the SQLite and I/O with the trade-off
of longer processing time for online delete. Online-delete is not a
time-critical function, so a little slowness in wall-clock time is not harmful.
Profiling indicated some performance issues coming from the
cache of 160-bit account IDs to base58 format. This replaces
the single cache with two caches and rotates out old
entries.
The abstract_clock is now templated on a type meeting the requirements of
the Clock concept. It inherits the nested types of the Clock on which it
is based. This resolves a problem with the original design which broke the
type-safety of time_point from different abstract clocks.
Makes rippled configurable to support deletion of all data in its key-value
store (nodestore) and ledger and transaction SQLite databases based on
validated ledger sequence numbers. All records from a specified ledger
and forward shall remain available in the key-value store and SQLite, and
all data prior to that specific ledger may be deleted.
Additionally, the administrator may require that an RPC command be
executed to enable deletion. This is to align data deletion with local
policy.
This introduces a considerable change in the way that peers handshake. Instead
of sending the TMHello protocol message, the peer making the connection (client
role) sends an HTTP Upgrade request along with some special headers. The peer
acting in the server role sends an HTTP response completing the upgrade and
transition to RTXP (Ripple Transaction Protocol, a.k.a. peer protocol). If the
server has no available slots, then it sends a 503 Service Unavailable HTTP
response with a JSON content-body containing IP addresses of other servers to
try. The information that was previously contained in the TMHello message is
now communicated in the HTTP request and HTTP response including the secure
cookie to prevent man in the middle attacks. This information is documented
in the overlay README.md file.
To prevent disruption on the network, the handshake feature is rolled out in
two parts. This is part 1, where new servents acting in the client role will
send the old style TMHello handshake, and new servents acting in the server
role can automatically detect and accept both the old style TMHello handshake,
or the HTTP request accordingly. This detection happens in the Server module,
which supports the universal port. An experimental .cfg setting allows clients
to instead send HTTP handshakes when establishing peer connections. When this
code has reached a significant fraction of the network, these clients will be
able to establish a connection to the Ripple network using HTTP handshakes.
These changes clean up the handling of the socket for peers. It fixes a long
standing bug in the graceful close sequence, where remaining data such as the
IP addresses of other servers to try, did not get sent. Redundant state
variables for the peer are removed and the treatment of completion handlers is
streamlined. The treatment of SSL short reads and secure shutdown is also fixed.
Logging for the peers in the overlay module are divided into two partitions:
"Peer" and "Protocol". The Peer partition records activity taking place on the
socket while the Protocol partition informs about RTXP specific actions such as
transaction relay, fetch packs, and consensus rounds. The severity on the log
partitions may be adjusted independently to diagnose problems. Every log
message for peers is prefixed with a small, unique integer id in brackets,
to accurately associate log messages with peers.
HTTP handshaking is the first step in implementing the Hub and Spoke feature,
which transforms the network from a homogeneous network where all peers are
the same, into a structured network where peers with above average capabilities
in their ability to process ledgers and transactions self-assemble to form a
backbone of high powered machines which in turn serve a much larger number of
'leaves' with lower capacities with a goal to improve the number of
transactions that may be retired over time.
Split out and rename STValidation
Split out and rename STBlob
Split out and rename STAccount
Split out STPathSet
Split STVector256 and move UintTypes to protocol/
Rename to STBase
Rename to STLedgerEntry
Rename to SOTemplate
Rename to STTx
Remove obsolete AgedHistory
Remove types.h and add missing includes
Remove unnecessary includes in app.h
Remove unnecessary includes in app.h
Remove include app.h from app1.cpp
* Use signal_set as cross platform way of handling SIGINT
* Remove polling on main thread for shutdown.
* Add extra logging for received signal.
* Clean up exit handling on error in setup routines.
* Reuse isStopped() from Stoppable for status (could be isStopping() instead).
* Ctrl-C should now work for standalone mode as well on Windows.
Also small fixes to Resolver:
* Add Resolver prefix to logging.
* Fix AsyncObject::removeReference() logic.
* Fix work remaining logic.
When the ServerHandler is constructed before the Overlay, an incoming
connection received after the server's listening ports have been opened
but before the Overlay object has been created causes a crash.
The RPC account_lines and account_offers commands respond with the correct
ledger info. account_offers, account_lines and book_offers allow admins
unlimited size on the limit param. Specifying a negative value on limit clamps
to the minimum value allowed. Incorrect types for limit are correctly reported
in the result.
* New src/ripple/crypto and src/ripple/protocol directories
* Merged src/ripple/common into src/ripple/basics
* Move resource/api files up a level
* Add headers for "include what you use"
* Normalized include guards
* Renamed to JsonFields.h
* Remove obsolete files
* Remove net.h unity header
* Remove resource.h unity header
* Removed some deprecated unity includes
The creation of self-signed certificates slows down the command
line client when launched repeatedly during unit test.
* Contexts are no longer generated for the command line client
* A port with no secure protocols generates an empty context
* Allow pathfinding requests where the starting currency may have
multiple issuers.
* Cache paths over all issuers to avoid repeating work.
* Clear the ledger checkpoint in one retry case.
* Add an additional node at the front of paths when the starting issuer
is not the source account.
* Restrict to 80-columns and other style cleanups.
* Make pathfinding a free function and hide the class Pathfinder.
* Split off unrelated utility functions into separate files.
Conflicts:
src/ripple/rpc/handlers/RipplePathFind.cpp
This changes the behavior and configuration specification of the listening
ports that rippled uses to accept incoming connections for the supported
protocols: peer (Peer Protocol), http (JSON-RPC over HTTP), https (JSON-RPC)
over HTTPS, ws (Websockets Clients), and wss (Secure Websockets Clients).
Each listening port is now capable of handshaking in multiple protocols
specified in the configuration file (subject to some restrictions). Each
port can be configured to provide its own SSL certificate, or to use a
self-signed certificate. Ports can be configured to share settings, this
allows multiple ports to use the same certificate or values. The list of
ports is dynamic, administrators can open as few or as many ports as they
like. Authentication settings such as user/password or admin user/admin
password (for administrative commands on RPC or Websockets interfaces) can
also be specified per-port.
As the configuration file has changed significantly, administrators will
need to update their ripple.cfg files and carefully review the documentation
and new settings.
Changes:
* rippled-example.cfg updated with documentation and new example settings:
All obsolete websocket, rpc, and peer configuration sections have been
removed, the documentation updated, and a new documented set of example
settings added.
* HTTP::Writer abstraction for sending HTTP server requests and responses
* HTTP::Handler handler improvements to support Universal Port
* HTTP::Handler handler supports legacy Peer protocol handshakes
* HTTP::Port uses shared_ptr<boost::asio::ssl::context>
* HTTP::PeerImp and Overlay use ssl_bundle to support Universal Port
* New JsonWriter to stream message and body through HTTP server
* ServerHandler refactored to support Universal Port and legacy peers
* ServerHandler Setup struct updated for Universal Port
* Refactor some PeerFinder members
* WSDoor and Websocket code stores and uses the HTTP::Port configuration
* Websocket autotls class receives the current secure/plain SSL setting
* Remove PeerDoor and obsolete Overlay peer accept code
* Remove obsolete RPCDoor and synchronous RPC handling code
* Remove other obsolete classes, types, and files
* Command line tool uses ServerHandler Setup for port and authorization info
* Fix handling of admin_user, admin_password in administrative commands
* Fix adminRole to check credentials for Universal Port
* Updated Overlay README.md
* Overlay sends IP:port redirects on HTTP Upgrade peer connection requests:
Incoming peers who handshake using the HTTP Upgrade mechanism don't get
a slot, and always get HTTP Status 503 redirect containing a JSON
content-body with a set of alternate IP and port addresses to try, learned
from PeerFinder. A future commit related to the Hub and Spoke feature will
change the response to grant the peer a slot when there are peer slots
available.
* HTTP responses to outgoing Peer connect requests parse redirect IP:ports:
When the [overlay] configuration section (which is experimental) has
http_handshake = 1, HTTP redirect responses will have the JSON content-body
parsed to obtain the redirect IP:port addresses.
* Use a single io_service for HTTP::Server and Overlay:
This is necessary to allow HTTP::Server to pass sockets to and from Overlay
and eventually Websockets. Unfortunately Websockets is not so easily changed
to use an externally provided io_service. This will be addressed in a future
commit, and is one step necessary ease the restriction on ports configured
to offer Websocket protocols in the .cfg file.
This fixes a case where stop can sometimes skip calling close on some
I/O objects or crash in a rare circumstance where a connection is in the
process of being torn down at the exact time the server is stopped. When
the acceptor receives errors, it logs the error and continues listening
instead of stopping.
These changes are necessary to support the Universal port feature. Synopsis:
* Persist HTTP peer io_service::work lifetime:
This simplification eliminates any potential for bugs caused by incorrect
lifetime management of the io_service::work object.
* Restructure Door to prevent data races, and handle clean exit:
The Server, Door, Door::detector, and Peer objects work together to
correctly implement graceful stop and destructors that block until
all child objects have been destroyed.
Cleanups:
* De-pimpl HTTP::Server
* Rename ServerImpl data members
* Tidy up HTTP::Port interface
These changes prepare Overlay for the Universal Port and Hub and Spoke
features.
* Add [overlay configuration section:
The [overlay] section uses the new BasicConfig interface that
supports key-value pairs in the section. Some exposition is added to the
example cfg file. The new settings for overlay are related to the Hub and
Spoke feature which is currently in development. Production servers should
not set these configuration options, they are clearly marked experimental
in the example cfg file.
Other changes:
* Use _MSC_VER to detect Visual Studio
* Use ssl_bundle in Overlay::Peer
* Use shared_ptr to SSL context in Overlay:
* Removed undocumented PEER_SSL_CIPHER_LIST configuration setting
* Add Section::name: The Section object now stores its name for better diagnostic messages.
This changes the http::message object to no longer contain a body. It modifies
the parser to store the body in a separate object, or to pass the body data
to a functor. This allows the body to be stored in more flexible ways. For
example, in HTTP responses the body can be generated procedurally instead
of being required to exist entirely in memory at once.
A string passed by the '--unittest-arg' command line parameter is passed to
suites when unit tests run and can be used to customize test behavior.
* Add '--unittest-arg' command line argument
* Remove obsolete '--unittest-format' command line argument
* Use static_assert where appropriate
* Use std::min and std::max where appropriate
* Simplify RippleD error reporting
* Remove use of beast::RandomAccessFile
This is a cleanup to the structure of the sources.
* Rename to ServerHandler
* Move private implementation declaration to separate header
* De-inline function definitions in the class declaration.
Many classes required to support type-erasure of handlers and boost::asio
types are now obsolete, so these classes and files are removed:
HTTPClientType, FixedInputBuffer, PeerRole, socket_wrapper,
client_session, basic_url, abstract_socket, buffer_sequence, memory_buffer,
enable_wait_for_async, shared_handler, wrap_handler, streambuf,
ContentBodyBuffer, SSLContext, completion-handler based handshake detectors.
These structural changes are made:
* Some missing includes added to headers
* asio module directory flattened
The MultiSocket class implements a socket that handshakes in multiple
protocols including SSL and PROXY. Unfortunately the way it type-erases the
handlers and buffers is incompatible with boost::asio coroutines. To pave the
way for coroutines this is part of a larger set of changes that roll back the
usage of MultiSocket to older code, and some custom implementations that use
templates. The custom implementations are more simple since they use
coroutines. Removing MultiSocket will make many other classes and source files
unused, a big win for trimming down the codebase size.
The implementation of multi-sign has a SigningAccounts array as a
member of the outermost object. This array could not be parsed
by the previous implementation of STParsedJSON, which only knew
how to parse objects. This refactor supports the required parsing.
The refactor divides the parsing into three separate functions:
o parseNoRecurse() which parses most rippled data types.
o parseObject() which parses object types that may contain
arbitrary other types.
o parseArray() which parses object types that may contain
arbitrary other types.
The change is required by the multi-sign implementation, but is
independent. So the parsing change is going in as a separate
commit.
The parsing is still far from perfect. But this was as much as
needs doing to accomplish the ends and mitigate risk of breaking
the parser.
This reworks the way SHAMaps are stored, flushed, backed, and
traversed. Rather than storing the linkages in the SHAMap itself,
that information is now stored in the nodes. This makes
snapshotting much cheaper and also allows traverse work done on
behalf of one SHAMap to be used by other SHAMaps that share inner
nodes with that SHAMap.
When a SHAMap is modified, nodes are modified all the way up to the
root. This means that the modified nodes in a SHAMap can easily be
traversed for flushing. So they don't need to be separately tracked.
Summary
* Remove mTNByID
* Remove mDirtyNodes
* Much faster traverses
* Much Faster snapshots
* New algorithm for flushing
* New vistNodes/visitLeaves
* Avoid I/O if a map is unbacked
