This introduces a considerable change in the way that peers handshake. Instead
of sending the TMHello protocol message, the peer making the connection (client
role) sends an HTTP Upgrade request along with some special headers. The peer
acting in the server role sends an HTTP response completing the upgrade and
transition to RTXP (Ripple Transaction Protocol, a.k.a. peer protocol). If the
server has no available slots, then it sends a 503 Service Unavailable HTTP
response with a JSON content-body containing IP addresses of other servers to
try. The information that was previously contained in the TMHello message is
now communicated in the HTTP request and HTTP response including the secure
cookie to prevent man in the middle attacks. This information is documented
in the overlay README.md file.
To prevent disruption on the network, the handshake feature is rolled out in
two parts. This is part 1, where new servents acting in the client role will
send the old style TMHello handshake, and new servents acting in the server
role can automatically detect and accept both the old style TMHello handshake,
or the HTTP request accordingly. This detection happens in the Server module,
which supports the universal port. An experimental .cfg setting allows clients
to instead send HTTP handshakes when establishing peer connections. When this
code has reached a significant fraction of the network, these clients will be
able to establish a connection to the Ripple network using HTTP handshakes.
These changes clean up the handling of the socket for peers. It fixes a long
standing bug in the graceful close sequence, where remaining data such as the
IP addresses of other servers to try, did not get sent. Redundant state
variables for the peer are removed and the treatment of completion handlers is
streamlined. The treatment of SSL short reads and secure shutdown is also fixed.
Logging for the peers in the overlay module are divided into two partitions:
"Peer" and "Protocol". The Peer partition records activity taking place on the
socket while the Protocol partition informs about RTXP specific actions such as
transaction relay, fetch packs, and consensus rounds. The severity on the log
partitions may be adjusted independently to diagnose problems. Every log
message for peers is prefixed with a small, unique integer id in brackets,
to accurately associate log messages with peers.
HTTP handshaking is the first step in implementing the Hub and Spoke feature,
which transforms the network from a homogeneous network where all peers are
the same, into a structured network where peers with above average capabilities
in their ability to process ledgers and transactions self-assemble to form a
backbone of high powered machines which in turn serve a much larger number of
'leaves' with lower capacities with a goal to improve the number of
transactions that may be retired over time.
Split out and rename STValidation
Split out and rename STBlob
Split out and rename STAccount
Split out STPathSet
Split STVector256 and move UintTypes to protocol/
Rename to STBase
Rename to STLedgerEntry
Rename to SOTemplate
Rename to STTx
Remove obsolete AgedHistory
Remove types.h and add missing includes
Remove unnecessary includes in app.h
Remove unnecessary includes in app.h
Remove include app.h from app1.cpp
* Remove CKey dependency on RippleAddress
* Create RAII ec_key wrapper that hides EC_KEY and other OpenSSL details
* Move CKey member logic into free functions
* Delete CKey class
* Rename units that are no longer CKey-related
* Delete code that was unused
* New src/ripple/crypto and src/ripple/protocol directories
* Merged src/ripple/common into src/ripple/basics
* Move resource/api files up a level
* Add headers for "include what you use"
* Normalized include guards
* Renamed to JsonFields.h
* Remove obsolete files
* Remove net.h unity header
* Remove resource.h unity header
* Removed some deprecated unity includes
* Restrict to 80-columns and other style cleanups.
* Make pathfinding a free function and hide the class Pathfinder.
* Split off unrelated utility functions into separate files.
Conflicts:
src/ripple/rpc/handlers/RipplePathFind.cpp
This changes the behavior and configuration specification of the listening
ports that rippled uses to accept incoming connections for the supported
protocols: peer (Peer Protocol), http (JSON-RPC over HTTP), https (JSON-RPC)
over HTTPS, ws (Websockets Clients), and wss (Secure Websockets Clients).
Each listening port is now capable of handshaking in multiple protocols
specified in the configuration file (subject to some restrictions). Each
port can be configured to provide its own SSL certificate, or to use a
self-signed certificate. Ports can be configured to share settings, this
allows multiple ports to use the same certificate or values. The list of
ports is dynamic, administrators can open as few or as many ports as they
like. Authentication settings such as user/password or admin user/admin
password (for administrative commands on RPC or Websockets interfaces) can
also be specified per-port.
As the configuration file has changed significantly, administrators will
need to update their ripple.cfg files and carefully review the documentation
and new settings.
Changes:
* rippled-example.cfg updated with documentation and new example settings:
All obsolete websocket, rpc, and peer configuration sections have been
removed, the documentation updated, and a new documented set of example
settings added.
* HTTP::Writer abstraction for sending HTTP server requests and responses
* HTTP::Handler handler improvements to support Universal Port
* HTTP::Handler handler supports legacy Peer protocol handshakes
* HTTP::Port uses shared_ptr<boost::asio::ssl::context>
* HTTP::PeerImp and Overlay use ssl_bundle to support Universal Port
* New JsonWriter to stream message and body through HTTP server
* ServerHandler refactored to support Universal Port and legacy peers
* ServerHandler Setup struct updated for Universal Port
* Refactor some PeerFinder members
* WSDoor and Websocket code stores and uses the HTTP::Port configuration
* Websocket autotls class receives the current secure/plain SSL setting
* Remove PeerDoor and obsolete Overlay peer accept code
* Remove obsolete RPCDoor and synchronous RPC handling code
* Remove other obsolete classes, types, and files
* Command line tool uses ServerHandler Setup for port and authorization info
* Fix handling of admin_user, admin_password in administrative commands
* Fix adminRole to check credentials for Universal Port
* Updated Overlay README.md
* Overlay sends IP:port redirects on HTTP Upgrade peer connection requests:
Incoming peers who handshake using the HTTP Upgrade mechanism don't get
a slot, and always get HTTP Status 503 redirect containing a JSON
content-body with a set of alternate IP and port addresses to try, learned
from PeerFinder. A future commit related to the Hub and Spoke feature will
change the response to grant the peer a slot when there are peer slots
available.
* HTTP responses to outgoing Peer connect requests parse redirect IP:ports:
When the [overlay] configuration section (which is experimental) has
http_handshake = 1, HTTP redirect responses will have the JSON content-body
parsed to obtain the redirect IP:port addresses.
* Use a single io_service for HTTP::Server and Overlay:
This is necessary to allow HTTP::Server to pass sockets to and from Overlay
and eventually Websockets. Unfortunately Websockets is not so easily changed
to use an externally provided io_service. This will be addressed in a future
commit, and is one step necessary ease the restriction on ports configured
to offer Websocket protocols in the .cfg file.
This new factory is intended for benchmarking against the existing RocksDBFactory and has the following differences.
* Does not use BatchWriter
* Disables WAL for writes to memtable
* Uses a hash index in blocks
* Uses RocksDB OptimizeFor… functions
See Benchmarks.md for further discussion of some of the issues raised by investigation of RocksDB performance.
This is a cleanup to the structure of the sources.
* Rename to ServerHandler
* Move private implementation declaration to separate header
* De-inline function definitions in the class declaration.
* Removed MultiSocket. Code that previously used the MultiSocket now uses
a combination of boost::asio coroutines and CRTP.
* Sitefiles headers rolled up and directory flattened.
* Disabled Sitefiles use of deprecated HTTPClient.
* Validators headers tidied up.
* Disabled Validators use of deprecated HTTPClient.
The MultiSocket is obsolete technology which is superceded by a more
straightforward, template based implementation that is compatible with
boost::asio::coroutines. This removes support for the unused PROXY handshake
feature. After this change a large number of classes and source files may be
removed.
When JSON-RPC and Websocket responses are calculated, the result is stored
in intermediate Json::Value objects and later composed in a single linear
memory buffer before being sent to the socket. These classes support a
new model for building responses that supports incremental construction
of JSON replies in constant time and removes the requirement that all
data returned be located in continuguous memory.
* New JsonWriter incrementally writes JSON with O(1) granularity and memory.
* Array, Object are RAII wrappers for the O(1) JsonWriter.
This class was used to allow stream style operator<< to write to the
HTTP::Session. This is being superceded by a more robust object-based model
that supports coroutines.
If beast::Time::currentTimeMillis is first called from a coroutine launched
using boost::asio::spawn, Win32 throws an exception. This workaround calls
getCurrentTime once in main to prevent the exception.
Reference:
https://svn.boost.org/trac/boost/ticket/10657
The MultiSocket class implements a socket that handshakes in multiple
protocols including SSL and PROXY. Unfortunately the way it type-erases the
handlers and buffers is incompatible with boost::asio coroutines. To pave the
way for coroutines this is part of a larger set of changes that roll back the
usage of MultiSocket to older code, and some custom implementations that use
templates. The custom implementations are more simple since they use
coroutines. Removing MultiSocket will make many other classes and source files
unused, a big win for trimming down the codebase size.
Previously, the PeerFinder manager constructed with a Callback object
provided by the owner which was used to perform operations like connecting,
disconnecting, and sending messages. This made it difficult to change the
overlay code because a single call into the PeerFinder could cause both
OverlayImpl and PeerImp to be re-entered one or more times, sometimes while
holding a recursive mutex. This change eliminates the callback by changing
PeerFinder functions to return values indicating the action the caller should
take.
As a result of this change the PeerFinder no longer needs its own dedicated
thread. OverlayImpl is changed to call into PeerFinder on a timer to perform
periodic activities. Furthermore the Checker class used to perform connectivity
checks has been refactored. It no longer uses an abstract base class, in order
to not type-erase the handler passed to async_connect (ensuring compatibility
with coroutines). To allow unit tests that don't need a network, the Logic
class is now templated on the Checker type. Currently the Manager provides its
own io_service. However, this can easily be changed so that the io_service is
provided upon construction.
Summary
* Remove unused SiteFiles dependency injection
* Remove Callback and update signatures for public APIs
* Remove obsolete functions
* Move timer to overlay
* Steps toward a shared io_service
* Templated, simplified Checker
* Tidy up Checker declaration
* Fix bug with more than one complete request in a read buffer
* Use stackful coroutines for simplified control flow
* Door refactored to detect handshakes
* Remove dependency on MultiSocket
* Remove dependency on handshake detect logic framework
* Remove unused functions
* Remove unused constructor
* Use delegating constructors
* Mark some observers deprecated
* Clean up declaration parameter names
* Add checked and unchecked constructors
* De-inline unnecessary inlined functions
* Reorder and regroup members into sections
* Move globals from the unity file to the .cpp
* Change some member functions to be free functions
* Put implementation in one .cpp and the test in another .cpp
Remove unused STAmount constructor and delegate two others No change in functionality.
* Remove obsolete config variables
* Reduce coupling
* Use C++11 ownership containers
* Use auto when it makes sense
* Detect edge-case in unit tests
* Reduce the number of LedgerEntrySet public members
* reduce duplicated code using templates
* replace BOOST_FOREACH with C++11 for loops
* remove most direct calls to new
* limit line length to 80 characters
* clearly identify virtual and overridden methods
* split STObject and STArray into their own files
* name files after the class they contain
* New CreateTicket transactor to create tickets
* New CancelTicket transactor to cancel tickets
* Ledger entries for tickets & associated functions
* First draft of M-of-N documentation
* Restrict access to SField constructors.
* Make all SField access const.
* Hide and simplify databases used to hold SField constants.
* Separate the two concerns of representing a field,
and maintaining a database of fields.
