Commit Graph

117 Commits

Author SHA1 Message Date
seelabs
d89c158a77 Replace qalloc with boost::pmr 2020-11-18 13:25:27 -08:00
Miguel Portilla
8707c15b9c Use NuDB burst size and use NuDB version 2.0.5 2020-11-18 13:07:26 -08:00
Scott Schurr
52adcc73d9 Add unit tests to check synchronization of KnownFormats and gRPC 2020-09-17 15:05:59 -07:00
Devon White
2bba79138f Support shard downloading via HTTP or HTTPS 2020-09-17 15:01:22 -07:00
Gregory Tsipenyuk
707868be33 Implement sticky DNS resolution for validator list retrieval:
When attempting to load a validator list from a configured
site, attempt to reuse the last IP that was successfully
used if that IP is still present in the DNS response.

Otherwise, randomly select an IP address from the list of
IPs provided by the DNS system.

This commit fixes #3494.
2020-09-01 10:37:19 -07:00
Gregory Tsipenyuk
9b9f34f881 Optimize relaying of validation and proposal messages:
With few exceptions, servers will typically receive multiple copies
of any given message from its directly connected peers. For servers
with several peers this can impact the processing latency and force
it to do redundant work. Proposal and validation messages are often
relayed with extremely high redundancy.

This commit, if merged, introduces experimental code that attempts
to optimize the relaying of proposals and validations by allowing
servers to instruct their peers to "squelch" delivery of selected
proposals and validations. Servers making squelching decisions by
a process that evaluates the fitness and performance of a given
server and randomly selecting a subset of the best candidates.

The experimental code is presently disabled and must be explicitly
enabled by server operators that wish to test it.
2020-09-01 09:07:32 -07:00
Scott Schurr
7724cca384 Implement enhanced Ticket support:
Tickets are a mechanism to allow for the "out-of-order" execution of
transactions on the XRP Ledger.

This commit, if merged, reworks the existing support for tickets and
introduces support for 'ticket batching', completing the feature set
needed for tickets.

The code is gated under the newly-introduced `TicketBatch` amendment
and the `Tickets` amendment, which is not presently active on the
network, is being removed.

The specification for this change can be found at:
https://github.com/xrp-community/standards-drafts/issues/16
2020-09-01 08:58:57 -07:00
manojsdoshi
dbd5f0073e Revert support for deterministic shards:
Commit 4dc08f820258ac6c62071c7070c175e48352f8b3 introduced support for
deterministic shards, which makes the sharding functionality provided
by rippled more useful.

After merging, several opportunities for further improvements to the
deterministic sharding implementation were identified and a significant
increase int memory usage during shard finalization was detected.

Because of these issues, the commit is being reverted and the feature is
being rolled back. It will be reintroduced in a future release.
2020-08-06 10:11:55 -07:00
Edward Hennis
72a9a2bdbb Reorder the Travis build:
* Builds Windows dependencies first.
* Builds ALL OSs in the last stage.
* Fix the MacOS builds.
* Windows dependency stages are allowed to fail so ALL configurations will
  attempt to build. Windows builds will probably fail if dependencies fail
  (caching may allow them to succeed), but they will at least be attempted.
* Remove broken AppVeyor config file, so it stops trying.
2020-08-06 10:05:43 -07:00
seelabs
80860fa8f5 Add preliminary support for Boost 1.74 2020-08-06 10:05:43 -07:00
seelabs
a931b020b5 Switch to updated date library exception handling:
Without defining `HAS_UNCAUGHT_EXCEPTIONS` the date library would use the
deprecated `uncaught_exception` which would generate a warning.
2020-08-06 09:56:04 -07:00
Peng Wang
706ca874b0 Implement negative UNL functionality:
This change can help improve the liveness of the network during periods of network
instability, by allowing the network to track which validators are presently not online
and to disregard them for the purposes of quorum calculations.
2020-06-30 09:15:37 -07:00
Peng Wang
51bd4626b1 Implement version upgrade warning:
If the 'HardenedValidations' amendment is enabled, this commit will
track the version of the software that validators embed in their
validations.

If a server notices that at least 60% of the validators on its UNL
are running a newer version than it is running, it will periodically
print an informational message, reminding the operator to check for
update.
2020-06-30 09:15:37 -07:00
Miguel Portilla
86e8f2e232 Add Shard Family 2020-06-30 08:52:18 -07:00
Miguel Portilla
8f50fd051e Use NuDB version 2.0.3 2020-06-30 08:52:17 -07:00
p2peer
94e8e94750 Add support for deterministic database shards (#2688):
This commit, if merged, adds support to allow multiple indepedent nodes to
produce a binary identical shard for a given range of ledgers. The advantage
is that servers can use content-addressable storage, and can more efficiently
retrieve shards by downloading from multiple peers at once and then verifying
the integrity of a shard by cross-checking its checksum with the checksum
other servers report.
2020-06-30 08:52:17 -07:00
Edward Hennis
54ece72b62 Update cmake so that rippled can build as a submodule 2020-06-25 19:47:20 -07:00
Yusuf Sahin HAMZA
27484f78a9 Use libarchive version 3.4.3 2020-05-29 17:54:19 -07:00
p2peer
93bf77bdec Unit tests for database shards 2020-05-29 17:54:19 -07:00
Miguel Portilla
fb74eefa9e Update LZ4 library to 1.9.2 2020-05-29 17:54:19 -07:00
John Freeman
0b9e935806 Find date::date in CMake package configuration file 2020-05-26 18:41:30 -07:00
John Freeman
8f984042f4 Try to fix usage of pkg-config for static linking 2020-05-26 18:37:52 -07:00
Devon White
ac766ec0eb Introduce ShardArchiveHandler improvements:
* Improve documentation
* Make the ShardArchiveHandler rather than the DatabaseShardImp perform
  LastLedgerHash verification for downloaded shards
* Remove ShardArchiveHandler's singleton implementation and make it an
  Application member
* Have the Application invoke ShardArchiveHandler initialization
  instead of clients
* Add RecoveryHandler as a ShardArchiveHandler derived class
* Improve commenting
2020-05-26 18:37:01 -07:00
Devon White
21340a1c1e Validate LastLedgerHash for downloaded shards:
* Add documentation for shard validation
* Retrieve last ledger hash for imported shards
* Verify the last ledger hash in Shard::finalize
* Limit last ledger hash retrieval attempts for imported shards
* Use a common function for removing failed shards
* Add new ShardInfo::State for imported shards
2020-05-26 18:36:54 -07:00
Nik Bougalis
62a3f33d72 Remove the built-in "sustain" watchdog:
The built-in watchdog is simplistic and can, sometimes, cause problems
especially on systems that have the ability to automatically start and
monitor processes.

This commit removes the sustain system entirely, changes the handling
of the SIGTERM signal to properly terminate the process and improves
the error message reported to the user when the command line used to
start `rippled` is incorrect and malformed.
2020-05-05 16:05:23 -07:00
Nik Bougalis
dbee3f01b7 Clean up and modernize code:
This commit removes obsolete comments, dead or no longer useful
code, and workarounds for several issues that were present in older
compilers that we no longer support.

Specifically:

- It improves the transaction metadata handling class, simplifying
  its use and making it less error-prone.
- It reduces the footprint of the Serializer class by consolidating
  code and leveraging templates.
- It cleanups the ST* class hierarchy, removing dead code, improving
  and consolidating code to reduce complexity and code duplication.
- It shores up the handling of currency codes and the conversation
  between 160-bit currency codes and their string representation.
- It migrates beast::secure_erase to the ripple namespace and uses
  a call to OpenSSL_cleanse instead of the custom implementation.
2020-05-05 16:05:22 -07:00
Miguel Portilla
bd8dbb87b6 Update to RocksBD 6.7.3 2020-04-23 18:09:45 -07:00
Nik Bougalis
bdd22e4d51 Improve reporting of missing node exceptions 2020-04-14 19:42:50 -07:00
John Freeman
858e93c7f8 Improve the Doxygen Workflow
* Use official build environment Docker image
* Update documentation and Doxygen website URLs
2020-04-14 19:42:42 -07:00
John Freeman
25b13978e7 Fix unity build 2020-04-07 11:31:23 -07:00
John Freeman
3e9cff9287 Fix Doxygen build 2020-04-06 17:28:53 -07:00
Gregory Tsipenyuk
758a3792eb Add protocol message compression support:
* Peers negotiate compression via HTTP Header "X-Offer-Compression: lz4"
* Messages greater than 70 bytes and protocol type messages MANIFESTS,
  ENDPOINTS, TRANSACTION, GET_LEDGER, LEDGER_DATA, GET_OBJECT,
  and VALIDATORLIST are compressed
* If the compressed message is larger than the uncompressed message
  then the uncompressed message is sent
* Compression flag and the compression algorithm type are included
  in the message header
* Only LZ4 block compression is currently supported
2020-04-06 17:22:59 -07:00
Devon White
905a97e0aa Make ShardArchiveHandler downloads more resilient:
* Make ShardArchiveHandler a singleton.
* Add state database for ShardArchiveHandler.
* Use temporary database for SSLHTTPDownloader downloads.
* Make ShardArchiveHandler a Stoppable class.
* Automatically resume interrupted downloads at server start.
2020-04-06 17:22:47 -07:00
Miguel Portilla
cc452dfa9b Improve shard concurrency:
* Reduce lock scope on all public functions
* Use TaskQueue to process shard finalization in separate thread
* Store shard last ledger hash and other info in backend
* Use temp SQLite DB versus control file when acquiring
* Remove boost serialization from cmake files
2020-04-06 17:22:23 -07:00
Mark Travis
5047b89922 Allow non-root users to build packages. 2020-03-24 11:36:00 -07:00
Nik Bougalis
16c659b22c Simplify HashPrefix by converting to an enum 2020-03-05 14:10:20 -05:00
Mo Morsi
ec137044a0 Protocol Amendment: Always Require Fully-Canonical Signatures 2020-03-05 14:10:06 -05:00
John Freeman
20d4a39b63 Cooperate with CMAKE_MODULE_PATH set by Conan 2020-02-25 19:21:55 -08:00
CJ Cobb
e7ce3909d2 gRPC support for account_tx and tx
- Add support for all transaction types and ledger object types to gRPC
  implementation of tx and account_tx.

- Create common handlers for tx and account_tx.

- Remove mutex and abort() from gRPC server. JobQueue is stopped before
  gRPC server, with all coroutines executed to completion, so no need for
  synchronization.
2020-02-25 19:21:55 -08:00
Mike Ellery
acf4b78892 Add getRippledInfo to packages:
* include info gathering script in rippled linux pkgs
* modify script to output results to single markdown file
* rename info script
2020-02-25 19:21:55 -08:00
Mike Ellery
e9b3c58043 Use native cmake unity support:
* Remove hand-rolled unity sources
* unity unavailable for cmake 3.15 and earlier
* remove deprecated target configure option
2020-02-25 19:21:50 -08:00
Mike Ellery
4315913a5d Update RocksDB to 6.5:
* update EP and find package requirements
* minor protobuf/libarchive build fixes
* change travis release builds to nounity to
  ameliorate vm memory exhaustion.

FIXES: #3223, #3232
2020-02-12 10:19:23 -08:00
Nik Bougalis
172ead8221 Remove unused 'old Beast' code (RIPD-1652) 2020-02-11 19:14:24 -08:00
Mo Morsi
3578acaf0b Add new validator_info and manifest rpc methods
Returns local validator details and specified manifest information
respectively. Folded and rebased on latest develop
2020-02-10 23:36:31 -08:00
seelabs
81326a6d08 Remove old payment code 2020-01-30 15:38:14 -08:00
seelabs
9d3626fec5 Fix bug in qualityUpperBound:
* In and Out parameters were swapped when calculating the rate
* In and out qualities were not calculated correctly; use existing functions
  to get the qualities
* Added tests to check that theoretical quality matches actual computed quality
* Remove in/out parameter from qualityUpperBound
* Rename an overload of qualityUpperBound to adjustQualityWithFees
* Add fix amendment
2020-01-30 13:35:52 -08:00
Mike Ellery
098086592b Resolve minor build issues:
- allow SOCI EP to build with vcpkg
 - fix container tagging in pkg build
 - correct libarchive name for win
2020-01-30 13:18:18 -08:00
Mike Ellery
eb016456a1 Streamline pkg and travis CI:
* use tagged containers for pkg build
* update build images
* continue to build container images in pipeline, but allow
  failure (non-block)
* limit travis macos cache
* add vs2019 windows to travis
* remove xcode 9 travis build
* remove clang5/6 from CI and update min version of Clang required in
  cmake
* break windows CI build into stages to reduce timeouts
* update datelib
* add if condition to travis builds to allow commit message to limit
  builds by platform
2020-01-12 07:26:19 -08:00
CJ Cobb
7d867b806d Add gRPC support (#3127):
* add support for AccountInfo, Fee and Submit RPCs

* add partial support for Tx RPC (only supports Payments)
2020-01-10 12:31:24 -08:00
Edward Hennis
e3b5b808c5 Add units to all fee calculations:
* Uses existing XRPAmount with units for drops, and a new TaggedFee for
  fee units (LoadFeeTrack), and fee levels (TxQ).
* Resolves #2451
2020-01-08 18:44:01 -05:00