Commit Graph

254 Commits

Author SHA1 Message Date
Miguel Portilla
86e8f2e232 Add Shard Family 2020-06-30 08:52:18 -07:00
Miguel Portilla
8f50fd051e Use NuDB version 2.0.3 2020-06-30 08:52:17 -07:00
p2peer
94e8e94750 Add support for deterministic database shards (#2688):
This commit, if merged, adds support to allow multiple indepedent nodes to
produce a binary identical shard for a given range of ledgers. The advantage
is that servers can use content-addressable storage, and can more efficiently
retrieve shards by downloading from multiple peers at once and then verifying
the integrity of a shard by cross-checking its checksum with the checksum
other servers report.
2020-06-30 08:52:17 -07:00
Edward Hennis
54ece72b62 Update cmake so that rippled can build as a submodule 2020-06-25 19:47:20 -07:00
Yusuf Sahin HAMZA
27484f78a9 Use libarchive version 3.4.3 2020-05-29 17:54:19 -07:00
p2peer
93bf77bdec Unit tests for database shards 2020-05-29 17:54:19 -07:00
Miguel Portilla
fb74eefa9e Update LZ4 library to 1.9.2 2020-05-29 17:54:19 -07:00
John Freeman
0b9e935806 Find date::date in CMake package configuration file 2020-05-26 18:41:30 -07:00
John Freeman
8f984042f4 Try to fix usage of pkg-config for static linking 2020-05-26 18:37:52 -07:00
Devon White
ac766ec0eb Introduce ShardArchiveHandler improvements:
* Improve documentation
* Make the ShardArchiveHandler rather than the DatabaseShardImp perform
  LastLedgerHash verification for downloaded shards
* Remove ShardArchiveHandler's singleton implementation and make it an
  Application member
* Have the Application invoke ShardArchiveHandler initialization
  instead of clients
* Add RecoveryHandler as a ShardArchiveHandler derived class
* Improve commenting
2020-05-26 18:37:01 -07:00
Devon White
21340a1c1e Validate LastLedgerHash for downloaded shards:
* Add documentation for shard validation
* Retrieve last ledger hash for imported shards
* Verify the last ledger hash in Shard::finalize
* Limit last ledger hash retrieval attempts for imported shards
* Use a common function for removing failed shards
* Add new ShardInfo::State for imported shards
2020-05-26 18:36:54 -07:00
Nik Bougalis
62a3f33d72 Remove the built-in "sustain" watchdog:
The built-in watchdog is simplistic and can, sometimes, cause problems
especially on systems that have the ability to automatically start and
monitor processes.

This commit removes the sustain system entirely, changes the handling
of the SIGTERM signal to properly terminate the process and improves
the error message reported to the user when the command line used to
start `rippled` is incorrect and malformed.
2020-05-05 16:05:23 -07:00
Nik Bougalis
dbee3f01b7 Clean up and modernize code:
This commit removes obsolete comments, dead or no longer useful
code, and workarounds for several issues that were present in older
compilers that we no longer support.

Specifically:

- It improves the transaction metadata handling class, simplifying
  its use and making it less error-prone.
- It reduces the footprint of the Serializer class by consolidating
  code and leveraging templates.
- It cleanups the ST* class hierarchy, removing dead code, improving
  and consolidating code to reduce complexity and code duplication.
- It shores up the handling of currency codes and the conversation
  between 160-bit currency codes and their string representation.
- It migrates beast::secure_erase to the ripple namespace and uses
  a call to OpenSSL_cleanse instead of the custom implementation.
2020-05-05 16:05:22 -07:00
Miguel Portilla
bd8dbb87b6 Update to RocksBD 6.7.3 2020-04-23 18:09:45 -07:00
Nik Bougalis
bdd22e4d51 Improve reporting of missing node exceptions 2020-04-14 19:42:50 -07:00
John Freeman
858e93c7f8 Improve the Doxygen Workflow
* Use official build environment Docker image
* Update documentation and Doxygen website URLs
2020-04-14 19:42:42 -07:00
John Freeman
25b13978e7 Fix unity build 2020-04-07 11:31:23 -07:00
John Freeman
3e9cff9287 Fix Doxygen build 2020-04-06 17:28:53 -07:00
Gregory Tsipenyuk
758a3792eb Add protocol message compression support:
* Peers negotiate compression via HTTP Header "X-Offer-Compression: lz4"
* Messages greater than 70 bytes and protocol type messages MANIFESTS,
  ENDPOINTS, TRANSACTION, GET_LEDGER, LEDGER_DATA, GET_OBJECT,
  and VALIDATORLIST are compressed
* If the compressed message is larger than the uncompressed message
  then the uncompressed message is sent
* Compression flag and the compression algorithm type are included
  in the message header
* Only LZ4 block compression is currently supported
2020-04-06 17:22:59 -07:00
Devon White
905a97e0aa Make ShardArchiveHandler downloads more resilient:
* Make ShardArchiveHandler a singleton.
* Add state database for ShardArchiveHandler.
* Use temporary database for SSLHTTPDownloader downloads.
* Make ShardArchiveHandler a Stoppable class.
* Automatically resume interrupted downloads at server start.
2020-04-06 17:22:47 -07:00
Miguel Portilla
cc452dfa9b Improve shard concurrency:
* Reduce lock scope on all public functions
* Use TaskQueue to process shard finalization in separate thread
* Store shard last ledger hash and other info in backend
* Use temp SQLite DB versus control file when acquiring
* Remove boost serialization from cmake files
2020-04-06 17:22:23 -07:00
Mark Travis
5047b89922 Allow non-root users to build packages. 2020-03-24 11:36:00 -07:00
Nik Bougalis
16c659b22c Simplify HashPrefix by converting to an enum 2020-03-05 14:10:20 -05:00
Mo Morsi
ec137044a0 Protocol Amendment: Always Require Fully-Canonical Signatures 2020-03-05 14:10:06 -05:00
John Freeman
20d4a39b63 Cooperate with CMAKE_MODULE_PATH set by Conan 2020-02-25 19:21:55 -08:00
CJ Cobb
e7ce3909d2 gRPC support for account_tx and tx
- Add support for all transaction types and ledger object types to gRPC
  implementation of tx and account_tx.

- Create common handlers for tx and account_tx.

- Remove mutex and abort() from gRPC server. JobQueue is stopped before
  gRPC server, with all coroutines executed to completion, so no need for
  synchronization.
2020-02-25 19:21:55 -08:00
Mike Ellery
acf4b78892 Add getRippledInfo to packages:
* include info gathering script in rippled linux pkgs
* modify script to output results to single markdown file
* rename info script
2020-02-25 19:21:55 -08:00
Mike Ellery
e9b3c58043 Use native cmake unity support:
* Remove hand-rolled unity sources
* unity unavailable for cmake 3.15 and earlier
* remove deprecated target configure option
2020-02-25 19:21:50 -08:00
Mike Ellery
4315913a5d Update RocksDB to 6.5:
* update EP and find package requirements
* minor protobuf/libarchive build fixes
* change travis release builds to nounity to
  ameliorate vm memory exhaustion.

FIXES: #3223, #3232
2020-02-12 10:19:23 -08:00
Nik Bougalis
172ead8221 Remove unused 'old Beast' code (RIPD-1652) 2020-02-11 19:14:24 -08:00
Mo Morsi
3578acaf0b Add new validator_info and manifest rpc methods
Returns local validator details and specified manifest information
respectively. Folded and rebased on latest develop
2020-02-10 23:36:31 -08:00
seelabs
81326a6d08 Remove old payment code 2020-01-30 15:38:14 -08:00
seelabs
9d3626fec5 Fix bug in qualityUpperBound:
* In and Out parameters were swapped when calculating the rate
* In and out qualities were not calculated correctly; use existing functions
  to get the qualities
* Added tests to check that theoretical quality matches actual computed quality
* Remove in/out parameter from qualityUpperBound
* Rename an overload of qualityUpperBound to adjustQualityWithFees
* Add fix amendment
2020-01-30 13:35:52 -08:00
Mike Ellery
098086592b Resolve minor build issues:
- allow SOCI EP to build with vcpkg
 - fix container tagging in pkg build
 - correct libarchive name for win
2020-01-30 13:18:18 -08:00
Mike Ellery
eb016456a1 Streamline pkg and travis CI:
* use tagged containers for pkg build
* update build images
* continue to build container images in pipeline, but allow
  failure (non-block)
* limit travis macos cache
* add vs2019 windows to travis
* remove xcode 9 travis build
* remove clang5/6 from CI and update min version of Clang required in
  cmake
* break windows CI build into stages to reduce timeouts
* update datelib
* add if condition to travis builds to allow commit message to limit
  builds by platform
2020-01-12 07:26:19 -08:00
CJ Cobb
7d867b806d Add gRPC support (#3127):
* add support for AccountInfo, Fee and Submit RPCs

* add partial support for Tx RPC (only supports Payments)
2020-01-10 12:31:24 -08:00
Edward Hennis
e3b5b808c5 Add units to all fee calculations:
* Uses existing XRPAmount with units for drops, and a new TaggedFee for
  fee units (LoadFeeTrack), and fee levels (TxQ).
* Resolves #2451
2020-01-08 18:44:01 -05:00
Peng Wang
2aa11fa41d Support API versioning 2020-01-01 18:12:55 -08:00
Nik Bougalis
63503ee8f0 Improve platform detection and reduce includes:
The existing platform detection code was derived from the old Beast
library, which was, itself, derived from JUCE.

This commit removes that code and replaces it with the Boost.Predef
library which defines a consistent set of compiler, architecture,
operating system, library, and other version numbers.

For more on Boost.Predef, please see the Boost documentation. The
documentation for the current version as of this writing is at:
https://www.boost.org/doc/libs/1_71_0/doc/html/predef.html
2019-12-30 20:20:31 -08:00
Nik Bougalis
f6916bfd42 Improve protocol-level handshaking protocol:
This commit restructures the HTTP based protocol negotiation that `rippled`
executes and introduces support for negotiation of compression for peer
links which, if implemented, should result in significant bandwidth savings
for some server roles.

This commit also introduces the new `[network_id]` configuration option
that administrators can use to specify which network the server is part of
and intends to join. This makes it possible for servers from different
networks to drop the link early.

The changeset also improves the log messages generated when negotiation
of a peer link upgrade fails. In the past, no useful information would
be logged, making it more difficult for admins to troubleshoot errors.

This commit also fixes RIPD-237 and RIPD-451
2019-11-28 09:46:17 -08:00
Devon White
47501b7f99 Provide additional info with txnNotFound errors.
* The `tx` command now supports min_ledger and max_ledger fields.
* If the requested transaction isn't found and these fields are
  provided, the error response indicates whether or not every
  ledger in the the provided range was searched.

This fixes #2924
2019-11-27 17:28:15 -08:00
Mo Morsi
6cda070fe0 Search for soci before vendoring 2019-11-27 16:58:56 -08:00
Mo Morsi
9bd470f2c7 Search for sqlite3 before vendoring 2019-11-27 16:58:56 -08:00
Mo Morsi
530f1939d6 Search for system snappy before vendoring 2019-11-27 16:58:56 -08:00
Mo Morsi
e4ea3752ac Search for system secp256k1 before vendoring 2019-11-27 16:58:56 -08:00
Mo Morsi
b728bf0d09 Search for system libarchive before vendoring 2019-11-27 16:58:56 -08:00
Mo Morsi
194fb2b86d Use find_package to search for system lz4 before vendoring 2019-11-27 16:58:56 -08:00
Mike Ellery
ce5d901e6e Switch to official date lib repo for date.h 2019-11-27 16:58:56 -08:00
Mike Ellery
e720a66076 Ensure permissions for initial config 2019-10-30 12:23:58 -07:00
Nik Bougalis
a3a9dc26b4 Introduce support for deletable accounts:
The XRP Ledger utilizes an account model. Unlike systems based on a UTXO
model, XRP Ledger accounts are first-class objects. This design choice
allows the XRP Ledger to offer rich functionality, including the ability
to own objects (offers, escrows, checks, signer lists) as well as other
advanced features, such as key rotation and configurable multi-signing
without needing to change a destination address.

The trade-off is that accounts must be stored on ledger. The XRP Ledger
applies reserve requirements, in XRP, to protect the shared global ledger
from growing excessively large as the result of spam or malicious usage.

Prior to this commit, accounts had been permanent objects; once created,
they could never be deleted.

This commit introduces a new amendment "DeletableAccounts" which, if
enabled, will allow account objects to be deleted by executing the new
"AccountDelete" transaction. Any funds remaining in the account will
be transferred to an account specified in the deletion transaction.

The amendment changes the mechanics of account creation; previously
a new account would have an initial sequence number of 1. Accounts
created after the amendment will have an initial sequence number that
is equal to the ledger in which the account was created.

Accounts can only be deleted if they are not associated with any
obligations (like RippleStates, Escrows, or PayChannels) and if the
current ledger sequence number exceeds the account's sequence number
by at least 256 so that, if recreated, the account can be protected
from transaction replay.
2019-10-18 16:44:16 -07:00