Commit Graph

1216 Commits

Author SHA1 Message Date
Miguel Portilla
86e8f2e232 Add Shard Family 2020-06-30 08:52:18 -07:00
Miguel Portilla
8f50fd051e Use NuDB version 2.0.3 2020-06-30 08:52:17 -07:00
p2peer
94e8e94750 Add support for deterministic database shards (#2688):
This commit, if merged, adds support to allow multiple indepedent nodes to
produce a binary identical shard for a given range of ledgers. The advantage
is that servers can use content-addressable storage, and can more efficiently
retrieve shards by downloading from multiple peers at once and then verifying
the integrity of a shard by cross-checking its checksum with the checksum
other servers report.
2020-06-30 08:52:17 -07:00
Edward Hennis
54ece72b62 Update cmake so that rippled can build as a submodule 2020-06-25 19:47:20 -07:00
Yusuf Sahin HAMZA
27484f78a9 Use libarchive version 3.4.3 2020-05-29 17:54:19 -07:00
p2peer
93bf77bdec Unit tests for database shards 2020-05-29 17:54:19 -07:00
Miguel Portilla
fb74eefa9e Update LZ4 library to 1.9.2 2020-05-29 17:54:19 -07:00
manojsdoshi
853c964194 Fix a build issue caused by the latest Docker version:
* The latest docker version is not supported by
   artifactory causing the package build to fail.
   Setting the docker version to 19.03.8 to fix the build.
2020-05-29 17:54:19 -07:00
John Freeman
0b9e935806 Find date::date in CMake package configuration file 2020-05-26 18:41:30 -07:00
John Freeman
8f984042f4 Try to fix usage of pkg-config for static linking 2020-05-26 18:37:52 -07:00
Devon White
ac766ec0eb Introduce ShardArchiveHandler improvements:
* Improve documentation
* Make the ShardArchiveHandler rather than the DatabaseShardImp perform
  LastLedgerHash verification for downloaded shards
* Remove ShardArchiveHandler's singleton implementation and make it an
  Application member
* Have the Application invoke ShardArchiveHandler initialization
  instead of clients
* Add RecoveryHandler as a ShardArchiveHandler derived class
* Improve commenting
2020-05-26 18:37:01 -07:00
Devon White
21340a1c1e Validate LastLedgerHash for downloaded shards:
* Add documentation for shard validation
* Retrieve last ledger hash for imported shards
* Verify the last ledger hash in Shard::finalize
* Limit last ledger hash retrieval attempts for imported shards
* Use a common function for removing failed shards
* Add new ShardInfo::State for imported shards
2020-05-26 18:36:54 -07:00
Nik Bougalis
62a3f33d72 Remove the built-in "sustain" watchdog:
The built-in watchdog is simplistic and can, sometimes, cause problems
especially on systems that have the ability to automatically start and
monitor processes.

This commit removes the sustain system entirely, changes the handling
of the SIGTERM signal to properly terminate the process and improves
the error message reported to the user when the command line used to
start `rippled` is incorrect and malformed.
2020-05-05 16:05:23 -07:00
Nik Bougalis
dbee3f01b7 Clean up and modernize code:
This commit removes obsolete comments, dead or no longer useful
code, and workarounds for several issues that were present in older
compilers that we no longer support.

Specifically:

- It improves the transaction metadata handling class, simplifying
  its use and making it less error-prone.
- It reduces the footprint of the Serializer class by consolidating
  code and leveraging templates.
- It cleanups the ST* class hierarchy, removing dead code, improving
  and consolidating code to reduce complexity and code duplication.
- It shores up the handling of currency codes and the conversation
  between 160-bit currency codes and their string representation.
- It migrates beast::secure_erase to the ripple namespace and uses
  a call to OpenSSL_cleanse instead of the custom implementation.
2020-05-05 16:05:22 -07:00
manojsdoshi
444ea56182 Adding support to build rippled packages for ubuntu 20.04 2020-04-27 16:16:56 -07:00
Miguel Portilla
bd8dbb87b6 Update to RocksBD 6.7.3 2020-04-23 18:09:45 -07:00
manojsdoshi
123e94c603 Fix a build issue involving Ubuntu Docker containers:
Upgrade the build pipeline to use ubuntu 19.10 container
in order for the build pipeline to successfully complete builds.
2020-04-23 14:27:12 -07:00
Nik Bougalis
bdd22e4d51 Improve reporting of missing node exceptions 2020-04-14 19:42:50 -07:00
manojsdoshi
ce6b427202 Fix a build issue involving Ubuntu Docker containers:
The package 'apt-transport-https' needs to be downloaded and installed
in order for the build pipeline to successfully complete builds.
2020-04-14 19:42:42 -07:00
John Freeman
858e93c7f8 Improve the Doxygen Workflow
* Use official build environment Docker image
* Update documentation and Doxygen website URLs
2020-04-14 19:42:42 -07:00
Gábor Lipták
1c3c69f8b5 Update Travis to bionic 2020-04-14 19:17:13 -07:00
manojsdoshi
393ca87572 Change the location for the project signer public keys:
The automated build system only builds packages signed with a list of
approved keys. This is a security measure to prevent someone who gains
push access to the repository from producing potentially malicious
packages that are signed by Ripple's trusted private keys.

Moving this list to the new location makes it easy to add and delete
new keys to the list.
2020-04-07 16:25:19 -07:00
John Freeman
25b13978e7 Fix unity build 2020-04-07 11:31:23 -07:00
John Freeman
3e9cff9287 Fix Doxygen build 2020-04-06 17:28:53 -07:00
Gregory Tsipenyuk
758a3792eb Add protocol message compression support:
* Peers negotiate compression via HTTP Header "X-Offer-Compression: lz4"
* Messages greater than 70 bytes and protocol type messages MANIFESTS,
  ENDPOINTS, TRANSACTION, GET_LEDGER, LEDGER_DATA, GET_OBJECT,
  and VALIDATORLIST are compressed
* If the compressed message is larger than the uncompressed message
  then the uncompressed message is sent
* Compression flag and the compression algorithm type are included
  in the message header
* Only LZ4 block compression is currently supported
2020-04-06 17:22:59 -07:00
Devon White
905a97e0aa Make ShardArchiveHandler downloads more resilient:
* Make ShardArchiveHandler a singleton.
* Add state database for ShardArchiveHandler.
* Use temporary database for SSLHTTPDownloader downloads.
* Make ShardArchiveHandler a Stoppable class.
* Automatically resume interrupted downloads at server start.
2020-04-06 17:22:47 -07:00
Miguel Portilla
cc452dfa9b Improve shard concurrency:
* Reduce lock scope on all public functions
* Use TaskQueue to process shard finalization in separate thread
* Store shard last ledger hash and other info in backend
* Use temp SQLite DB versus control file when acquiring
* Remove boost serialization from cmake files
2020-04-06 17:22:23 -07:00
Mark Travis
5047b89922 Allow non-root users to build packages. 2020-03-24 11:36:00 -07:00
Nik Bougalis
16c659b22c Simplify HashPrefix by converting to an enum 2020-03-05 14:10:20 -05:00
Mo Morsi
ec137044a0 Protocol Amendment: Always Require Fully-Canonical Signatures 2020-03-05 14:10:06 -05:00
John Freeman
20d4a39b63 Cooperate with CMAKE_MODULE_PATH set by Conan 2020-02-25 19:21:55 -08:00
CJ Cobb
e7ce3909d2 gRPC support for account_tx and tx
- Add support for all transaction types and ledger object types to gRPC
  implementation of tx and account_tx.

- Create common handlers for tx and account_tx.

- Remove mutex and abort() from gRPC server. JobQueue is stopped before
  gRPC server, with all coroutines executed to completion, so no need for
  synchronization.
2020-02-25 19:21:55 -08:00
Mike Ellery
acf4b78892 Add getRippledInfo to packages:
* include info gathering script in rippled linux pkgs
* modify script to output results to single markdown file
* rename info script
2020-02-25 19:21:55 -08:00
Mike Ellery
e9b3c58043 Use native cmake unity support:
* Remove hand-rolled unity sources
* unity unavailable for cmake 3.15 and earlier
* remove deprecated target configure option
2020-02-25 19:21:50 -08:00
Edward Hennis
3753840de5 Handle build directories for specific versions of gcc/clang:
* Example: gcc.Debug will use the the default version of gcc installed on the
  system. gcc-9.Debug will use version 9, regardless of the default. This will
  be most useful when the default is older than required or desired.
2020-02-12 10:19:23 -08:00
Mike Ellery
4315913a5d Update RocksDB to 6.5:
* update EP and find package requirements
* minor protobuf/libarchive build fixes
* change travis release builds to nounity to
  ameliorate vm memory exhaustion.

FIXES: #3223, #3232
2020-02-12 10:19:23 -08:00
Nik Bougalis
172ead8221 Remove unused 'old Beast' code (RIPD-1652) 2020-02-11 19:14:24 -08:00
Mo Morsi
3578acaf0b Add new validator_info and manifest rpc methods
Returns local validator details and specified manifest information
respectively. Folded and rebased on latest develop
2020-02-10 23:36:31 -08:00
seelabs
81326a6d08 Remove old payment code 2020-01-30 15:38:14 -08:00
seelabs
9d3626fec5 Fix bug in qualityUpperBound:
* In and Out parameters were swapped when calculating the rate
* In and out qualities were not calculated correctly; use existing functions
  to get the qualities
* Added tests to check that theoretical quality matches actual computed quality
* Remove in/out parameter from qualityUpperBound
* Rename an overload of qualityUpperBound to adjustQualityWithFees
* Add fix amendment
2020-01-30 13:35:52 -08:00
Mike Ellery
098086592b Resolve minor build issues:
- allow SOCI EP to build with vcpkg
 - fix container tagging in pkg build
 - correct libarchive name for win
2020-01-30 13:18:18 -08:00
Mike Ellery
eb016456a1 Streamline pkg and travis CI:
* use tagged containers for pkg build
* update build images
* continue to build container images in pipeline, but allow
  failure (non-block)
* limit travis macos cache
* add vs2019 windows to travis
* remove xcode 9 travis build
* remove clang5/6 from CI and update min version of Clang required in
  cmake
* break windows CI build into stages to reduce timeouts
* update datelib
* add if condition to travis builds to allow commit message to limit
  builds by platform
2020-01-12 07:26:19 -08:00
CJ Cobb
7d867b806d Add gRPC support (#3127):
* add support for AccountInfo, Fee and Submit RPCs

* add partial support for Tx RPC (only supports Payments)
2020-01-10 12:31:24 -08:00
Edward Hennis
e3b5b808c5 Add units to all fee calculations:
* Uses existing XRPAmount with units for drops, and a new TaggedFee for
  fee units (LoadFeeTrack), and fee levels (TxQ).
* Resolves #2451
2020-01-08 18:44:01 -05:00
Peng Wang
2aa11fa41d Support API versioning 2020-01-01 18:12:55 -08:00
Nik Bougalis
63503ee8f0 Improve platform detection and reduce includes:
The existing platform detection code was derived from the old Beast
library, which was, itself, derived from JUCE.

This commit removes that code and replaces it with the Boost.Predef
library which defines a consistent set of compiler, architecture,
operating system, library, and other version numbers.

For more on Boost.Predef, please see the Boost documentation. The
documentation for the current version as of this writing is at:
https://www.boost.org/doc/libs/1_71_0/doc/html/predef.html
2019-12-30 20:20:31 -08:00
Nik Bougalis
f6916bfd42 Improve protocol-level handshaking protocol:
This commit restructures the HTTP based protocol negotiation that `rippled`
executes and introduces support for negotiation of compression for peer
links which, if implemented, should result in significant bandwidth savings
for some server roles.

This commit also introduces the new `[network_id]` configuration option
that administrators can use to specify which network the server is part of
and intends to join. This makes it possible for servers from different
networks to drop the link early.

The changeset also improves the log messages generated when negotiation
of a peer link upgrade fails. In the past, no useful information would
be logged, making it more difficult for admins to troubleshoot errors.

This commit also fixes RIPD-237 and RIPD-451
2019-11-28 09:46:17 -08:00
John Northrup
fb0d065723 Move signing of deb to use Ubuntu 18.04 LTS 2019-11-27 17:28:15 -08:00
Devon White
47501b7f99 Provide additional info with txnNotFound errors.
* The `tx` command now supports min_ledger and max_ledger fields.
* If the requested transaction isn't found and these fields are
  provided, the error response indicates whether or not every
  ledger in the the provided range was searched.

This fixes #2924
2019-11-27 17:28:15 -08:00
Mo Morsi
6cda070fe0 Search for soci before vendoring 2019-11-27 16:58:56 -08:00