Commit Graph

13301 Commits

Author SHA1 Message Date
Devon White
271e79095b Consolidate programmer documentation on sharding 2020-09-01 16:39:00 -07:00
Devon White
6c268a3e9c Allow multiple paths for shard storage:
* Distinguish between recent and historical shards
* Allow multiple storage paths for historical shards
* Add documentation for this feature
* Add unit tests
2020-09-01 16:39:00 -07:00
Elliot Lee
e5ff70f606 Add pull request template
Based on the PR template used by xpring-eng:

https://raw.githubusercontent.com/xpring-eng/.github/master/.github/pull_request_template.md
2020-09-01 16:39:00 -07:00
Gregory Tsipenyuk
801b1580f5 Improve handling of RPC ledger_index argument:
Some RPC commands return `ledger_index` as a quoted numeric
string. This change allows the returned value to be directly
copied and used for follow-on RPC commands.

This commit fixes #3533
2020-09-01 16:39:00 -07:00
Gregory Tsipenyuk
707868be33 Implement sticky DNS resolution for validator list retrieval:
When attempting to load a validator list from a configured
site, attempt to reuse the last IP that was successfully
used if that IP is still present in the DNS response.

Otherwise, randomly select an IP address from the list of
IPs provided by the DNS system.

This commit fixes #3494.
2020-09-01 10:37:19 -07:00
Gregory Tsipenyuk
9b9f34f881 Optimize relaying of validation and proposal messages:
With few exceptions, servers will typically receive multiple copies
of any given message from its directly connected peers. For servers
with several peers this can impact the processing latency and force
it to do redundant work. Proposal and validation messages are often
relayed with extremely high redundancy.

This commit, if merged, introduces experimental code that attempts
to optimize the relaying of proposals and validations by allowing
servers to instruct their peers to "squelch" delivery of selected
proposals and validations. Servers making squelching decisions by
a process that evaluates the fitness and performance of a given
server and randomly selecting a subset of the best candidates.

The experimental code is presently disabled and must be explicitly
enabled by server operators that wish to test it.
2020-09-01 09:07:32 -07:00
Scott Schurr
7724cca384 Implement enhanced Ticket support:
Tickets are a mechanism to allow for the "out-of-order" execution of
transactions on the XRP Ledger.

This commit, if merged, reworks the existing support for tickets and
introduces support for 'ticket batching', completing the feature set
needed for tickets.

The code is gated under the newly-introduced `TicketBatch` amendment
and the `Tickets` amendment, which is not presently active on the
network, is being removed.

The specification for this change can be found at:
https://github.com/xrp-community/standards-drafts/issues/16
2020-09-01 08:58:57 -07:00
manojsdoshi
01bd5a2646 Set version to 1.6.0 1.6.0 2020-08-18 15:33:02 -07:00
manojsdoshi
2b44868373 Set version to 1.6.0-rc3 2020-08-07 13:45:44 -07:00
seelabs
f79a4a8cdb Revert "Remove CryptoConditionsSuite stub amendment:"
This reverts commit d3798f6290.
2020-08-07 13:45:22 -07:00
seelabs
7bb6b75f3c Use the correct root hash for the tx tree 2020-08-07 11:17:48 -07:00
Nik Bougalis
e5d17a9452 Set version to 1.6.0-rc2 2020-08-06 10:14:03 -07:00
manojsdoshi
dbd5f0073e Revert support for deterministic shards:
Commit 4dc08f8202 introduced support for
deterministic shards, which makes the sharding functionality provided
by rippled more useful.

After merging, several opportunities for further improvements to the
deterministic sharding implementation were identified and a significant
increase int memory usage during shard finalization was detected.

Because of these issues, the commit is being reverted and the feature is
being rolled back. It will be reintroduced in a future release.
2020-08-06 10:11:55 -07:00
Peng Wang
12c0e8148b Improve naming of fields associated with the NegativeUNL 2020-08-06 10:05:43 -07:00
Edward Hennis
72a9a2bdbb Reorder the Travis build:
* Builds Windows dependencies first.
* Builds ALL OSs in the last stage.
* Fix the MacOS builds.
* Windows dependency stages are allowed to fail so ALL configurations will
  attempt to build. Windows builds will probably fail if dependencies fail
  (caching may allow them to succeed), but they will at least be attempted.
* Remove broken AppVeyor config file, so it stops trying.
2020-08-06 10:05:43 -07:00
seelabs
80860fa8f5 Add preliminary support for Boost 1.74 2020-08-06 10:05:43 -07:00
seelabs
8cf542abb0 Fix memory management issues with checkpointers:
The checkpointer class had assumed that the database would exist for the
lifetime of the application. This is no long true. These changes resolve bugs
involving dangling pointers.
2020-08-06 10:05:43 -07:00
seelabs
a931b020b5 Switch to updated date library exception handling:
Without defining `HAS_UNCAUGHT_EXCEPTIONS` the date library would use the
deprecated `uncaught_exception` which would generate a warning.
2020-08-06 09:56:04 -07:00
seelabs
d317060ae4 Fix a race condition with TrustedPublisherServer:
There was a race condition in `on_accept` where the object's destructor
could run while `on_accept` was called.

This patch ensures that if `on_accept` is called then the object remains
valid for the duration of the call.
2020-08-06 09:56:04 -07:00
seelabs
eee07a4f96 Add missing cstdint include 2020-08-06 09:55:37 -07:00
manojsdoshi
7b048b423e Set version to 1.6.0-rc1 2020-07-08 13:40:03 -07:00
manojsdoshi
a459206848 Set version to 1.6.0-b9 2020-07-08 13:38:27 -07:00
Howard Hinnant
3a3b0b4c14 Modify health check API
* Fixes #3486
* load factor computation normalized by load_base.
* last validated ledger age set to -1 while syncing.
* Return status changed:
*    healthy  -> ok
*    warning  -> service_unavailable
*    critical -> internal_server_error
2020-07-08 13:35:07 -07:00
manojsdoshi
de0c527387 Set version to 1.6.0-b8 2020-06-30 12:40:51 -07:00
manojsdoshi
b54453d1a7 Fix a build issue caused by pip3 being unavailable 2020-06-30 09:15:37 -07:00
Peng Wang
706ca874b0 Implement negative UNL functionality:
This change can help improve the liveness of the network during periods of network
instability, by allowing the network to track which validators are presently not online
and to disregard them for the purposes of quorum calculations.
2020-06-30 09:15:37 -07:00
Peng Wang
51bd4626b1 Implement version upgrade warning:
If the 'HardenedValidations' amendment is enabled, this commit will
track the version of the software that validators embed in their
validations.

If a server notices that at least 60% of the validators on its UNL
are running a newer version than it is running, it will periodically
print an informational message, reminding the operator to check for
update.
2020-06-30 09:15:37 -07:00
mDuo13
cf6f40ea8f Deprecate unused/obsolete error codes:
The tecUNFUNDED code is actively used when attempting to create payment
channels; the messages incorrectly list it as deprecated.

Meanwhile, the tecUNFUNDED_ADD code actually is an unused legacy code,
dating back to when there was a WalletAdd transactor. The terLAST and
terFUNDS_SPENT codes are also unused legacy codes.

Engine result messages are not part of the binary format and are
documented as subject to change without notice, so this should not
require an amendment nor a new API version.

Align error code table for human readability.
2020-06-30 08:52:18 -07:00
seelabs
d3798f6290 Remove CryptoConditionsSuite stub amendment:
The amendment was partially complete, included no functional code
and, even if activated, it would result in no changes to transaction
proessing. Despite this, removing the amendment is the prudent course
of action and avoids the possibility of an accidental activation.

If additional cryptoconditions are implemented, they will be each
assigned a new, unique amendment code.
2020-06-30 08:52:18 -07:00
Rome Reginelli
4e33a1abf7 crawl_shards comment fix
Corrects the public_key parameter name in the comment.
See https://github.com/ripple/xrpl-dev-portal/pull/854 for context.
2020-06-30 08:52:18 -07:00
Miguel Portilla
86e8f2e232 Add Shard Family 2020-06-30 08:52:18 -07:00
Miguel Portilla
91e857874f Improve LedgerMaster shard acquisition 2020-06-30 08:52:17 -07:00
Miguel Portilla
8f50fd051e Use NuDB version 2.0.3 2020-06-30 08:52:17 -07:00
p2peer
94e8e94750 Add support for deterministic database shards (#2688):
This commit, if merged, adds support to allow multiple indepedent nodes to
produce a binary identical shard for a given range of ledgers. The advantage
is that servers can use content-addressable storage, and can more efficiently
retrieve shards by downloading from multiple peers at once and then verifying
the integrity of a shard by cross-checking its checksum with the checksum
other servers report.
2020-06-30 08:52:17 -07:00
Edward Hennis
54ece72b62 Update cmake so that rippled can build as a submodule 2020-06-25 19:47:20 -07:00
Edward Hennis
4702c8b591 Improve online_delete configuration and DB tuning:
* Document delete_batch, back_off_milliseconds, age_threshold_seconds.
* Convert those time values to chrono types.
* Fix bug that ignored age_threshold_seconds.
* Add a "recovery buffer" to the config that gives the node a chance to
  recover before aborting online delete.
* Add begin/end log messages around the SQL queries.
* Add a new configuration section: [sqlite] to allow tuning the sqlite
  database operations. Ignored on full/large history servers.
* Update documentation of [node_db] and [sqlite] in the
  rippled-example.cfg file.

Resolves #3321
2020-06-25 19:46:43 -07:00
Mark Travis
00702f28c2 Improve reporting of ledger age in server_info 2020-06-25 19:46:43 -07:00
Gregory Tsipenyuk
df29e98ea5 Improve amendment processing and activation logic:
* The amendment ballot counting code contained a minor technical
  flaw, caused by the use of integer arithmetic and rounding
  semantics, that could allow amendments to reach majority with
  slightly less than 80% support. This commit introduces an
  amendment which, if enabled, will ensure that activation
  requires at least 80% support.
* This commit also introduces a configuration option to adjust
  the amendment activation hysteresis. This option is useful on
  test networks, but should not be used on the main network as
  is a network-wide consensus parameter that should not be
  changed on a per-server basis; doing so can result in a
  hard-fork.

Fixes #3396
2020-06-25 19:46:43 -07:00
Nik Bougalis
fe9922d654 Improve compression support:
* Optimize parsing of compressed message headers
* Enforce protocol-defined message size maxima
* Update comments
2020-06-25 19:46:43 -07:00
Nik Bougalis
362a017eee Cleanup SHAMap and simplify interfaces:
* Improve error reporting (more readable exception messages)
* Reduce function complexity (split oversized function to smaller pieces)
* Reduce code duplication
* Reduce buffer copying
2020-06-25 19:46:43 -07:00
Nik Bougalis
e8f3525226 Improve Slice:
Slice should, eventually, be replaced by std::string_view<std::uint8_t>
so begin adding some helpful functions to align its interface.
2020-06-25 19:46:43 -07:00
Howard Hinnant
1067086f71 Consolidate "Not Synced" error messages:
Work on a version 2 of the XRP Network API has begun. The new
API returns:

* `notSynced` in place of `noClosed`, `noCurrent`, and `noNetwork`;
* `invalidParams` in place of `lgrIdxInvalid`.

The new version 2 API cannot be selected yet, as it remains a work
in progress.

Fixes #3269
2020-06-25 19:45:59 -07:00
Howard Hinnant
0214d83aa5 Improve handling of empty buffer in varint parsing (RIPD-1683) 2020-06-25 19:45:59 -07:00
CJ Cobb
d69a902876 Add comments to protobuf files 2020-06-25 19:09:19 -07:00
manojsdoshi
f43aeda49c Set version to 1.6.0-b7 2020-05-29 17:55:01 -07:00
Yusuf Sahin HAMZA
27484f78a9 Use libarchive version 3.4.3 2020-05-29 17:54:19 -07:00
p2peer
93bf77bdec Unit tests for database shards 2020-05-29 17:54:19 -07:00
Miguel Portilla
728651b5d5 Use LZ4_decompress_safe 2020-05-29 17:54:19 -07:00
Miguel Portilla
fb74eefa9e Update LZ4 library to 1.9.2 2020-05-29 17:54:19 -07:00
manojsdoshi
853c964194 Fix a build issue caused by the latest Docker version:
* The latest docker version is not supported by
   artifactory causing the package build to fail.
   Setting the docker version to 19.03.8 to fix the build.
2020-05-29 17:54:19 -07:00