* It is now an invariant that all constructed Public Keys are valid,
non-empty and contain 33 bytes of data.
* Additionally, the memory footprint of the PublicKey class is reduced.
The size_ data member is declared as static.
* Distinguish and identify the PublisherList retrieved from the local
config file, versus the ones obtained from other validators.
* Fixes#2942
* Add logging for Application.cpp sweep()
* Improve lifetime management of ledger objects (`SLE`s)
* Only store SLE digest in CachedView; get SLEs from CachedSLEs
* Also force release of last ledger used for path finding if there are
no path finding requests to process
* Count more ST objects (derive from `CountedObject`)
* Track CachedView stats in CountedObjects
* Rename the CachedView counters
* Fix the scope of the digest lookup lock
Before this patch, if you asked "is it caching?" It was always caching.
This reverts commit 002893f280.
There were two files with conflicts in the automated revert:
- src/ripple/rpc/impl/RPCHelpers.h and
- src/test/rpc/JSONRPC_test.cpp
Those files were manually resolved.
Amendment "flapping" (an amendment repeatedly gaining and losing
majority) usually occurs when an amendment is on the verge of gaining
majority, and a validator not in favor of the amendment goes offline or
loses sync. This fix makes two changes:
1. The number of validators in the UNL determines the threshold required
for an amendment to gain majority.
2. The AmendmentTable keeps a record of the most recent Amendment vote
received from each trusted validator (and, with `trustChanged`, stays
up-to-date when the set of trusted validators changes). If no
validation arrives from a given validator, then the AmendmentTable
assumes that the previously-received vote has not changed.
In other words, when missing an `STValidation` from a remote validator,
each server now uses the last vote seen. There is a 24 hour timeout for
recorded validator votes.
These changes do not require an amendment because they do not impact
transaction processing, but only the threshold at which each individual
validator decides to propose an EnableAmendment pseudo-transaction.
Fix#4350
A bridge connects two blockchains: a locking chain and an issuing
chain (also called a mainchain and a sidechain). Both are independent
ledgers, with their own validators and potentially their own custom
transactions. Importantly, there is a way to move assets from the
locking chain to the issuing chain and a way to return those assets from
the issuing chain back to the locking chain: the bridge. This key
operation is called a cross-chain transfer. A cross-chain transfer is
not a single transaction. It happens on two chains, requires multiple
transactions, and involves an additional server type called a "witness".
A bridge does not exchange assets between two ledgers. Instead, it locks
assets on one ledger (the "locking chain") and represents those assets
with wrapped assets on another chain (the "issuing chain"). A good model
to keep in mind is a box with an infinite supply of wrapped assets.
Putting an asset from the locking chain into the box will release a
wrapped asset onto the issuing chain. Putting a wrapped asset from the
issuing chain back into the box will release one of the existing locking
chain assets back onto the locking chain. There is no other way to get
assets into or out of the box. Note that there is no way for the box to
"run out of" wrapped assets - it has an infinite supply.
Co-authored-by: Gregory Popovitch <greg7mdp@gmail.com>
Add new transaction submission API field, "sync", which
determines behavior of the server while submitting transactions:
1) sync (default): Process transactions in a batch immediately,
and return only once the transaction has been processed.
2) async: Put transaction into the batch for the next processing
interval and return immediately.
3) wait: Put transaction into the batch for the next processing
interval and return only after it is processed.
Add new transaction submission API field, "sync", which
determines behavior of the server while submitting transactions:
1) sync (default): Process transactions in a batch immediately,
and return only once the transaction has been processed.
2) async: Put transaction into the batch for the next processing
interval and return immediately.
3) wait: Put transaction into the batch for the next processing
interval and return only after it is processed.
* Commits 0b812cd (#4427) and 11e914f (#4516) conflict. The first added
references to `ServerHandlerImp` in files outside of that class's
organizational unit (which is technically incorrect). The second
removed `ServerHandlerImp`, but was not up to date with develop. This
results in the build failing.
* Fixes the build by changing references to `ServerHandlerImp` to
the more correct `ServerHandler`.
Enhance the /crawl endpoint by publishing WebSocket/RPC ports in the
server_info response. The function processing requests to the /crawl
endpoint actually calls server_info internally, so this change enables a
server to advertise its WebSocket/RPC port(s) to peers via the /crawl
endpoint. `grpc` and `peer` ports are included as well.
The new `ports` array contains objects, each containing a `port` for the
listening port (number string), and an array `protocol` listing the
supported protocol(s).
This allows crawlers to build a richer topology without needing to
port-scan nodes. For non-admin users (including peers), the info about
*admin* ports is excluded.
Also increase test coverage for RPC ServerInfo.
Fix#2837.
* Create the FeeSettings object in genesis ledger.
* Initialize with default values from the config. Removes the need to
pass a Config down into the Ledger initialization functions, including
setup().
* Drop the undocumented fee config settings in favor of the [voting]
section.
* Fix#3734.
* If you previously used fee_account_reserve and/or fee_owner_reserve,
you should change to using the [voting] section instead. Example:
```
[voting]
account_reserve=10000000
owner_reserve=2000000
```
* Because old Mainnet ledgers (prior to 562177 - yes, I looked it up)
don't have FeeSettings, some of the other ctors will default them to
the config values before setup() tries to load the object.
* Update default Config fee values to match Mainnet.
* Fix unit tests:
* Updated fees: Some tests are converted to use computed values of fee
object, but the default Env config was also updated to fix the rest.
* Unit tests that check the structure of the ledger have updated
hashes and counts.
Caching the base58check encoded version of an `AccountID` has
performance advantages, because because of the computationally
heavy cost associated with the conversion, which requires the
application of SHA-256 twice.
This commit makes the cache significantly more efficient in terms
of memory used: it eliminates the map, using a vector with a size
that is determined by the configured size of the node, and a hash
function to directly map any given `AccountID` to a specific slot
in the cache; the eviction policy is simple: in case of collision
the existing entry is removed and replaced with the new data.
Previously, use of the cache was optional and required additional
effort by the programmer. Now the cache is automatic and does not
require any additional work or information.
The new cache also utilizes a 64-way spinlock, to help reduce any
contention that the pressure on the cache would impose.
Each node on the network is supposed to have a unique cryptographic
identity. Typically, this identity is generated randomly at startup
and stored for later reuse in the (poorly named) file `wallet.db`.
If the file is copied, it is possible for two nodes to share the
same node identity. This is generally not desirable and existing
servers will detect and reject connections to other servers that
have the same key.
This commit achives three things:
1. It improves the detection code to pinpoint instances where two
distinct servers with the same key connect with each other. In
that case, servers will log an appropriate error and shut down
pending intervention by the server's operator.
2. It makes it possible for server administrators to securely and
easily generate new cryptographic identities for servers using
the new `--newnodeid` command line arguments. When a server is
started using this command, it will generate and save a random
secure identity.
3. It makes it possible to configure the identity using a command
line option, which makes it possible to derive it from data or
parameters associated with the container or hardware where the
instance is running by passing the `--nodeid` option, followed
by a single argument identifying the infomation from which the
node's identity is derived. For example, the following command
will result in nodes with different hostnames having different
node identities: `rippled --nodeid $HOSTNAME`
The last option is particularly useful for automated cloud-based
deployments that minimize the need for storing state and provide
unique deployment identifiers.
**Important note for server operators:**
Depending on variables outside of the the control of this code,
such as operating system version or configuration, permissions,
and more, it may be possible for other users or programs to be
able to access the command line arguments of other processes
on the system.
If you are operating in a shared environment, you should avoid
using this option, preferring instead to use the `[node_seed]`
option in the configuration file, and use permissions to limit
exposure of the node seed.
A user who gains access to the value used to derive the node's
unique identity could impersonate that node.
The commit also updates the minimum supported server protocol
version to `XRPL/2.1`, which has been supported since version
1.5.0 and eliminates support for `XPRL/2.0`.
* Remove Application & Database dependency in PerfLog. Replace it with
a callback passed into the constructor.
* Fixes the circular dependency between ripple/nodestore and ripple/basics
This is a refactor aimed at cleaning up and simplifying the existing
job queue.
As of now, all jobs are cancelled at the same time and in the same
way, so this commit removes the per-job cancellation token. If the
need for such support is demonstrated, support can be re-added.
* Revise documentation for ClosureCounter and Workers.
* Simplify code, removing unnecessary function arguments and
deduplicating expressions
* Restructure job handlers to no longer need to pass a job's
handle to the job.
- Only duplicate records from archive to writable during online_delete.
- Log duration of nodestore reads.
- Include nodestore counters in perf_log output.
- Remove gratuitous nodestore activity counting.
- Report initial sync duration in server_info and perfLog.
- Report state_accounting in perfLog.
- Make state_accounting durations more accurate.
- Parallel ledger loader.
- Config parameter to load ledgers on start.
This commit implements partitioned unordered maps and makes it possible
to traverse such a map in parallel, allowing for more efficient use of
CPU resources.
The `CachedSLEs`, `TaggedCache`, and `KeyCache` classes make use of the
new functionality, which should improve performance.
The existing calculation would limit the maximum number of threads
that would be created by default to at most 6; this may have been
reasonable a few years ago, but given both the load on the network
as of today and the increase in the number of CPU cores, the value
should be revisited.
This commit, if merged, changes the default calculation for nodes
that are configured as `large` or `huge` to allow for up to twelve
threads.
The "sweep interval" is the amount of time between successive sweeps of
of various in-memory data structures to remove stale items.
Prior to this commit, the interval was automatically adjusted, based on
the value of the `[node_size]` option in a server's configuration file.
If merged, this commit introduces a new configuration option that makes
it possible for a server operator to adjust the sweep interval and make
a CPU/memory tradeoff:
[sweep_interval]
<integer>
The specified value represents the number of seconds between successive
sweeps. The range of valid values is between 10 and 600.
Important operator notes:
This is an advanced configuration option that should not be used unless
there is empirical data which suggests that the default sweep frequency
is either resulting in performance problems or is causing undue load to
the server.
Note that adjusting the sweep interval may not have the intended effect
on the server. Lower values will not always translate to a reduction of
memory usage and higher values will not always translate to a reduction
of CPU usage and/or load.
* Only require adding the new feature names in one place. (Also need to
increment a counter, but a check on startup will catch that.)
* Allows rippled to have the code to support a given amendment, but
not vote for it by default. This allows the amendment to be enabled in
a future version without necessarily amendment blocking these older
versions.
* The default vote is carried with the amendment name in the list of
supported amendments.
* The amendment table is constructed with the amendment and default
vote.
* Add a new operating mode to rippled called reporting mode
* Add ETL mechanism for a reporting node to extract data from a p2p node
* Add new gRPC methods to faciliate ETL
* Use Postgres in place of SQLite in reporting mode
* Add Cassandra as a nodestore option
* Update logic of RPC handlers when running in reporting mode
* Add ability to forward RPCs to a p2p node
* Creates a version 2 of the UNL file format allowing publishers to
pre-publish the next UNL while the current one is still valid.
* Version 1 of the UNL file format is still valid and backward
compatible.
* Also causes rippled to lock down if it has no valid UNLs, similar to
being amendment blocked, except reversible.
* Resolves#3548
* Resolves#3470
This comment explains this patch and the associated patches
that should be folded into it. This paragraph should be removed
when the patches are folded after review.
This change significantly improves ledger sync and fetch
times while reducing memory consumption. The change affects
the code from that begins with SHAMap::getMissingNodes and runs
through to Database::threadEntry.
The existing code issues a number of async fetches which are then
handed off to the Database's pool of read threads to execute.
The results of each read are placed in the Database's positive
and negative caches. The caller waits for all reads to complete
and then retrieves the results out of these caches.
Among other issues, this means that the results of the first read
cannot be processed until the last read completes. Additionally,
all the results must sit in memory.
This patch changes the behavior so that each read operation has a
completion handler associated with it. The completion of the read
calls the handler, allowing the results of each read to be
processed as it completes. As this was the only reason the
negative and positive caches were needed, they can now be removed.
The read generation code is also no longer needed and is removed.
The batch fetch logic was never implemented or supported and is
removed.
Prior to this commit, the amendments that a server would vote in support
of or against could be configured both via the configuration file and
via the command line "feature" command. Changes made in the configuration
file would only be loaded once at server startup and changes made via the
command line take effect immediately but are not persisted across
restarts.
This commit deprecates management of amendments via the configuration
file and stores the relevant information in the `wallet.db` database
file.
1. On startup, the new code parses the configuration file.
2. If the `[veto_amendments]` or `[amendments]` sections are present,
we check if the `FeatureVotes` table is present in `wallet.db`.
3. If it is not, we create the `FeatureVotes` table and transfer the
settings from the config file.
4. Proceed normally but only reference the `FeatureVotes` table instead
of the config file.
5. Warns if the voting table already exists in `wallet.db` and there
exists voting sections in the config file. The config file is ignored
in this case.
This change addresses & closes#3366
This commit combines a number of cleanups, targeting both the
code structure and the code logic. Large changes include:
- Using more strongly-typed classes for SHAMap nodes, instead of relying
on runtime-time detection of class types. This change saves 16 bytes
of memory per node.
- Improving the interface of SHAMap::addGiveItem and SHAMap::addItem to
avoid the need for passing two bool arguments.
- Documenting the "copy-on-write" semantics that SHAMap uses to
efficiently track changes in individual nodes.
- Removing unused code and simplifying several APIs.
- Improving function naming.
- Simplify and consolidate code for parsing hex input.
- Replace beast::endian::order with boost::endian::order.
- Simplify CountedObject code.
- Remove pre-C++17 workarounds in favor of C++17 based solutions.
- Improve `base_uint` and simplify its hex-parsing interface by
consolidating the `SexHex` and `SetHexExact` methods into one
API: `parseHex` which forces callers to verify the result of
the operation; as a result some public-facing API endpoints
may now return errors when passed values that were previously
accepted.
- Remove the simple fallback implementations of SHA2 and RIPEMD
introduced to reduce our dependency on OpenSSL. The code is
slow and rarely, if ever, exercised and we rely on OpenSSL
functionality for Boost.ASIO as well.
When evaluating the fitness and usefulness of an outbound peer, the code
would incorrectly calculate the amount of time that the peer spent in
a non-useful state.
This commit, if merged, corrects the calculation and makes the timeout
values configurable by server operators.
Two new options are introduced in the 'overlay' stanza of the config
file. The default values, in seconds, are:
[overlay]
max_unknown_time = 600
max_diverged_time = 300
The checkpointer class had assumed that the database would exist for the
lifetime of the application. This is no long true. These changes resolve bugs
involving dangling pointers.
* Document delete_batch, back_off_milliseconds, age_threshold_seconds.
* Convert those time values to chrono types.
* Fix bug that ignored age_threshold_seconds.
* Add a "recovery buffer" to the config that gives the node a chance to
recover before aborting online delete.
* Add begin/end log messages around the SQL queries.
* Add a new configuration section: [sqlite] to allow tuning the sqlite
database operations. Ignored on full/large history servers.
* Update documentation of [node_db] and [sqlite] in the
rippled-example.cfg file.
Resolves#3321
* The amendment ballot counting code contained a minor technical
flaw, caused by the use of integer arithmetic and rounding
semantics, that could allow amendments to reach majority with
slightly less than 80% support. This commit introduces an
amendment which, if enabled, will ensure that activation
requires at least 80% support.
* This commit also introduces a configuration option to adjust
the amendment activation hysteresis. This option is useful on
test networks, but should not be used on the main network as
is a network-wide consensus parameter that should not be
changed on a per-server basis; doing so can result in a
hard-fork.
Fixes#3396
